An industrial flexible object detection method based on machine vision

Abstract

This invention discloses a machine vision-based method for detecting flexible industrial objects, belonging to the field of industrial inspection technology. The method includes: generating a visual feature vector by combining texture feature vectors with extracted geometric shape feature vectors; extracting global tactile features based on a pre-trained ViT-B / 16 model; extracting position and velocity feature vectors based on a radar mask image using a CSPDarknet network architecture; and generating a comprehensive feature vector using a Transformer model and modal weights. The method efficiently generates radar mask images through a depth mapping transformation network, extracts multimodal features, and uses adaptive model fusion to generate a comprehensive feature vector. It employs smoothing features and online adaptive optimization to dynamically adjust model parameters, reducing computational complexity and annotation costs, improving robustness to lighting changes and deformation scenarios, and enhancing the detection accuracy of flexible object bounding boxes, material classification, and deformation state probabilities.

Description

Technical Field

[0001] This invention relates to the field of industrial inspection technology, and in particular to a machine vision-based method for inspecting flexible industrial objects. Background Technology

[0002] In recent years, machine vision-based industrial flexible object detection technology has made significant progress in intelligent manufacturing, robotic automation, and quality inspection. With the advancement of Industry 4.0, the accurate detection and classification of flexible objects (such as fabrics, leather, and plastics) has become a crucial link in robot grasping, sorting, and quality inspection tasks. In industrial scenarios, flexible object detection needs to simultaneously consider bounding box positioning accuracy, material classification accuracy, and deformation state recognition capabilities. Existing multimodal fusion methods typically rely on high-performance computing resources, making real-time inference difficult on edge devices. Traditional model training's dependence on large amounts of labeled data increases costs, especially in flexible object scenarios where acquiring labeled data is difficult and time-consuming. For dynamic scenarios (such as changes in lighting or object deformation), existing methods lack effective online adaptive optimization mechanisms, leading to a decrease in detection accuracy in practical applications. A high-efficiency, low-cost detection method is needed to achieve real-time, accurate flexible object detection on edge devices, while reducing dependence on labeled data and improving adaptability to dynamic scenarios. Existing machine vision-based industrial flexible object detection technologies suffer from limited computing resources on edge devices, heavy reliance on large amounts of labeled data, and insufficient adaptability to dynamic scenarios. Summary of the Invention

[0003] In view of the aforementioned existing problems, the present invention is proposed.

[0004] Therefore, this invention provides a machine vision-based method for detecting flexible industrial objects, which solves the problems of limited computing resources for edge devices, heavy reliance on a large amount of labeled data, and insufficient adaptability to dynamic scenes in existing machine vision-based industrial flexible object detection technologies.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0006] In a first aspect, the present invention provides a machine vision-based method for detecting flexible industrial objects, comprising: acquiring sensor data through a multimodal sensor and performing preprocessing; and generating a radar mask image based on the preprocessed data through a depth mapping transformation network (DMN).

[0007] Texture feature vectors are extracted using a pre-trained ResNet-50 model. These texture feature vectors are combined with the extracted geometric feature vectors to generate visual feature vectors. Global tactile features are extracted using a pre-trained ViT-B / 16 model. Position and velocity feature vectors are extracted using a CSPDarknet network architecture based on a radar mask image. A comprehensive feature vector is generated using a Transformer model and modal weights.

[0008] Flexible object detection is performed using a lightweight YOLOv5s model based on comprehensive feature vectors. Based on the comprehensive error between the detection results and the real labels, the Transformer model architecture and YOLOv5s model detection head parameters are optimized through online fine-tuning.

[0009] Using the optimized YOLOv5s model, accurate flexible object detection results are generated by smooth feature fusion and edge deployment.

[0010] By analyzing the actual operation results fed back from the industrial control system, the regularization parameters of the YOLOv5s model are dynamically adjusted to generate the final flexible object detection results.

[0011] As a preferred embodiment of the machine vision-based industrial flexible object detection method of the present invention, the step of acquiring sensor data through a multimodal sensor and performing preprocessing refers to using an Intel RealSense D435 camera to acquire RGB images and depth point clouds, and outputting an RGB matrix. ∈ and deep point cloud ∈ Deploy a GelSight tactile sensor to output a contact force matrix U, using a Continental ARS 408-21 to output a 7-dimensional vector from millimeter-wave radar. =[ , , , The system triggers the camera, haptic sensor, and radar via TTL signals, records frame timestamps using the ROS time synchronization protocol, and preprocesses the acquired data to obtain synchronized, denoised RGB images. Smooth depth point cloud and normalized tactile matrix Radar data vector list This represents the number of targets detected per frame.

[0012] As a preferred embodiment of the machine vision-based industrial flexible object detection method of the present invention, the step of generating a radar mask image based on preprocessed data through a Depth Mapping Transformation Network (DMN) refers to processing radar data vectors through a Depth Mapping Transformation Network. The network includes 1D convolutional layers to extract spatial features, 3 layers of ResNet residual modules to fuse shallow and deep features, and finally a fully connected layer to output bounding box coordinates. =[ , , , During training, a self-made MSDataset was used, with the mean squared error loss function measuring the difference between predicted and true coordinates. The Adam optimizer and LeakyReLU activation function were employed for optimization. The output of training and optimization was the optimized depth mapping transformation network parameters, used to accurately map radar data to the image coordinate system for each frame of radar data. one by one target Generate bounding boxes using DMN ,according to The velocity components in the equation are used to calculate the velocity modulus. Then normalize and create a blank mask image, Mradar, initially all zeros, for each target. In Mradar Set the region pixel value to .

[0013] As a preferred embodiment of the machine vision-based industrial flexible object detection method of the present invention, the method comprises: generating a visual feature vector by combining texture feature vectors with extracted geometric shape feature vectors; extracting position and velocity feature vectors based on a radar mask image using a CSPDarknet network architecture; and generating a comprehensive feature vector based on an RGB image using a Transformer model and modal weights. Texture features are extracted by performing convolution operations using a pre-trained ResNet-50. Based on smooth depth point cloud Extracting geometric feature vectors using PointNet Using the feature concatenation method to and Concatenation to form visual feature vectors ;

[0014] Normalized tactile force matrix By interpolating and upsampling to the ViT standard input size Fresized, Fresized is divided into 16×16 segments and flattened into sequences. , convert the sequence Input into the pre-trained ViT-B / 16 to extract global haptic features Based on the radar mask map Mradar, position and velocity features are extracted using the CSPDarknet network architecture. For each feature vector ∈{ , , Calculate the mean and variance, then normalize.

[0015] Based on fully connected networks (MLP) , , Mapping to a unified dimension, the sequence q=[ , , The sequence is encoded using an encoder in a 4-layer Transformer model architecture, and the output sequence is... Calculating modal weights based on fully connected networks Based on weight Calculate the comprehensive feature vector .

[0016] As a preferred embodiment of the machine vision-based industrial flexible object detection method of the present invention, wherein: the flexible object detection based on a lightweight YOLOv5s model using a comprehensive feature vector refers to the detection based on a fused feature vector. A lightweight YOLOv5s model is used to detect flexible objects. The YOLOv5s model includes a trunk, neck, and detection head. The coordinate difference between the predicted bounding box and the ground truth bounding box is calculated using GIoU loss to obtain the position loss. The predicted confidence and the ground truth confidence are calculated using binary cross-entropy loss to obtain the confidence loss. The predicted class probability and the ground truth class probability are calculated using cross-entropy loss to obtain the classification loss. Based on multimodal labeled data, the Adam optimizer is used to minimize the loss and adjust the model parameters. The optimization process gradually reduces the error. After training, the bounding boxes, material classification probabilities, and deformation state probabilities of the flexible objects generated by the detection model are output.

[0017] As a preferred embodiment of the machine vision-based industrial flexible object detection method of the present invention, the method involves: optimizing the Transformer model architecture and YOLOv5s detection head parameters based on the comprehensive error between the detection results and the real labels using an online fine-tuning method; calculating the squared difference between the predicted bounding box center coordinates and the actual coordinates of the flexible object generated by the detection model to measure the positioning deviation; comparing the predicted material and deformation state probabilities with the real categories to calculate the classification error; weighted summing the positioning error and classification error to form a comprehensive error value; calculating the gradient of the Transformer and detection head parameters based on the comprehensive error value; processing a batch of data every fixed number of frames; combining the positioning and classification errors; adjusting parameters to reduce the error; setting the learning rate to a specified value; combining unlabeled data; and using comparative learning to adjust auxiliary parameters to reduce the need for labeled data; using specified labeled data for supervised fine-tuning to correct model deviations.

[0018] As a preferred embodiment of the machine vision-based industrial flexible object detection method of the present invention, wherein: the generation of accurate flexible object detection results by smoothly fusing features and edge deployment using the optimized YOLOv5s model refers to the generation of accurate flexible object detection results based on the optimized YOLOv5s model parameters and the comprehensive feature vector of the current frame. The most recent four frames of features are combined to form a five-frame buffer queue, which is input into a single-layer LSTM network to analyze the time trend and generate a smooth fused feature vector to reduce the impact of noise and illumination fluctuations. The updated YOLOv5s model is converted from 32-bit floating-point numbers to 8-bit integers using the TensorRT tool and deployed to edge devices. The smooth features are input for inference, and the inference speed and detection accuracy are tested. The result is saved as an executable file and integrated into the industrial control system. The smooth features are input into the optimized YOLOv5s model, and three-scale prediction results are generated through the trunk, neck, and detection head. Non-maximum suppression is applied to filter overlapping boxes, and the data is organized into structured data to output accurate detection results.

[0019] As a preferred embodiment of the machine vision-based industrial flexible object detection method of the present invention, the step of dynamically adjusting the regularization parameters of the YOLOv5s model by analyzing real-time feedback, optimizing generalization ability, and generating the final flexible object detection result refers to comparing the detection results with the actual feedback based on accurate detection results and feedback from the industrial control system, quantifying the bounding box positioning deviation and classification error rate, generating a performance deviation index, adjusting the regularization parameters of the YOLOv5s model based on the performance deviation, increasing or decreasing the regularization intensity for the detection head and Transformer fusion module, adjusting every few frames, setting the learning rate to a specified value to maintain stability, applying the adjusted model to the next batch of data, testing the bounding box accuracy and capture success rate, confirming whether it is better than the previous round of performance, if the new model performs better than the previous round, replacing the old model; otherwise, retaining the original model and recording the anomaly log, continuously monitoring feedback, and generating the final flexible object detection result.

[0020] In a second aspect, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein the computer program, when executed by the processor, implements any step of the machine vision-based industrial flexible object detection method as described in the first aspect of the present invention.

[0021] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the machine vision-based industrial flexible object detection method as described in the first aspect of the present invention.

[0022] The beneficial effects of this invention are as follows: It efficiently generates radar mask images through a Deep Map Transformer Network (DMN), extracts multimodal features by combining pre-trained ResNet-50, PointNet, ViT-B / 16, and CSPDarknet, adaptively fuses the features using a Transformer model to generate a comprehensive feature vector, and achieves real-time detection of edge devices through a lightweight YOLOv5s model and TensorRT quantization. It also reduces labeling dependence by combining unlabeled data through comparative learning, and uses LSTM to smooth features and online adaptive optimization to dynamically adjust model parameters, effectively reducing computational complexity and labeling costs, improving robustness to lighting changes and deformation scenarios, and significantly improving the detection accuracy of bounding boxes, material classification probabilities, and deformation state probabilities of flexible objects. This provides an efficient and accurate solution for industrial robot grasping and sorting tasks. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of the machine vision-based industrial flexible object detection method in Example 1.

[0025] Figure 2 This is a structural diagram of the machine vision-based industrial flexible object detection method in Example 1. Detailed Implementation

[0026] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0027] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0028] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0029] Example 1, referring to Figure 1 and Figure 2 This is the first embodiment of the present invention, which provides a machine vision-based method for detecting flexible industrial objects, including the following steps:

[0030] S1. Collect sensor data through a multimodal sensor and preprocess it. Generate a radar mask image based on the preprocessed data through a depth mapping transformation network (DMN).

[0031] Specifically, acquiring sensor data through a multimodal sensor and performing preprocessing refers to using an Intel RealSense D435 camera (1280×720 resolution, 30fps) to acquire RGB images and depth point clouds, and outputting an RGB matrix. ∈ (R represents the real number field, indicating that the matrix elements are real numbers, describing the dimensions of the RGB image (720 high, 1280 wide, 3 channels)) and depth point cloud. ∈ Deploy GelSight tactile sensors (16×16 array) to output a contact force matrix U, recording force values, friction coefficients, and local deformation. Use a Continental ARS 408-21 to output a 7-dimensional vector of millimeter-wave radar. =[ , , , [,c,l,w], where, and The target's longitudinal / lateral distance (meters). and Let c represent the target's longitudinal / lateral velocity (m / s), c represent the target category (integer encoding), and l and w represent the target's length and width (m). The camera, haptic sensor, and radar are triggered via TTL signals. The acquisition timestamp error is ensured to be less than a set threshold. Frame timestamps are recorded using the ROS (Robot Operating System) time synchronization protocol. The acquired data (including RGB images, depth point clouds, haptic data, and radar vectors) is preprocessed to obtain a synchronized, denoised RGB image. Smooth depth point cloud and normalized tactile matrix Radar data vector list This represents the number of targets detected per frame.

[0032] By using TTL signals and the ROS time synchronization protocol, the timestamp error of multimodal data acquisition is ensured to be less than a set threshold. The acquired RGB images, depth point clouds, tactile data, and radar vectors are preprocessed to generate synchronized and aligned denoised RGB images, smoothed depth point clouds, normalized tactile matrices, and radar data vector lists, effectively improving data quality and consistency, reducing noise interference, and enhancing adaptability to dynamic scenes. The preprocessed data supports lightweight model processing, reducing the computational burden on edge devices, while providing high-quality input for subsequent feature extraction and fusion, reducing dependence on large amounts of labeled data. This significantly improves the accuracy, real-time performance, and robustness of industrial flexible object detection, making it suitable for robot grasping and sorting tasks.

[0033] Furthermore, a radar mask image is generated based on the preprocessed data using a Deep Mapping Transformation Network (DMN). This involves processing the radar data vectors using the DMN network, which includes 1D convolutional layers (1 input channel, 4 output channels, 1×2 kernels, 2 strides) to extract spatial features, 3 ResNet residual modules (64 neurons per layer) to fuse shallow and deep features, and finally, a fully connected layer outputs the bounding box coordinates. =[ , , , ] represents the top left corner of the target in the image coordinate system. , ) and bottom right corner ( , For each pixel location, a custom-made MSDataset (700 training samples, 100 test samples) was used during training. The mean squared error loss function was used to measure the difference between the predicted and true coordinates. The Adam optimizer and LeakyReLU activation function were employed for optimization, reducing the test set error to low pixel error and high inference speed. The output of training and optimization is the optimized depth mapping transformation network parameters, used to accurately map radar data to the image coordinate system. one by one target Generate bounding boxes using DMN ,according to The velocity components in the equation are used to calculate the velocity modulus. ( and Represents the bounding box of a flexible object The velocity components in the image coordinate system (corresponding to the motion velocities along the x and y axes, respectively) are normalized, and a blank mask image, Mradar, is created, initially set to all zeros for each target. In Mradar (referring to a radar mask image, representing a single-channel image generated after mapping millimeter-wave radar data to an image coordinate system), Set the region pixel value to The remaining areas remain at 0.

[0034] By processing the preprocessed radar data vectors through a deep mapping transformation network (DMN), the performance of industrial flexible object detection is significantly improved. The radar mask map (Mradar) generated by DMN accurately maps the radar data to the image coordinate system through velocity modulus normalization, reducing noise interference and improving adaptability to dynamic scenes (such as changes in lighting and object deformation). Its lightweight design reduces the computational burden on edge devices (such as Jetson TX2) and reduces the dependence on a large amount of labeled data, providing high-quality input for subsequent multimodal feature fusion and YOLOv5s detection, thereby significantly improving the real-time performance, accuracy, and robustness of flexible object detection.

[0035] S2. Extract texture feature vectors using a pre-trained ResNet-50 model. Combine the texture feature vectors with the extracted geometric shape feature vectors to generate visual feature vectors. Extract global tactile features based on a pre-trained ViT-B / 16 model. Extract position and velocity feature vectors based on a radar mask image using the CSPDarknet network architecture. Generate a comprehensive feature vector using a Transformer model and modal weights.

[0036] Flexible object detection using a lightweight YOLOv5s model based on comprehensive feature vectors;

[0037] Specifically, texture feature vectors are extracted using a pre-trained ResNet-50 model. These texture feature vectors are combined with the extracted geometric shape feature vectors to generate visual feature vectors. Global tactile features are extracted using a pre-trained ViT-B / 16 model. Position and velocity feature vectors are extracted using a CSPDarknet network architecture based on radar mask images. A comprehensive feature vector based on RGB images is generated using a Transformer model and modal weights. Texture features are extracted by performing convolution operations on a pre-trained ResNet-50 (with the first 50 layers frozen and ImageNet weights). :

[0038]

[0039] Based on smooth depth point cloud Geometric feature vectors are extracted using PointNet (a 3-layer Set Abstraction module, with 512, 256, and 128 points sampled per layer). :

[0040]

[0041] Using feature concatenation method and Concatenation to form visual feature vectors (Includes texture and geometry information):

[0042]

[0043] Normalized tactile force matrix By interpolating and upsampling to the ViT standard input size (referring to the input image size required by the VisionTransformer (ViT) model), the Fresized image is divided into 16×16 blocks (each block being 14×14 pixels) and flattened into a sequence. , convert the sequence Input is fed into a pre-trained ViT-B / 16 (12-layer Transformer encoder, 8-head attention, ImageNet-21k weights) to extract global haptic features. (Capturing the hardness, friction, and deformation characteristics of flexible objects):

[0044]

[0045] Based on the radar mask map Mradar, position and velocity features are extracted using a CSPDarknet network architecture (3-layer CSP modules with 64, 128, and 256 channels respectively, including a Bottleneck structure). :

[0046]

[0047] For each feature vector ∈{ , , Calculate the mean and variance, then normalize.

[0048] Based on fully connected networks (MLP) , , Mapped to a unified dimension (256 dimensions):

[0049] ,

[0050] in, It is a 2-layer fully connected network (512-dimensional hidden layer, ReLU activation). ;

[0051] Set sequence q=[ , , The sequence is encoded using an encoder (8-head attention, 32-dimensional per head) in a 4-layer Transformer model architecture, and the output sequence is... ;

[0052] Calculating modal weights based on fully connected networks (The weighting coefficients of the fusion features determine how to combine them) (Achieving adaptive feature fusion)

[0053]

[0054] in, This is an activation function used to map input values ​​to the (0,1) interval. Here is the modal weight matrix. For the modal bias term, adjust the offset in the weight calculation. and It is obtained by training in the Transformer fusion module through a fully connected layer;

[0055] Based on weight Calculate the comprehensive feature vector :

[0056] ,

[0057] in, This is the weight matrix for the fused features, used for the modal features after linear transformation and weighting. To adjust the feature offset as the bias vector for fused features, and All were trained using a fully connected layer.

[0058] This method significantly improves the detection performance of flexible objects in industrial applications by employing multimodal feature extraction and adaptive fusion. It extracts texture features from RGB images using a pre-trained ResNet-50 model and combines this with PointNet to extract geometric features from smooth depth point clouds, concatenating them to form a visual feature vector. Global tactile features are extracted, and position and velocity features are extracted from radar mask images using CSPDarknet. A Transformer model combined with a fully connected network is used to calculate modal weights, adaptively fusing visual, tactile, and radar features to generate a comprehensive feature vector. This multimodal fusion method reduces computational complexity through efficient feature extraction and normalization, adapting to the real-time inference requirements of edge devices. Simultaneously, the Transformer adaptive weight mechanism enhances the model's robustness to dynamic scenes and reduces reliance on large amounts of labeled data, thereby significantly improving the detection accuracy of flexible object bounding boxes, material classification probabilities, and deformation state probabilities. This provides an efficient and accurate solution for industrial robot grasping and sorting tasks.

[0059] Furthermore, flexible object detection using a lightweight YOLOv5s model based on integrated feature vectors refers to the detection of objects based on fused feature vectors. This study uses a lightweight YOLOv5s model to detect flexible objects. The YOLOv5s model consists of a backbone (extracting multi-scale features), a neck (fusing features), and a detection head (outputting bounding boxes, confidence scores, and classification probabilities). The model uses GIoU loss to calculate the coordinate difference between the predicted bounding box and the ground truth bounding box, resulting in a positional loss. Binary cross-entropy loss is used to calculate the predicted confidence score and the ground truth confidence score, resulting in a confidence loss. Cross-entropy loss is used to calculate the predicted class probability and the ground truth class probability, resulting in a classification loss. Based on multimodal labeled data, the Adam optimizer is used to comprehensively minimize the losses (including positional loss, confidence loss, and classification loss). Model parameters are adjusted, and the optimization process gradually reduces errors, improving the accuracy of flexible object detection. After training, the model outputs the bounding box (target location and size), material classification probability (fabric, leather, etc.), and deformation state probability (stretching, compression, bending) of the flexible object.

[0060] Efficient flexible object detection is achieved through a lightweight YOLOv5s model based on comprehensive feature vectors. The YOLOv5s model utilizes multi-scale feature extraction from the backbone, feature fusion from the neck, bounding boxes output by the detection head, material classification probability, and deformation state probability. GIoU loss, binary cross-entropy loss, and cross-entropy loss are used to optimize bounding box localization, confidence, and classification probability, respectively. The Adam optimizer is used to comprehensively minimize the loss. Model parameters are adjusted based on multimodal labeled data, significantly reducing errors and improving detection accuracy. The lightweight design adapts to the real-time inference requirements of edge devices, reducing computational complexity. Combined with adaptive feature vector fusion, robustness to dynamic scenes is enhanced, and efficient training reduces dependence on large amounts of labeled data.

[0061] S3. Based on the combined error between the detection results and the true labels, optimize the Transformer model architecture and YOLOv5s model detection head parameters through online fine-tuning methods;

[0062] Using the optimized YOLOv5s model, accurate flexible object detection results are generated by smooth feature fusion and edge deployment.

[0063] By analyzing the actual operation results fed back from the industrial control system, the regularization parameters of the YOLOv5s model are dynamically adjusted to generate the final flexible object detection results.

[0064] Specifically, based on the combined error between the detection results and the true labels, the Transformer model architecture and YOLOv5s detection head parameters are optimized through online fine-tuning. This involves calculating the squared difference between the predicted bounding box center coordinates and the actual coordinates (for each target, the squared difference between the x-axis and y-axis is calculated, and the average is taken to form the localization error; for example, the x-axis error is the predicted x-coordinate minus the square of the actual x-coordinate, and the y-axis is calculated similarly, resulting in the total localization error). The localization deviation is measured, and the predicted material and deformation state probabilities are compared with the true categories (predicted probabilities (probability values ​​for each category, 6 materials + 3 deformations) are compared with the true categories (one-hot encoded, such as [1,0,0,…] representing the true category). The logarithmic loss between the predicted probabilities and the true categories is calculated and accumulated to form the total classification error). The classification error is then calculated, and the localization error and classification error are weighted and summed to form the combined error value. To guide fine-tuning, based on the comprehensive error value, the gradients of the Transformer and detector head parameters are calculated (through error backpropagation, the impact of each parameter on the error is calculated, forming a gradient vector, which provides the adjustment direction for the Adam optimizer, used to update the weight parameters of the Transformer fusion module, and the optimized parameters are integrated into the YOLOv5s model). A batch of data is processed every fixed number of frames (e.g., every 100 frames), and the localization and classification errors are combined to adjust the parameters to reduce the error. The learning rate is set to a specified value (e.g., 1 / 100,000) to ensure stability. Combined with unlabeled data (referring to RGB images, haptic matrices, and radar masks without manually labeled bounding boxes and classification labels, accounting for about 50% of the dataset), parameter adjustment is assisted by contrastive learning (comparing features of similar targets), reducing the need for labeled data. Supervised fine-tuning is performed using specified labeled data (about 10 sets, containing ground truth bounding boxes and classification labels) to correct model bias.

[0065] By optimizing the Transformer fusion module and YOLOv5s detection head parameters through online fine-tuning, the performance of industrial flexible object detection is significantly improved. Based on the flexible object bounding box, material classification probability, and deformation state probability output by the detection model, comparative learning is used to assist in optimization, reducing the need for labeled data, correcting model bias, reducing the computational burden on edge devices, and improving adaptability to dynamic scenes (such as changes in lighting and deformation). The optimized parameters are integrated into the YOLOv5s model, which significantly improves detection accuracy.

[0066] Furthermore, by utilizing the optimized YOLOv5s model, accurate flexible object detection results are generated through smooth feature fusion and edge deployment, based on the optimized YOLOv5s model parameters and the comprehensive feature vector of the current frame. The features of the most recent four frames are stored (the combined feature vectors of the previous four frames are stored (from the Transformer fusion module), and the combined feature vectors of the most recent four frames are recorded sequentially in time series). A five-frame buffer queue is formed, and a single-layer LSTM network is input to analyze the time trend, generating a smooth fused feature vector to reduce the impact of noise and illumination fluctuations. The updated YOLOv5s model (including the optimized Transformer fusion module and detection head parameters) is converted from 32-bit floating-point numbers to 8-bit integers using the TensorRT tool and deployed to an edge device (such as Jetson TX2). The smooth features are input for inference, and the inference speed (ensuring that the processing time per frame is within tens of milliseconds) and detection accuracy (average accuracy mAP is higher than the preset standard) are tested. The result is saved as an executable file and integrated into the industrial control system. The smooth features are input into the optimized YOLOv5s model, and three-scale prediction results (bounding box, confidence, 6 material types and 3 deformation probabilities) are generated through the trunk, neck and detection head. Non-maximum suppression (NMS) is applied to filter overlapping boxes, organize the data into structured data, and output accurate detection results, including the bounding box (center coordinates, width and height), material classification probability (fabric, leather, etc.) and deformation state probability (stretching, compression, etc.) of flexible objects, which are adapted to industrial real-time robot grasping and sorting tasks.

[0067] By generating three-scale prediction results from the trunk, neck, and detection head, and applying non-maximum suppression (NMS) to filter overlapping boxes, the method outputs structured data that is compatible with industrial control system interfaces. This approach reduces computational complexity through lightweight models and efficient feature smoothing, and reduces reliance on large amounts of labeled data by combining prior multimodal fusion and online fine-tuning. It provides a precise and efficient flexible object detection solution for industrial robot grasping and sorting tasks.

[0068] Furthermore, by analyzing real-time feedback, the regularization parameters of the YOLOv5s model are dynamically adjusted to optimize generalization capabilities and generate the final flexible object detection result. Based on the final detection result (boundary box, material, and deformation probability) and feedback from the industrial control system (such as grasping point deviation and material misclassification), the detection result is compared with the actual feedback. The bounding box positioning deviation (by the average absolute value of the coordinate difference) and classification error rate (the proportion of misclassified materials or deformations) are quantified to generate performance deviation indicators. Based on the performance deviation, the regularization parameters of the YOLOv5s model (such as L2 regularization weight) are adjusted. For the detection head and Transformer fusion module, the regularization strength is increased or decreased (for example, reducing regularization when the deviation is large to enhance learning ability). The adjustment is performed every few frames, and the learning rate is set to a specified value to maintain stability. The adjusted model is applied to the next batch of data to test the bounding box accuracy (mAP) and grasping success rate to confirm whether it is better than the previous round of performance. The adjusted model parameters are saved, and the model on the edge device is updated to ensure continuous adaptation to industrial scenarios.

[0069] By dynamically optimizing the regularization parameters of the YOLOv5s model through real-time feedback analysis, the generalization ability and application effect of industrial flexible object detection are significantly enhanced. The regularization parameters of the YOLOv5s model are dynamically adjusted according to the performance deviation index, and the parameter configuration of the detection head and Transformer fusion module is optimized. The model is updated regularly with a low learning rate to maintain its stability. The updated model is tested on subsequent data, which effectively reduces the computational requirements, improves the adaptability to dynamic environments, and provides a stable detection solution for industrial robot grasping and sorting tasks.

[0070] This embodiment also provides a computer device applicable to the machine vision-based industrial flexible object detection method, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the machine vision-based industrial flexible object detection method proposed in the above embodiment.

[0071] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0072] This embodiment also provides a storage medium on which a computer program is stored. When executed by a processor, the program implements the machine vision-based industrial flexible object detection method proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0073] In summary, this invention efficiently generates radar mask images through a Deep Map Transformer Network (DMN), extracts multimodal features by combining pre-trained ResNet-50, PointNet, ViT-B / 16, and CSPDarknet, adaptively fuses these features using a Transformer model to generate a comprehensive feature vector, and achieves real-time edge device detection through a lightweight YOLOv5s model and TensorRT quantization. Furthermore, it reduces labeling dependence by combining unlabeled data through comparative learning, employs LSTM to smooth features, and dynamically adjusts model parameters through online adaptive optimization. This effectively reduces computational complexity and labeling costs, improves robustness to lighting changes and deformation scenarios, and significantly enhances the detection accuracy of bounding boxes, material classification probabilities, and deformation state probabilities for flexible objects, providing an efficient and accurate solution for industrial robot grasping and sorting tasks.

[0074] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A machine vision-based method for detecting flexible industrial objects, characterized in that: include, Sensor data is collected by a multimodal sensor and preprocessed. Based on the preprocessed data, a radar mask image is generated through a depth mapping transformation network. Texture feature vectors are extracted using a pre-trained ResNet-50 model. These texture feature vectors are then combined with the extracted geometric shape feature vectors to generate visual feature vectors. Global tactile features are extracted using a pre-trained ViT-B / 16 model. Position and velocity feature vectors are extracted using a CSPDarknet network architecture based on radar mask images. A comprehensive feature vector is generated using a Transformer model and modal weights, including features based on RGB images. Texture features are extracted by performing convolution operations using a pre-trained ResNet-50. Based on smooth depth point cloud Extracting geometric feature vectors using PointNet Using the feature concatenation method to and Concatenation to form visual feature vectors ; Normalized tactile force matrix By interpolating and upsampling to the ViT standard input size Fresized, Fresized is divided into 16×16 segments and flattened into sequences. , convert the sequence Input into the pre-trained ViT-B / 16 to extract global haptic feature vectors. Based on the radar mask map Mradar, position and velocity feature vectors are extracted using the CSPDarknet network architecture. For each feature vector Calculate the mean and variance, then normalize. Based on fully connected network MLP Mapping to a unified dimension, the sequence Encoding is performed using an encoder in a 4-layer Transformer model architecture, outputting a sequence. Calculating modal weights based on fully connected networks Based on modal weights Calculate the comprehensive feature vector ; The modal weights are calculated based on a fully connected network. : in, This is an activation function used to map input values ​​to the (0,1) interval. Here is the modal weight matrix. For the modal bias term, adjust the offset in the weight calculation. and It is obtained by training in the Transformer fusion module through a fully connected layer; The modality weight-based Calculate the comprehensive feature vector : in, The weight matrix for fused features, This is the bias vector for the fused features; Flexible object detection is performed using a lightweight YOLOv5s model based on comprehensive feature vectors. Based on the combined error between the detection results and the ground truth labels, an online fine-tuning method is used to optimize the Transformer model architecture and YOLOv5s model detection head parameters. This includes optimizing the bounding box, material classification probability, and deformation state probability of the flexible object generated by the detection model; calculating the squared difference between the predicted bounding box center coordinates and the actual coordinates to measure the localization error; comparing the predicted material and deformation state probabilities with the ground truth categories to calculate the classification error; and weighted summing the localization and classification errors to form a comprehensive error value. Based on this comprehensive error value, the gradients of the Transformer and detection head parameters are calculated. A batch of data is processed every fixed number of frames. By combining localization and classification errors, parameters are adjusted to reduce the overall error. The learning rate is set to a specified value. Combined with unlabeled data, auxiliary parameter adjustments are made through comparative learning to reduce the need for labeled data. Supervised fine-tuning is then performed using specified labeled data to correct model biases. Using the optimized YOLOv5s model, accurate flexible object detection results are generated by smooth feature fusion and edge deployment. By analyzing the actual operation results fed back from the industrial control system, the regularization parameters of the YOLOv5s model are dynamically adjusted to generate the final flexible object detection results.

2. The industrial flexible object detection method based on machine vision as described in claim 1, characterized in that: The process of acquiring sensor data and preprocessing it using a multimodal sensor refers to using an Intel RealSense D435 camera to acquire RGB images and depth point clouds, and outputting an RGB matrix. and deep point cloud R represents the real number field, indicating that the matrix elements are real numbers. It describes the dimensions of the RGB image: 720 for height, 1280 for width, and 3 for channels. A GelSight tactile sensor is deployed, outputting a contact force matrix U. A Continental ARS 408-21 is used to output a 7-dimensional vector from the millimeter-wave radar. ,in, and The vertical / horizontal distance of the target. and Let c represent the target's longitudinal / lateral velocity, c represent the target category, and l and w represent the target's length and width. The camera, haptic sensor, and radar are triggered via TTL signals. Frame timestamps are recorded using the ROS time synchronization protocol. The acquired data is preprocessed to obtain a synchronized, denoised RGB image. Smooth depth point cloud and normalized tactile matrix Radar data vector list This represents the number of targets detected per frame.

3. The industrial flexible object detection method based on machine vision as described in claim 2, characterized in that: The process of generating a radar mask image based on preprocessed data using a Deep Mapping Transformation Network (DMN) involves processing radar data vectors through the DMN network. The network includes 1D convolutional layers to extract spatial features, three ResNet residual modules to fuse shallow and deep features, and finally, a fully connected layer to output bounding box coordinates. , representing the top left corner of the target in the image coordinate system. and bottom right corner For pixel location, a custom-made MSDataset was used during training. The mean squared error loss function was used to measure the difference between the predicted and true coordinates. The Adam optimizer and LeakyReLU activation function were employed for optimization. The output of training and optimization were the optimized depth mapping transformation network parameters, which were used to accurately map radar data to the image coordinate system for each frame of radar data. one by one target Generate bounding boxes using DMN ,according to The velocity components in the equation are used to calculate the velocity modulus. And normalize, and Represents the bounding box of a flexible object For the velocity components in the image coordinate system, create a blank mask map Mradar, initially set to all zeros, for each target. In Mradar Set the region pixel value to .

4. The industrial flexible object detection method based on machine vision as described in claim 3, characterized in that: The flexible object detection based on a lightweight YOLOv5s model using integrated feature vectors refers to the detection of flexible objects based on fused feature vectors. A lightweight YOLOv5s model is used to detect flexible objects. The YOLOv5s model includes a trunk, neck, and detection head. The coordinate difference between the predicted bounding box and the ground truth bounding box is calculated using GIoU loss to obtain the position loss. The predicted confidence and the ground truth confidence are calculated using binary cross-entropy loss to obtain the confidence loss. The predicted class probability and the ground truth class probability are calculated using cross-entropy loss to obtain the classification loss. Based on multimodal labeled data, the Adam optimizer is used to minimize the loss and adjust the model parameters. The optimization process gradually reduces the error. After training, the bounding boxes, material classification probabilities, and deformation state probabilities of the flexible objects generated by the detection model are output.

5. The industrial flexible object detection method based on machine vision as described in claim 4, characterized in that: Using an optimized YOLOv5s model, accurate flexible object detection results are generated through smooth feature fusion and edge deployment. This is based on the optimized YOLOv5s model parameters and the comprehensive feature vector of the current frame. The most recent four frames of features are combined to form a five-frame buffer queue, which is input into a single-layer LSTM network to analyze the time trend and generate a smooth fused feature vector to reduce the impact of noise and illumination fluctuations. The updated YOLOv5s model is converted from 32-bit floating-point numbers to 8-bit integers using the TensorRT tool and deployed to edge devices. The smooth features are input for inference, and the inference speed and detection accuracy are tested. The result is saved as an executable file and integrated into the industrial control system. The smooth features are input into the optimized YOLOv5s model, and three-scale prediction results are generated through the trunk, neck, and detection head. Non-maximum suppression is applied to filter overlapping boxes, and the data is organized into structured data to output accurate detection results.

6. The industrial flexible object detection method based on machine vision as described in claim 5, characterized in that: By analyzing real-time feedback, the regularization parameters of the YOLOv5s model are dynamically adjusted to optimize generalization ability and generate the final flexible object detection result. Based on accurate detection results and feedback from the industrial control system, the detection results are compared with the actual feedback to quantify the bounding box localization deviation and classification error rate, generating a performance deviation index. Based on the performance deviation, the regularization parameters of the YOLOv5s model are adjusted, increasing or decreasing the regularization strength for the detection head and Transformer fusion module, adjusting every few frames. The learning rate is set to a specified value to maintain stability. The adjusted model is applied to the next batch of data to test the bounding box accuracy and capture success rate to confirm whether it is better than the previous round of performance. If the new model performs better than the previous round, the old model is replaced; otherwise, the original model is retained and anomaly logs are recorded. Feedback is continuously monitored to generate the final flexible object detection result.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the machine vision-based industrial flexible object detection method according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the machine vision-based industrial flexible object detection method according to any one of claims 1 to 6.

Images