Industrial robot visual sorting method and system

Abstract

The invention provides an industrial robot visual sorting method and system. The method comprises the steps that an initial grabbing detection model is generated through synthetic training images in all analog simulation environments; outputting a grabbing point prediction result according to the acquired real image and the initial grabbing detection model, and determining a grabbing deviation coefficient during visual sorting according to the grabbing point prediction result; image high-level semantic features of the workpieces to be sorted are extracted, and the cross-domain feature distribution difference degree between a simulation domain and a reality domain during visual sorting is determined through all the synthesized training images and the image high-level semantic features; and according to the grabbing deviation coefficient and the cross-domain feature distribution difference degree, self-adaptive optimization training is conducted on the initial grabbing point detection model, and then the industrial robot is guided to dynamically sort and grab the to-be-sorted workpiece based on the optimized grabbing point detection model. By adopting the scheme, the sorting flexibility of the industrial robot to the sorted workpieces can be improved in a real-time changing production environment based on simulation and real domain difference self-adaptive optimization.

Description

Technical Field

[0001] This application relates to the field of robot sorting technology, and more specifically, to an industrial robot vision sorting method and system. Background Technology

[0002] Robotic vision sorting is a key technology in modern intelligent manufacturing and logistics sorting. Its main goal is to use visual perception and intelligent algorithms to enable industrial robots to quickly and accurately identify and grasp workpieces to be sorted, thereby achieving automated production and intelligent warehousing. Currently, robotic vision sorting has been widely used in production lines, express sorting centers, and electronic component assembly, which can significantly improve operational efficiency, reduce labor costs, and minimize human error.

[0003] Existing industrial robot vision sorting methods mainly rely on grasping and detection models trained in laboratory or simulated scenarios. These models are typically trained using a large number of synthetic images to learn the shape, position, and posture features of the workpiece. However, there are significant differences between simulated synthetic images and real sorting environments, such as variations in lighting, background complexity, and differences in workpiece surface texture. This leads to insufficient generalization ability of the models in real sorting stations, making them prone to grasping point prediction bias. Furthermore, existing technologies lack effective mechanisms for cross-domain feature transfer, making it difficult for models to adaptively adjust. This results in a decrease in accuracy for models that perform well in the simulation domain but not in the real-world application. Consequently, industrial robots fail to fully integrate visual prediction results with real-world deviations for dynamic optimization during actual sorting and grasping operations, leading to insufficient stability of sorting actions in constantly changing workstation environments. Therefore, how to adaptively optimize based on the differences between the simulation and real domains to improve the sorting flexibility of industrial robots in real-time changing production environments has become a challenge for the industry. Summary of the Invention

[0004] This application provides an industrial robot vision sorting method and system, which can improve the sorting flexibility of industrial robots in real-time changing production environments by adaptively optimizing the difference between the simulation and reality domains.

[0005] In a first aspect, this application provides an industrial robot vision sorting method, comprising the following steps: Acquire multiple synthetic training images of the workpieces to be sorted under different postures and lighting conditions in a preset simulation environment; An initial grasping and detection model for the industrial robot to perform visual sorting of workpieces is generated based on all the synthetic training images. The system acquires real images of the workpieces to be sorted in the sorting station, and then outputs the gripping point prediction results of the workpieces to be sorted from the real images and the initial gripping detection model. The spatial position between the obtained gripping point prediction results and the gripping points marked by the human on the real images is used to determine the gripping deviation coefficient of the industrial robot when performing visual sorting of the workpieces to be sorted. The real image is input into a pre-trained feature extractor to extract the high-level semantic features of the workpiece to be sorted. The cross-domain feature distribution difference between the simulation domain and the real domain is determined by all the synthetic training images and the high-level semantic features of the image. The initial gripping point detection model is adaptively optimized and trained based on the gripping deviation coefficient and the cross-domain feature distribution difference, and then the industrial robot is guided to dynamically sort and grip the workpieces to be sorted based on the optimized gripping point detection model.

[0006] In some embodiments, generating an initial grasping detection model for the industrial robot to perform visual sorting of workpieces based on all synthetic training images specifically includes: Obtain the pre-trained grasping point detection network of the industrial robot; The gripping point detection network is fine-tuned based on all the synthetic training images to obtain the initial gripping point detection model for the industrial robot to perform visual sorting of workpieces.

[0007] In some embodiments, the prediction result of the gripping point of the workpiece to be sorted, output by the real image and the initial gripping detection model, specifically includes: The real image is input into the initial grasping point detection model; The initial gripping point detection model identifies and detects candidate gripping point positions for the workpieces to be sorted; The identified candidate gripping point locations are filtered to obtain the gripping point prediction results for the workpieces to be sorted.

[0008] In some embodiments, determining the gripping deviation coefficient of the industrial robot when performing visual sorting of workpieces by the spatial position between the obtained gripping point prediction result and the gripping point marked by the human on the real image specifically includes: Manually mark the capture points in the real image; The pixel offset between the manually labeled grab points and the predicted grab points is calculated based on the spatial position between the manually labeled grab points and the predicted grab points. The pixel offset is used to determine the gripping deviation coefficient of the industrial robot when performing visual sorting of the workpieces to be sorted.

[0009] In some embodiments, inputting the real image into a pre-trained feature extractor to extract high-level semantic features of the workpiece to be sorted specifically includes: Obtain a pre-trained feature extractor; The real image is input into the feature extractor; The feature extractor performs feature mapping on the input real image to extract high-level semantic features of the workpiece to be sorted.

[0010] In some embodiments, determining the cross-domain feature distribution difference between the simulation domain and the real domain when an industrial robot performs visual sorting of workpieces to be sorted by all synthetic training images and the high-level semantic features of the images specifically includes: The high-level semantic features of the image corresponding to the real image of the workpiece to be sorted are used as the feature vector set corresponding to the real domain; All the synthetic training images are input into a pre-trained feature extractor to extract the corresponding high-level semantic features of the images as the feature vector set corresponding to the simulation domain; By comparing the feature vector set corresponding to the real domain and the feature vector set corresponding to the simulation domain, the cross-domain feature distribution difference degree between the simulation domain and the real domain is obtained when the industrial robot performs visual sorting of the workpieces to be sorted.

[0011] In some embodiments, adaptive optimization training of the initial grasping point detection model based on the grasping deviation coefficient and the cross-domain feature distribution difference specifically includes: The simulation-reality difference index is determined based on the grasping deviation coefficient and the cross-domain feature distribution difference degree; Using the simulation-reality difference index as weighting coefficients, a joint loss function is constructed for the strong supervision signal of the synthetic image and the adaptive signal of the real image domain. The initial grab point detection model is adaptively optimized and trained based on the joint loss function to obtain the optimized grab point detection model.

[0012] In some embodiments, guiding an industrial robot to dynamically sort and grasp workpieces based on an optimized gripping point detection model specifically includes: Acquire real-time images of workpieces to be sorted at the sorting station, and input the real-time images into the optimized gripping point detection model; The optimized gripping point detection model performs deep feature analysis on the input real-time image and outputs the gripping point detection results of the workpiece to be sorted in the image coordinate system. The gripping execution path of the industrial robot is generated based on the output gripping point detection results, and the end effector of the industrial robot is driven to complete the gripping action of the workpiece to be sorted.

[0013] In some embodiments, an industrial camera is used to capture real images of the workpieces to be sorted at the sorting station.

[0014] Secondly, this application provides an industrial robot vision sorting system, comprising: The acquisition module is used to acquire multiple synthetic training images of the workpiece to be sorted under different postures and lighting conditions in a preset simulation environment. The processing module is used to generate an initial grasping detection model for the industrial robot to perform visual sorting of workpieces based on all the synthetic training images. The processing module is also used to acquire real images of the workpieces to be sorted in the sorting station, and then output the gripping point prediction results of the workpieces to be sorted from the real images and the initial gripping detection model. The gripping deviation coefficient of the industrial robot when performing visual sorting of the workpieces to be sorted is determined by the spatial position between the obtained gripping point prediction results and the gripping points marked by the human on the real images. The processing module is also used to input the real image into a pre-trained feature extractor to extract the high-level semantic features of the image of the workpiece to be sorted, and to determine the cross-domain feature distribution difference between the simulation domain and the real domain when the industrial robot performs visual sorting of the workpiece to be sorted by all the synthetic training images and the high-level semantic features of the image. The execution module is used to adaptively optimize and train the initial gripping point detection model based on the gripping deviation coefficient and the cross-domain feature distribution difference, and then guide the industrial robot to dynamically sort and grip the workpieces to be sorted based on the optimized gripping point detection model.

[0015] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects: In this application, multiple synthetic training images of the workpiece to be sorted under different postures and lighting conditions in a preset simulated environment are acquired. An initial grasping detection model for the industrial robot to visually sort the workpiece is generated based on all the synthetic training images. Real images of the workpiece to be sorted in the sorting station are acquired, and the grasping point prediction results of the workpiece to be sorted are output from the real images and the initial grasping detection model. The grasping deviation coefficient of the industrial robot during visual sorting of the workpiece is determined by the spatial position between the obtained grasping point prediction results and the manually marked grasping points on the real images. The real images are input into a pre-trained feature extractor to extract the high-level semantic features of the workpiece image. The cross-domain feature distribution difference between the simulated domain and the real domain is determined by all the synthetic training images and the high-level semantic features. The initial grasping point detection model is adaptively optimized and trained based on the grasping deviation coefficient and the cross-domain feature distribution difference, and then the optimized grasping point detection model guides the industrial robot to dynamically sort and grasp the workpiece.

[0016] Therefore, in this application, firstly, by determining the spatial position between the obtained gripping point prediction results and the manually marked gripping points on the real image, the gripping deviation coefficient of the industrial robot during visual sorting of the workpieces to be sorted can be determined. This can quantitatively reflect the degree of error in the initial gripping point detection model's positioning of the workpieces to be sorted in the real sorting station, providing direct feedback for subsequent model optimization. This allows the model to not only rely on the training patterns of synthetic data during adjustment but also incorporate error performance in the real environment. Secondly, by determining the cross-domain feature distribution difference between the simulated and real domains when the industrial robot performs visual sorting of the workpieces to be sorted, using all synthetic training images and the high-level semantic features of the images, the feature inconsistency between the simulated and real domains can be quantitatively measured. This allows the model to specifically reduce inter-domain deviations during optimization, enabling it to have better transfer capabilities. Thus, the industrial robot can maintain efficient recognition not only in the simulated environment but also in the real workstation during visual sorting. The system is designed to adapt to different environmental conditions, enabling it to flexibly complete sorting and grasping tasks. Finally, based on the grasping deviation coefficient and the cross-domain feature distribution difference, the initial grasping point detection model is adaptively optimized and trained. This allows for dynamic adjustment of the model under real-world conditions, simultaneously correcting positioning errors and feature distribution differences. This ensures the model maintains high prediction accuracy in constantly changing sorting environments, effectively improving the industrial robot's adaptability and sorting flexibility in complex and variable scenarios. Furthermore, the optimized grasping point detection model guides the industrial robot to dynamically sort and grasp workpieces, enabling it to maintain stable and accurate grasping capabilities even under constantly changing lighting, workpiece stacking conditions, or interference factors in the production environment. This effectively improves the adaptability and efficiency of industrial robot sorting. In summary, this solution can improve the sorting flexibility of industrial robots in real-time changing production environments by adaptively optimizing the difference between simulated and real-world domains. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is an exemplary flowchart of an industrial robot vision sorting method according to some embodiments of this application; Figure 2 This is an exemplary flowchart illustrating the determination of the grasping deviation coefficient according to some embodiments of this application; Figure 3This is an exemplary flowchart illustrating the extraction of high-level semantic features from an image according to some embodiments of this application; Figure 4 This is a schematic diagram of the structure of an industrial robot vision sorting system according to some embodiments of this application; Figure 5 This is a schematic diagram of the structure of a computer device for implementing an industrial robot vision sorting method according to some embodiments of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] refer to Figure 1 The figure is an exemplary flowchart of an industrial robot vision sorting method according to some embodiments of this application. The industrial robot vision sorting method mainly includes the following steps: In step 101, multiple synthetic training images of the workpiece to be sorted under different postures and lighting conditions in a preset simulation environment are acquired.

[0021] In specific implementation, acquiring multiple synthetic training images of the workpiece to be sorted under different postures and lighting conditions in a preset simulation environment can be achieved in the following way: First, the 3D computer-aided design (CAD) model of the workpiece to be sorted is imported into a preset simulation software, which refers to a tool platform capable of virtually constructing the workpiece's spatial environment and supporting rendering output; second, posture change processing is performed in the simulation software, such as setting the workpiece's posture in the simulation environment and generating various spatial placement states by controlling rotation angles, translation positions, etc.; third, lighting change processing is performed in the simulation software, such as setting different lighting parameters, including light source intensity, direction, and quantity, to simulate the lighting conditions that may occur in the actual sorting scenario; subsequently, the rendering engine renders each posture and lighting combination to obtain the corresponding synthetic image; finally, all the obtained synthetic images are output in batches as multiple synthetic training images of the workpiece to be sorted under different postures and lighting conditions in the preset simulation environment, which are used as input data for subsequent deep learning model training; other methods can also be used in other embodiments, which are not specifically limited here.

[0022] It should be noted that the simulation environment in this application refers to a virtual scene used to generate image data and sensor outputs consistent with the actual sorting station behavior, which can quickly generate controllable and sufficiently labeled training samples; the synthetic training image in this application refers to image data generated in the simulation environment that can be used for model training.

[0023] In step 102, an initial grasping detection model for the industrial robot to perform visual sorting of workpieces is generated based on all the synthetic training images.

[0024] In some embodiments, generating an initial grasping detection model for the industrial robot to perform visual sorting of workpieces based on all synthetic training images can be achieved through the following steps: Obtain the pre-trained grasping point detection network of the industrial robot; The gripping point detection network is fine-tuned based on all the synthetic training images to obtain the initial gripping point detection model for the industrial robot to perform visual sorting of workpieces.

[0025] In specific implementation, obtaining the pre-trained grasping point detection network for the industrial robot can be achieved in the following way: A deep neural network that has been trained on large-scale visual grasping data or similar tasks can be selected from a publicly available model library or an existing internal model library of the enterprise as the pre-trained grasping point detection network for the industrial robot. Transfer fine-tuning of the grasping point detection network based on all synthetic training images can be performed to obtain the initial grasping point detection model for the industrial robot when visually sorting the workpieces. This can be achieved in the following way: First, the structure of the grasping point detection network can be frozen, retaining only the output layer for training to prevent overfitting. Then, all synthetic training images and their corresponding ground truth labels for grasping points are input into the grasping point detection network, and standardized processing is performed. To increase image diversity, processing methods such as normalization, random cropping, and illumination perturbation enhancement are applied. Then, a supervised learning method is used to fine-tune the grasping point detection network, enabling the network to gradually learn the appearance features of the workpiece to be sorted while maintaining general visual features. Next, some backbone network layers are gradually unfrozen, and weights are updated with a low learning rate, so that the resulting model can adapt to the workpiece to be sorted in terms of feature extraction and grasping point prediction. Finally, the grasping point prediction error is evaluated on a validation set. When the performance reaches a set threshold, the model is output and saved as an initial model that can complete grasping point recognition in actual tasks. This model is used as the initial grasping point detection model for industrial robots to perform visual sorting of workpieces. Other methods can also be used in other embodiments, which are not limited here.

[0026] It should be noted that the gripping point detection network in this application refers to a neural network structure that can take an image of the workpiece to be sorted as input and output a prediction of the gripping point position or posture. As a basic detection tool for visual sorting of industrial robots, it is used to provide transferable gripping point recognition capabilities. The transfer fine-tuning in this application refers to the deep learning process of retraining some parameters of the model on the basis of an existing pre-trained model using a small-scale new dataset to quickly adapt to the workpiece to be sorted, so as to improve the model's recognition and prediction performance in the new field. The initial gripping point detection model in this application refers to a neural network model that can effectively identify the gripping point position on the synthetic image of the workpiece to be sorted. It is used to provide the industrial robot with initial visual sorting capabilities and serves as the basic model for further optimization and training in a real sorting environment.

[0027] In step 103, a real image of the workpiece to be sorted in the sorting station is acquired, and then the gripping point prediction result of the workpiece to be sorted is output by the real image and the initial gripping detection model. The gripping deviation coefficient of the industrial robot when visually sorting the workpiece to be sorted is determined by the spatial position between the obtained gripping point prediction result and the gripping point marked by the human on the real image.

[0028] In specific implementation, the acquisition of real images of the workpieces to be sorted in the sorting station can be achieved in the following way: an industrial camera can be installed at the sorting station. The industrial camera refers to an image acquisition device with high resolution and high-speed imaging capabilities, and then the real images of the workpieces to be sorted in the sorting station can be acquired through the industrial camera. Other methods can also be used for acquisition in other embodiments, which are not specifically limited here.

[0029] It should be noted that the sorting station in this application refers to the fixed working area where the industrial robot performs sorting tasks; the real image in this application refers to the image data of the workpiece to be sorted collected in the actual working environment, which can reflect the appearance characteristics of the workpiece to be sorted in the real scene, and provide a training and verification data basis for the industrial robot grasping and detection model that is consistent with the actual environment.

[0030] In some embodiments, the prediction result of the gripping point of the workpiece to be sorted, output from the real image and the initial gripping detection model, can be achieved by the following steps: The real image is input into the initial grasping point detection model; The initial gripping point detection model identifies and detects candidate gripping point positions for the workpieces to be sorted; The identified candidate gripping point locations are filtered to obtain the gripping point prediction results for the workpieces to be sorted.

[0031] In specific implementation, the initial gripping point detection model can identify candidate gripping point positions for the workpiece to be sorted in the following way: the initial gripping point detection model extracts features from the input real image through its convolutional neural network structure to obtain the edge contour, surface texture, and geometric features of the workpiece to be sorted in space. Then, based on the feature map, the model uses a prediction head to output the candidate gripping point positions for the workpiece to be sorted. The identification of candidate gripping point positions can be filtered to obtain the predicted gripping point results for the workpiece to be sorted in the following way: the optimal gripping point can be determined based on the identified candidate gripping point positions through regression or classification. The obtained optimal gripping point is then post-processed, including non-maximum suppression to remove redundant points and threshold filtering to ensure the confidence of the predicted gripping point. Finally, all the filtered optimal gripping point positions are output as the predicted gripping point results for the workpiece to be sorted, which serve as the input instructions for the subsequent sorting and gripping by the industrial robot. Other methods can also be used in other embodiments, which are not limited here.

[0032] It should be noted that the candidate gripping point position in this application refers to the spatial pose elements of one or more candidate gripping targets identified by the initial gripping point detection model, which serves as the basic unit for evaluating the motion target and gripping action of the industrial robot end effector; the gripping point prediction result in this application refers to the spatial position of the workpiece to be sorted that can be gripped or picked up by the initial gripping point detection model based on the input image, which is used to provide an accurate gripping position reference for the industrial robot.

[0033] In some embodiments, reference Figure 2 As shown, this figure is an exemplary flowchart for determining the gripping deviation coefficient in some embodiments of this application. In this embodiment, the gripping deviation coefficient of the industrial robot when performing visual sorting of workpieces can be determined by the following steps based on the spatial position between the obtained gripping point prediction result and the manually marked gripping point on the real image: First, in step 1031, the grab points are manually marked in the real image; Secondly, in step 1032, the pixel offset between the manually labeled grab point and the obtained grab point prediction result is calculated based on the spatial position between the manually labeled grab point and the grab point prediction result. Finally, in step 1033, the grasping deviation coefficient of the industrial robot when performing visual sorting of the workpieces to be sorted is determined by the pixel offset.

[0034] In specific implementation, the manual annotation of gripping points in the real image can be achieved in the following way: Using an image annotation tool, the gripping positions of the workpieces to be sorted that can be clamped or picked up can be manually marked in the real image, and these positions serve as the manually marked gripping points in the real image. The calculation of the pixel offset between the manually marked gripping points and the predicted gripping points can be achieved in the following way: The predicted gripping points and the manually marked gripping points can be compared one by one in the image coordinate system, and the Euclidean distance between each gripping point in the real image and the corresponding gripping point in the predicted gripping point can be calculated in spatial position. The pixel offset is used as the distance between the manually labeled gripping point and the predicted gripping point. The gripping deviation coefficient of the industrial robot when visually sorting the workpiece to be sorted can be determined by the pixel offset in the following way: statistical analysis can be performed on the pixel offsets corresponding to each gripping point and the predicted gripping point in the real image, such as average or weighted average, to form a quantitative description of the gripping accuracy in the real scene. Finally, the value of the quantitative description is used as the gripping deviation coefficient of the industrial robot when visually sorting the workpiece to be sorted, so as to measure the spatial error between the model predicted point and the real labeled point. Other methods can also be used to determine this in other embodiments, which are not limited here.

[0035] It should be noted that the manually labeled gripping points in this application refer to the reference gripping positions marked by humans on real images, which can be used by industrial robots to perform gripping operations; the pixel offset in this application represents the spatial difference between the model-predicted gripping points and the manually labeled gripping points in the image coordinate system; the gripping deviation coefficient in this application represents the degree of gripping deviation of the industrial robot when performing visual sorting of workpieces in an actual sorting environment. It is used to reflect the gripping accuracy of the industrial robot in a real sorting environment and can guide the adaptive optimization of the subsequent gripping point detection model.

[0036] In step 104, the real image is input into a pre-trained feature extractor to extract the high-level semantic features of the workpiece to be sorted. The cross-domain feature distribution difference between the simulation domain and the real domain when the industrial robot performs visual sorting of the workpiece to be sorted is determined by comparing all the synthetic training images with the high-level semantic features of the image.

[0037] In some embodiments, reference Figure 3 As shown, this figure is an exemplary flowchart of extracting high-level semantic features of an image in some embodiments of this application. In this embodiment, the real image is input into a pre-trained feature extractor, and the extraction of high-level semantic features of the image of the workpiece to be sorted can be achieved by the following steps: First, in step 1041, a pre-trained feature extractor is obtained; Next, in step 1042, the real image is input into the feature extractor; Finally, in step 1043, the input real image is feature-mapped by the feature extractor to extract high-level semantic features of the workpiece to be sorted.

[0038] In specific implementation, the pre-trained feature extractor can be obtained in the following way: a network backbone trained on large-scale visual data can be selected from a public model library or internal model repository as the feature extractor in this application, such as ResNet, EfficientNet, ViT, etc.; the feature extractor can be used to perform feature mapping on the input real image to extract the high-level semantic features of the workpiece to be sorted, which can be achieved in the following way: the real image is input to the feature extractor, and the feature extractor processes the input real image through forward propagation, abstracting layer by layer from low-level edge and texture features to high-level shape, component combination and semantic features of the workpiece to be sorted, thereby completing the feature mapping, and then extracting the high-dimensional feature vector or feature map of the output layer as the high-level semantic features of the workpiece to be sorted, so as to describe the information of the workpiece in terms of spatial layout, shape structure and visual semantics. Then, the extracted high-level semantic features of the image are standardized or dimensionality reduced to facilitate cross-domain comparison with the features of the synthesized training image. Other methods can also be used in other embodiments, which are not limited here.

[0039] It should be noted that, in this application, the feature extractor refers to a deep neural network that has been trained on large-scale image data or similar tasks, which is used to automatically extract multi-level semantic representations from images; the feature mapping in this application refers to the intermediate tensors or vector representations generated by each layer during the forward propagation of the neural network, which abstracts the real image information of the workpiece to be sorted layer by layer, providing a usable representation from low-level texture to high-level semantics; the high-level semantic features of the image in this application refer to the semantic representation vectors generated by the feature extractor that can describe the structure, shape and semantic information of the workpiece to be sorted, which can perform semantic-level feature alignment between the simulated domain and the real domain, improving the adaptability and accuracy of the model in predicting grasping points in real environment.

[0040] In some embodiments, determining the cross-domain feature distribution difference between the simulated and real-world domains when an industrial robot performs visual sorting of workpieces to be sorted using all synthetic training images and the high-level semantic features of the images can be achieved through the following steps: The high-level semantic features of the image corresponding to the real image of the workpiece to be sorted are used as the feature vector set corresponding to the real domain; All the synthetic training images are input into a pre-trained feature extractor to extract the corresponding high-level semantic features of the images as the feature vector set corresponding to the simulation domain; By comparing the feature vector set corresponding to the real domain and the feature vector set corresponding to the simulation domain, the cross-domain feature distribution difference degree between the simulation domain and the real domain is obtained when the industrial robot performs visual sorting of the workpieces to be sorted.

[0041] In specific implementation, the high-level semantic features of the image corresponding to the real image of the workpiece to be sorted can be used as the feature vector set corresponding to the real domain in the following way: the high-level semantic features of the image corresponding to the real image of the workpiece to be sorted can be used as the feature vector set corresponding to the real domain; all synthetic training images can be input into a pre-trained feature extractor to extract the corresponding high-level semantic features of the image as the feature vector set corresponding to the simulation domain in the following way: all synthetic training images can be input into the same pre-trained feature extractor as the real image, and the same operation can be performed to extract the high-level semantic features of the image corresponding to all synthetic training images, and use them as the feature vector set corresponding to the simulation domain; the difference between the feature vector set corresponding to the real domain and the feature vector set corresponding to the simulation domain can be compared to obtain the cross-domain feature distribution difference degree between the simulation domain and the real domain when the industrial robot performs visual sorting of the workpiece to be sorted. This can be achieved by using distribution difference measurement methods, such as maximum mean difference, Fréchet Inception Distance, etc. FID or Euclidean distance statistics of feature embedding are used to compare the feature vector set corresponding to the real domain and the feature vector set corresponding to the simulated domain to calculate the degree of difference in feature distribution between the two. The obtained degree of difference is then used as the cross-domain feature distribution difference degree between the simulated domain and the real domain when the industrial robot performs visual sorting of the workpiece to be sorted. Other methods may also be used to determine this in other embodiments, which are not limited here.

[0042] It should be noted that the feature vector set corresponding to the real domain in this application refers to the set of vectors obtained and normalized by the real image feature extractor of the workpiece to be sorted, which can be used as the distribution representation of the real domain for statistical comparison and alignment; the feature vector set corresponding to the simulated domain in this application refers to the set of vectors after all synthetic training images have been processed and normalized by the same feature extractor, which is used to represent the semantic distribution of the simulated domain; the cross-domain feature distribution difference degree in this application refers to the quantitative index of the statistical distribution difference between the feature sets of the simulated domain and the real domain in the semantic representation space, which is used to quantify the degree of semantic representation deviation between the simulated domain and the real domain, so as to guide the subsequent domain adaptation strategy and model optimization.

[0043] In step 105, the initial gripping point detection model is adaptively optimized and trained based on the gripping deviation coefficient and the cross-domain feature distribution difference degree, and then the industrial robot is guided to dynamically sort and grip the workpieces to be sorted based on the optimized gripping point detection model.

[0044] In some embodiments, adaptive optimization training of the initial grasping point detection model based on the grasping deviation coefficient and the cross-domain feature distribution difference can be achieved using the following steps: The simulation-reality difference index is determined based on the grasping deviation coefficient and the cross-domain feature distribution difference degree; Using the simulation-reality difference index as weighting coefficients, a joint loss function is constructed for the strong supervision signal of the synthetic image and the adaptive signal of the real image domain. The initial grab point detection model is adaptively optimized and trained based on the joint loss function to obtain the optimized grab point detection model.

[0045] In specific implementation, determining the simulation-reality difference index based on the grasping deviation coefficient and the cross-domain feature distribution difference can be achieved in the following way: the grasping deviation coefficient and the cross-domain feature distribution difference can be weighted and fused, and the fused result can be used as the simulation-reality difference index of the difference between the simulation domain and the real domain, so as to comprehensively reflect the prediction error and feature distribution offset of the grasping point detection model in the real scene; constructing a joint loss function of synthetic image strong supervision signal and real image domain adaptive signal using the simulation-reality difference index as weight coefficient can be achieved in the following way: design a joint loss function that fuses synthetic image strong supervision signal and real image domain adaptive signal using the simulation-reality difference index as weight coefficient. This joint loss function includes strong supervision signal provided by synthetic training image, such as cross-entropy loss or regression loss based on ground truth annotation of synthetic training image, as well as real image The provided domain-adaptive signal, such as distribution alignment loss based on adversarial learning or maximum mean difference, enables the model to maintain its learning performance in the simulated domain during training while gradually narrowing the gap with the real domain. The initial grasping point detection model is adaptively optimized and trained based on the joint loss function. The optimized grasping point detection model can be implemented as follows: the parameters of the initial grasping point detection model can be iteratively updated using the joint loss function through backpropagation and gradient descent, continuously adjusting the weights to constrain the model's output on real images, reducing prediction bias and making the features extracted by the model in the simulated and real domains more consistent to improve cross-domain adaptability. After multiple rounds of optimization training, a domain-adapted grasping point detection model is finally obtained, enabling it to output more accurate and stable grasping point prediction results in real sorting stations. Other methods can also be used in other embodiments, which are not limited here.

[0046] It should be noted that the simulation-reality difference index in this application refers to a quantitative index calculated by combining the grasping deviation coefficient and the cross-domain feature distribution difference, which is used to guide the optimization weight allocation in the adaptive optimization training process; the joint loss function in this application refers to a loss function that combines strong supervision in the simulation domain and adaptive signals in the real domain, which is used to simultaneously constrain prediction accuracy and cross-domain adaptability in the adaptive optimization training process; the adaptive optimization training in this application refers to the process of iteratively updating the existing model by combining multiple loss functions, which can dynamically adjust the model parameters of the grasping point detection model so that it can reduce grasping deviation and adapt to different domain feature distributions, thereby improving the actual sorting performance of industrial robots; the optimized grasping point detection model in this application refers to the domain-adaptive grasping point detection model obtained after joint loss optimization training, which can adapt to real sorting scenarios and output accurate grasping points of the workpieces to be sorted.

[0047] In some embodiments, guiding an industrial robot to dynamically sort and grasp workpieces based on an optimized gripping point detection model can be achieved through the following steps: Acquire real-time images of workpieces to be sorted at the sorting station, and input the real-time images into the optimized gripping point detection model; The optimized gripping point detection model performs deep feature analysis on the input real-time image and outputs the gripping point detection results of the workpiece to be sorted in the image coordinate system. The gripping execution path of the industrial robot is generated based on the output gripping point detection results, and the end effector of the industrial robot is driven to complete the gripping action of the workpiece to be sorted.

[0048] In specific implementation, acquiring real-time images of the workpieces to be sorted at the sorting station and inputting these real-time images into the optimized gripping point detection model can be achieved in the following way: Real-time images of the workpieces to be sorted at the sorting station can be acquired in real-time using an industrial camera at the sorting site, and these real-time images can be input into the optimized gripping point detection model. The optimized gripping point detection model can perform deep feature analysis on the input real-time images and output the gripping point detection results of the workpieces to be sorted in the image coordinate system. This can be achieved in the following way: The optimized gripping point detection model can utilize deep learning structures such as convolutional neural networks to perform multi-level feature analysis on the input real-time images to automatically locate the potential gripping points of the workpieces to be sorted in the image coordinate system. The final output is the detection result of the gripping point of the workpiece to be sorted in the image coordinate system. Based on the output gripping point detection result, the gripping execution path of the industrial robot is generated, and the end effector of the industrial robot is driven to complete the gripping action of the workpiece to be sorted. This can be achieved in the following way: the coordinates of the gripping points in the output gripping point detection result are converted into a three-dimensional gripping pose in the mechanical coordinate system of the industrial robot, and the gripping execution path of the industrial robot is generated by combining the robot kinematic model. Finally, the industrial robot drives its end effector (such as a gripper or suction device) to complete the gripping action of the workpiece to be sorted according to the generated gripping execution path, thereby realizing the dynamic sorting and gripping of the workpiece to be sorted. Other methods can also be used in other embodiments, which are not limited here.

[0049] It should be noted that the gripping point detection result in this application refers to the target gripping position coordinates of the workpiece to be sorted output by the optimized gripping point detection model on the input image, which is used to provide the industrial robot with an operable gripping position for the workpiece to be sorted; the dynamic sorting gripping in this application refers to the process of the industrial robot identifying, gripping and placing the workpiece to be sorted in real time in a constantly updated sorting station environment, which is used to improve the adaptability and sorting efficiency of the industrial robot in flexible production.

[0050] In another aspect, in some embodiments, this application provides an industrial robot vision sorting system, with reference to... Figure 4 The figure is a schematic diagram of the structure of an industrial robot vision sorting system 400 according to some embodiments of this application. The industrial robot vision sorting system 400 includes: an acquisition module 401, a processing module 402, and an execution module 403, which are described below: The acquisition module 401 in this application is mainly used to acquire multiple synthetic training images of the workpiece to be sorted under different postures and lighting conditions in a preset simulation environment. Processing module 402, in this application, is mainly used to generate an initial grasping detection model for the industrial robot to perform visual sorting of workpieces to be sorted based on all the synthetic training images. The processing module 402 described in this application is also used to acquire real images of the workpiece to be sorted in the sorting station, and then output the gripping point prediction result of the workpiece to be sorted from the real image and the initial gripping detection model. The gripping deviation coefficient of the industrial robot when performing visual sorting of the workpiece to be sorted is determined by the spatial position between the obtained gripping point prediction result and the gripping point marked by the human on the real image. The processing module 402 described in this application is further used to input the real image into a pre-trained feature extractor to extract the high-level semantic features of the image of the workpiece to be sorted, and to determine the cross-domain feature distribution difference between the simulation domain and the real domain when the industrial robot performs visual sorting of the workpiece to be sorted by all the synthetic training images and the high-level semantic features of the image. The execution module 403 in this application is mainly used to adaptively optimize and train the initial gripping point detection model based on the gripping deviation coefficient and the cross-domain feature distribution difference degree, and then guide the industrial robot to dynamically sort and grip the workpieces to be sorted based on the optimized gripping point detection model.

[0051] The foregoing has detailed examples of the industrial robot vision sorting method and system provided in the embodiments of this application. It is understood that the corresponding apparatus, in order to achieve the above functions, includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0052] In some embodiments, this application also provides a computer device, the computer device including a memory and a processor, the memory for storing a computer program, and the processor for calling and running the computer program from the memory, so that the computer device performs the above-described industrial robot vision sorting method.

[0053] In some embodiments, reference Figure 5 The dashed lines in the figure indicate that the unit or module is optional. This figure is a schematic diagram of the structure of a computer device implementing the industrial robot vision sorting method of this application. The industrial robot vision sorting method in the above embodiments can be achieved through… Figure 5The computer device 500 shown is used to implement this, and the computer device 500 includes at least one processor 501, a memory 502 and at least one communication unit 505. The computer device 500 may be a terminal device, a server or a chip.

[0054] The processor 501 can be a general-purpose processor or a special-purpose processor. For example, the processor 501 can be a central processing unit (CPU). The CPU can be used to control the computer device 500, execute software programs, and process data from the software programs. The computer device 500 may also include a communication unit 505 for inputting (receiving) and outputting (transmitting) signals.

[0055] For example, computer device 500 may be a chip, communication unit 505 may be the input and / or output circuit of the chip, or communication unit 505 may be the communication interface of the chip, and the chip may be a component of terminal device, network device or other device.

[0056] For example, computer device 500 may be a terminal device or a server, and communication unit 505 may be a transceiver of the terminal device or the server, or communication unit 505 may be a transceiver circuit of the terminal device or the server.

[0057] The computer device 500 may include one or more memories 502 storing a program 504. The program 504 can be executed by a processor 501 to generate instructions 503, causing the processor 501 to perform the methods described in the above method embodiments according to the instructions 503. Optionally, the memory 502 may also store data (such as a target audit model). Optionally, the processor 501 may also read data stored in the memory 502, which may be stored at the same storage address as the program 504, or the data may be stored at a different storage address than the program 504.

[0058] The processor 501 and memory 502 can be configured separately or integrated together, for example, integrated on the system-on-chip (SOC) of the terminal device.

[0059] It should be understood that each step of the above method embodiment can be completed by hardware logic circuits or software instructions in the processor 501. The processor 501 can be a CPU, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, such as discrete gate, transistor logic devices, or discrete hardware components.

[0060] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0061] For example, in some embodiments, this application also provides a computer-readable storage medium storing instructions or code that, when executed on a computer, cause the computer to implement the above-described industrial robot vision sorting method.

[0062] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0063] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A visual sorting method for industrial robots, characterized in that, Includes the following steps: Acquire multiple synthetic training images of the workpieces to be sorted under different postures and lighting conditions in a preset simulation environment; An initial grasping and detection model for the industrial robot to perform visual sorting of workpieces is generated based on all the synthetic training images. The system acquires real images of the workpieces to be sorted in the sorting station, and then outputs the gripping point prediction results of the workpieces to be sorted from the real images and the initial gripping detection model. The system determines the gripping deviation coefficient of the industrial robot when performing visual sorting of the workpieces to be sorted by the spatial position between the obtained gripping point prediction results and the gripping points marked by the human on the real images. The real image is input into a pre-trained feature extractor to extract the high-level semantic features of the workpiece to be sorted. The cross-domain feature distribution difference between the simulation domain and the real domain is determined by all the synthetic training images and the high-level semantic features of the image. The initial gripping point detection model is adaptively optimized and trained based on the gripping deviation coefficient and the cross-domain feature distribution difference, and then the industrial robot is guided to dynamically sort and grip the workpieces to be sorted based on the optimized gripping point detection model.

2. The method as described in claim 1, characterized in that, The initial grasping and detection model for the industrial robot to perform visual sorting of workpieces based on all the synthetic training images specifically includes: Obtain the pre-trained grasping point detection network of the industrial robot; The gripping point detection network is fine-tuned based on all the synthetic training images to obtain the initial gripping point detection model for the industrial robot to perform visual sorting of workpieces.

3. The method as described in claim 1, characterized in that, The specific results of the gripping point prediction of the workpiece to be sorted, output by the real image and the initial gripping detection model, include: The real image is input into the initial grasping point detection model; The initial gripping point detection model identifies and detects candidate gripping point positions for the workpieces to be sorted; The identified candidate gripping point locations are filtered to obtain the gripping point prediction results for the workpieces to be sorted.

4. The method as described in claim 1, characterized in that, The grasping deviation coefficient of the industrial robot during visual sorting of workpieces is determined by the spatial position between the obtained grasping point prediction results and the manually marked grasping points on the real image. Specifically, this includes: Manually mark the capture points in the real image; The pixel offset between the manually labeled grab points and the predicted grab points is calculated based on the spatial position between the manually labeled grab points and the predicted grab points. The pixel offset is used to determine the gripping deviation coefficient of the industrial robot when performing visual sorting of the workpieces to be sorted.

5. The method as described in claim 1, characterized in that, The real image is input into a pre-trained feature extractor to extract high-level semantic features of the workpiece to be sorted, specifically including: Obtain a pre-trained feature extractor; The real image is input into the feature extractor; The feature extractor performs feature mapping on the input real image to extract high-level semantic features of the workpiece to be sorted.

6. The method as described in claim 1, characterized in that, The cross-domain feature distribution difference between the simulated domain and the real domain is determined by using all synthetic training images and the high-level semantic features of the images. Specifically, this includes: The high-level semantic features of the image corresponding to the real image of the workpiece to be sorted are used as the feature vector set corresponding to the real domain; All the synthetic training images are input into a pre-trained feature extractor to extract the corresponding high-level semantic features of the images as the feature vector set corresponding to the simulation domain; By comparing the feature vector set corresponding to the real domain and the feature vector set corresponding to the simulation domain, the cross-domain feature distribution difference degree between the simulation domain and the real domain is obtained when the industrial robot performs visual sorting of the workpieces to be sorted.

7. The method as described in claim 1, characterized in that, The adaptive optimization training of the initial grasping point detection model based on the grasping deviation coefficient and the cross-domain feature distribution difference specifically includes: The simulation-reality difference index is determined based on the grasping deviation coefficient and the cross-domain feature distribution difference degree; Using the simulation-reality difference index as weighting coefficients, a joint loss function is constructed for the strong supervision signal of the synthetic image and the adaptive signal of the real image domain. The initial grab point detection model is adaptively optimized and trained based on the joint loss function to obtain the optimized grab point detection model.

8. The method as described in claim 1, characterized in that, The optimized gripping point detection model guides the industrial robot to dynamically sort and grip the workpieces to be sorted, specifically including: Acquire real-time images of workpieces to be sorted at the sorting station, and input the real-time images into the optimized gripping point detection model; The optimized gripping point detection model performs deep feature analysis on the input real-time image and outputs the gripping point detection results of the workpiece to be sorted in the image coordinate system. The gripping execution path of the industrial robot is generated based on the output gripping point detection results, and the end effector of the industrial robot is driven to complete the gripping action of the workpiece to be sorted.

9. The method as described in claim 1, characterized in that, Industrial cameras are used to capture real images of workpieces to be sorted at the sorting station.

10. An industrial robot vision sorting system, characterized in that, include: The acquisition module is used to acquire multiple synthetic training images of the workpiece to be sorted under different postures and lighting conditions in a preset simulation environment. The processing module is used to generate an initial grasping and detection model for the industrial robot to perform visual sorting of workpieces based on all the synthetic training images. The processing module is also used to acquire real images of the workpieces to be sorted in the sorting station, and then output the gripping point prediction results of the workpieces to be sorted from the real images and the initial gripping detection model. The gripping deviation coefficient of the industrial robot when performing visual sorting of the workpieces to be sorted is determined by the spatial position between the obtained gripping point prediction results and the gripping points marked by the human on the real images. The processing module is also used to input the real image into a pre-trained feature extractor to extract the high-level semantic features of the image of the workpiece to be sorted, and to determine the cross-domain feature distribution difference between the simulation domain and the real domain when the industrial robot performs visual sorting of the workpiece to be sorted by all the synthetic training images and the high-level semantic features of the image. The execution module is used to adaptively optimize and train the initial gripping point detection model based on the gripping deviation coefficient and the cross-domain feature distribution difference, and then guide the industrial robot to dynamically sort and grip the workpieces to be sorted based on the optimized gripping point detection model.

Images