Methods, systems and related equipment for detecting objects manipulated by mobile robotic arms

Abstract

This invention relates to the field of computer vision technology applications, and particularly to a method, system, and related equipment for detecting objects manipulated by a mobile robotic arm. The detection method includes the following steps: S1, acquiring spatial data of the scene where the mobile robotic arm is located; S2, performing target recognition in the spatial data and marking the objects manipulated by the mobile robotic arm; S3, spatially locating the objects in the spatial data and outputting the location information of the objects. This invention proposes a detection method for self-positioning operations of mobile robotic arms suitable for large-scale workpieces in large scenes. It utilizes deep learning methods to improve the detection accuracy of the manipulated objects and also performs Laplacian enhancement on the training sample images to improve the network detection performance. By first performing two-dimensional detection and then three-dimensional detection, it can detect and locate manipulated objects with diverse types and random distribution locations.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology applications, and in particular to a method, system and related equipment for detecting objects manipulated by a mobile robotic arm. Background Technology

[0002] With the development of industrial structures, more and more industrial production scenarios are using intelligent equipment such as industrial robotic arms to replace tedious and unhealthy manual labor. Mobile robotic arms, with their flexible on-site operation capabilities, are particularly valued in the processing of large workpieces. However, identifying and locating the objects manipulated by the mobile robotic arm in unstructured, large processing spaces is quite difficult. This is partly because the placement of the corresponding vision inspection system is hard to determine, and partly because the characteristics and distribution of the manipulated objects vary, leading to inaccurate identification.

[0003] Current object detection methods primarily utilize traditional feature extraction algorithms and deep learning.

[0004] Traditional feature extraction algorithms are mostly based on sliding window frameworks or matching based on feature points. Their aim is to quickly perform feature calculation and prediction while ensuring the extraction of rich and accurate features. However, the features extracted by traditional algorithms are basically low-level and manually selected features. These features are relatively more intuitive and easier to understand, and more targeted to specific objects, but they cannot well express a large number of multi-class targets.

[0005] There are two main types of deep learning-based object detection algorithms: one is the R-CNN series of object detection frameworks (two-stage) that combine region proposals and CNN networks; the other is algorithms that transform object detection into a regression problem (single-stage). Although deep learning-based object recognition and detection methods can improve the detection and recognition rate, these methods are only suitable for high-quality, large-scale datasets and high-configuration hardware. Furthermore, they only exhibit excellent performance when the training and testing domains remain unchanged. In other words, when there is insufficient training data in the target domain, the trained model is prone to overfitting, meaning it performs well on training data but has poor generalization ability on test data. In complex environments, the objects manipulated by robotic arms for large workpieces are diverse in type and randomly distributed, making training data extraction difficult. Therefore, purely using deep learning methods is still insufficient for detecting and locating objects in large-scale scenes. Summary of the Invention

[0006] This invention provides a method, system, and related equipment for detecting objects operated by a mobile robotic arm, aiming to solve the problems of limited recognition types and difficulty in data acquisition in existing technologies that use traditional feature extraction algorithms and simple deep learning to identify objects operated by mobile robotic arms.

[0007] In a first aspect, embodiments of the present invention provide a method for detecting an object manipulated by a mobile robotic arm, the detection method comprising the following steps:

[0008] S1. Obtain spatial data of the scene where the mobile robotic arm is located;

[0009] S2. Target identification is performed in the spatial data, and the target object operated by the mobile robotic arm is marked;

[0010] S3. Perform spatial positioning of the operation object in the spatial data and output the positioning information of the operation object.

[0011] Furthermore, step S1 includes the following sub-steps:

[0012] The fixed position of the mobile robotic arm is obtained through UBW technology and positioning tags;

[0013] The scene in which the mobile robotic arm is located is structured, and the fixed position of the mobile robotic arm is adjusted.

[0014] The two-dimensional and three-dimensional spatial data of the scene where the mobile robotic arm is located are obtained as the spatial data.

[0015] Furthermore, step S2 includes the following sub-steps:

[0016] Obtain an image containing the object to be operated on as a dataset, and label the object to be operated on in the dataset;

[0017] A target recognition model is obtained by training a pre-defined convolutional neural network based on the dataset.

[0018] The target recognition model is used to identify the spatial data to obtain the identification information of the object being operated on.

[0019] Furthermore, step S3 includes the following sub-steps:

[0020] The scene in which the mobile robotic arm is located is calibrated to establish a spatial coordinate system;

[0021] In the two-dimensional spatial data, a selection box is located based on the identification information to obtain the two-dimensional positioning information of the operation object in the spatial coordinate system;

[0022] Based on the identification information and the two-dimensional positioning information, local positioning and coordinate system transformation are performed in the three-dimensional spatial data to obtain the positioning information of the operation object in the spatial coordinate system.

[0023] Furthermore, the step of acquiring an image containing the operation object as a dataset and labeling the operation object in the dataset further includes the step of:

[0024] Laplacian augmentation is performed on the images in the dataset.

[0025] Furthermore, the step of training a target recognition model using a pre-defined feature convolutional neural network based on the dataset also includes the following step:

[0026] The dataset is expanded by migrating instances from the source domain;

[0027] Knowledge sharing and transfer of the target region are performed during the training process of the pre-defined feature convolutional neural network.

[0028] Secondly, embodiments of the present invention also provide a detection system for a mobile robotic arm operating object, comprising:

[0029] The spatial modeling module is used to acquire spatial data of the scene where the mobile robotic arm is located;

[0030] The target recognition module is used to identify targets in the spatial data and mark the objects operated by the mobile robotic arm;

[0031] The target positioning module is used to spatially locate the operation object in the spatial data and output the positioning information of the operation object.

[0032] Thirdly, embodiments of the present invention also provide a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the detection method for the mobile robotic arm operating object as described in any of the above embodiments.

[0033] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the detection method for the mobile robotic arm operating object as described in any of the above embodiments.

[0034] The beneficial effects achieved by this invention lie in proposing a detection method for self-positioning operations of mobile robotic arms suitable for large-scale workpieces in large scenes. This method utilizes deep learning to improve the detection accuracy of the manipulated object, and further enhances the network detection performance by applying Laplacian augmentation to the training sample images. Simultaneously, by employing a two-dimensional detection followed by a three-dimensional detection approach, the spatial positioning of the manipulated object becomes more accurate, enabling the method to detect and locate manipulated objects with diverse types and random distribution. Furthermore, the detection method of this invention is concise, computationally efficient, and has high detection efficiency, making it highly valuable for practical applications.

[0035] Attached Figure

[0036] Figure 1 This is a flowchart illustrating the steps of the method for detecting the object operated by the mobile robotic arm provided in an embodiment of the present invention;

[0037] Figure 2 This is a schematic diagram of the structure of the detection system for the mobile robotic arm's manipulated object provided in an embodiment of the present invention;

[0038] Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0040] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of a method for detecting an object manipulated by a mobile robotic arm according to an embodiment of the present invention. The detection method includes the following steps:

[0041] S1. Obtain spatial data of the scene where the mobile robotic arm is located.

[0042] Furthermore, step S1 includes the following sub-steps:

[0043] The fixed position of the mobile robotic arm is obtained through UBW technology and positioning tags;

[0044] The scene in which the mobile robotic arm is located is structured, and the fixed position of the mobile robotic arm is adjusted.

[0045] The two-dimensional and three-dimensional spatial data of the scene where the mobile robotic arm is located are obtained as the spatial data.

[0046] Specifically, after acquiring the two-dimensional and three-dimensional spatial data of the scene where the mobile robotic arm is located, the different features in the scene are stitched together according to the correlation between the two-dimensional and three-dimensional views through a geometric field of view stitching planning strategy, thereby obtaining the overall spatial data.

[0047] S2. Target identification is performed in the spatial data, and the target object operated by the mobile robotic arm is marked.

[0048] Obtain an image containing the object to be operated on as a dataset, and label the object to be operated on in the dataset;

[0049] A target recognition model is obtained by training a pre-defined convolutional neural network based on the dataset.

[0050] The target recognition model is used to identify the spatial data to obtain the identification information of the object being operated on.

[0051] Specifically, the preset feature convolutional neural network can be any existing neural network model used for feature recognition and extraction.

[0052] Furthermore, the step of acquiring an image containing the operation object as a dataset and labeling the operation object in the dataset further includes the step of:

[0053] Laplacian augmentation is performed on the images in the dataset.

[0054] Furthermore, the step of training a target recognition model using a pre-defined feature convolutional neural network based on the dataset also includes the following step:

[0055] The dataset is expanded by migrating instances from the source domain;

[0056] Knowledge sharing and transfer of the target region are performed during the training process of the pre-defined feature convolutional neural network.

[0057] S3. Perform spatial positioning of the operation object in the spatial data and output the positioning information of the operation object.

[0058] Furthermore, step S3 includes the following sub-steps:

[0059] The scene in which the mobile robotic arm is located is calibrated to establish a spatial coordinate system;

[0060] In the two-dimensional spatial data, a selection box is located based on the identification information to obtain the two-dimensional positioning information of the operation object in the spatial coordinate system;

[0061] Based on the identification information and the two-dimensional positioning information, local positioning and coordinate system transformation are performed in the three-dimensional spatial data to obtain the positioning information of the operation object in the spatial coordinate system.

[0062] The beneficial effects achieved by this invention lie in proposing a detection method for self-positioning operations of mobile robotic arms suitable for large-scale workpieces in large scenes. This method utilizes deep learning to improve the detection accuracy of the manipulated object, and further enhances the network detection performance by applying Laplacian augmentation to the training sample images. Simultaneously, by employing a two-dimensional detection followed by a three-dimensional detection approach, the spatial positioning of the manipulated object becomes more accurate, enabling the method to detect and locate manipulated objects with diverse types and random distribution. Furthermore, the detection method of this invention is concise, computationally efficient, and has high detection efficiency, making it highly valuable for practical applications.

[0063] This invention also provides a detection system for objects manipulated by a mobile robotic arm. Please refer to... Figure 2 , Figure 2 This is a schematic diagram of the structure of the detection system for the mobile robotic arm's manipulated object provided in an embodiment of the present invention. The detection system 200 for the mobile robotic arm's manipulated object includes:

[0064] Spatial modeling module 201 is used to acquire spatial data of the scene where the mobile robotic arm is located.

[0065] The target recognition module 202 is used to perform target recognition in the spatial data and mark the operation object of the mobile robotic arm;

[0066] The target positioning module 203 is used to spatially locate the operation object in the spatial data and output the positioning information of the operation object.

[0067] The detection system 200 for the mobile robotic arm's manipulated object can implement the steps in the detection method for the mobile robotic arm's manipulated object as described in the above embodiments, and can achieve the same technical effect. Refer to the description in the above embodiments, which will not be repeated here.

[0068] This invention also provides a computer device, please refer to... Figure 3 , Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. The computer device 300 includes: a memory 302, a processor 301, and a computer program stored in the memory 302 and executable on the processor 301.

[0069] The processor 301 calls the computer program stored in the memory 302 to execute the steps in the detection method for the mobile robotic arm's manipulated object provided in this embodiment of the invention. Please refer to... Figure 1Specifically, it includes the following steps:

[0070] S1. Obtain spatial data of the scene where the mobile robotic arm is located.

[0071] Furthermore, step S1 includes the following sub-steps:

[0072] The fixed position of the mobile robotic arm is obtained through UBW technology and positioning tags;

[0073] The scene in which the mobile robotic arm is located is structured, and the fixed position of the mobile robotic arm is adjusted.

[0074] The two-dimensional and three-dimensional spatial data of the scene where the mobile robotic arm is located are obtained as the spatial data.

[0075] S2. Target identification is performed in the spatial data, and the target object operated by the mobile robotic arm is marked.

[0076] Obtain an image containing the object to be operated on as a dataset, and label the object to be operated on in the dataset;

[0077] A target recognition model is obtained by training a pre-defined convolutional neural network based on the dataset.

[0078] The target recognition model is used to identify the spatial data to obtain the identification information of the object being operated on.

[0079] Furthermore, the step of acquiring an image containing the operation object as a dataset and labeling the operation object in the dataset further includes the step of:

[0080] Laplacian augmentation is performed on the images in the dataset.

[0081] Furthermore, the step of training a target recognition model using a pre-defined feature convolutional neural network based on the dataset also includes the following step:

[0082] The dataset is expanded by migrating instances from the source domain;

[0083] Knowledge sharing and transfer of the target region are performed during the training process of the pre-defined feature convolutional neural network.

[0084] S3. Perform spatial positioning of the operation object in the spatial data and output the positioning information of the operation object.

[0085] Furthermore, step S3 includes the following sub-steps:

[0086] The scene in which the mobile robotic arm is located is calibrated to establish a spatial coordinate system;

[0087] In the two-dimensional spatial data, a selection box is located based on the identification information to obtain the two-dimensional positioning information of the operation object in the spatial coordinate system;

[0088] Based on the identification information and the two-dimensional positioning information, local positioning and coordinate system transformation are performed in the three-dimensional spatial data to obtain the positioning information of the operation object in the spatial coordinate system.

[0089] The computer device 300 provided in this embodiment of the invention can implement the steps in the detection method of the mobile robotic arm operating object as described in the above embodiments, and can achieve the same technical effect. Refer to the description in the above embodiments, which will not be repeated here.

[0090] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes and steps in the mobile robotic arm object detection method provided in this invention, and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0091] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0092] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0093] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0094] The embodiments of the present invention have been described above with reference to the accompanying drawings. The disclosed embodiments are merely preferred embodiments of the present invention. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many equivalent changes in form without departing from the spirit and scope of the claims of the present invention, and all such changes are within the protection scope of the present invention.

Claims

1. A method for detecting an object manipulated by a mobile robotic arm, characterized in that, The detection method includes the following steps: S1. Obtain spatial data of the scene where the mobile robotic arm is located; S2. Target identification is performed in the spatial data, and the target object operated by the mobile robotic arm is marked; S3. Perform spatial positioning of the operation object in the spatial data and output the positioning information of the operation object; Step S1 includes the following sub-steps: The fixed position of the mobile robotic arm is obtained through UBW technology and positioning tags; The scene in which the mobile robotic arm is located is structured, and the fixed position of the mobile robotic arm is adjusted. The two-dimensional and three-dimensional spatial data of the scene where the mobile robotic arm is located are obtained as the spatial data; Step S2 includes the following sub-steps: Obtain images containing the operation object as a dataset, and label the operation object in the dataset, wherein Laplacian enhancement is performed on the images in the dataset; A target recognition model is obtained by training a pre-defined feature convolutional neural network based on the dataset, wherein the size of the dataset is expanded by transferring source domain instances; knowledge sharing and transfer of the target region are performed during the training process of the pre-defined feature convolutional neural network. The target recognition model is used to identify the object in the spatial data to obtain the recognition information of the object being operated on. Step S3 includes the following sub-steps: The scene in which the mobile robotic arm is located is calibrated to establish a spatial coordinate system; In the two-dimensional spatial data, a selection box is located based on the identification information to obtain the two-dimensional positioning information of the operation object in the spatial coordinate system; Based on the identification information and the two-dimensional positioning information, local positioning and coordinate system transformation are performed in the three-dimensional spatial data to obtain the positioning information of the operation object in the spatial coordinate system.

2. A detection system for an object manipulated by a mobile robotic arm, characterized in that, include: The spatial modeling module is used to acquire spatial data of the scene where the mobile robotic arm is located; The target recognition module is used to identify targets in the spatial data and mark the objects operated by the mobile robotic arm; The target positioning module is used to spatially locate the operation object in the spatial data and output the positioning information of the operation object; The spatial modeling module is also used for: The fixed position of the mobile robotic arm is obtained through UBW technology and positioning tags; The scene in which the mobile robotic arm is located is structured, and the fixed position of the mobile robotic arm is adjusted. The two-dimensional and three-dimensional spatial data of the scene where the mobile robotic arm is located are obtained as the spatial data; The target recognition module is also used for: Obtain images containing the operation object as a dataset, and label the operation object in the dataset, wherein Laplacian enhancement is performed on the images in the dataset; A target recognition model is obtained by training a pre-defined feature convolutional neural network based on the dataset, wherein the size of the dataset is expanded by transferring source domain instances; knowledge sharing and transfer of the target region are performed during the training process of the pre-defined feature convolutional neural network. The target recognition model is used to identify the object in the spatial data to obtain the recognition information of the object being operated on. The target localization module is also used for: The scene in which the mobile robotic arm is located is calibrated to establish a spatial coordinate system; In the two-dimensional spatial data, a selection box is located based on the identification information to obtain the two-dimensional positioning information of the operation object in the spatial coordinate system; Based on the identification information and the two-dimensional positioning information, local positioning and coordinate system transformation are performed in the three-dimensional spatial data to obtain the positioning information of the operation object in the spatial coordinate system.

3. A computer device, characterized in that, include: The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the method for detecting an object operated by a mobile robotic arm as described in claim 1.

4. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps in the method for detecting an object operated by a mobile robotic arm as described in claim 1.

Images