A dust source tracking control method and system for a multi-degree-of-freedom robotic arm

Abstract

The present invention discloses a multi-degree-of-freedom robotic arm dust source tracking control method and system, which relates to the field of dust removal technology. A multi-degree-of-freedom robotic arm terminal vision system is constructed and calibrated to obtain the visual system parameters. Then, by setting the motion mode of the robotic arm terminal vision system, continuous images are obtained, and feature extraction and matching are performed on the images to form feature point trajectories. Based on these feature point trajectories, three-dimensional reconstruction can be performed to obtain a dust source tracking model, thereby achieving accurate dust source tracking and dust removal operations. The present invention can achieve accurate tracking of dust sources and dust removal operations by using advanced visual systems and image processing technologies. This method can improve dust removal efficiency, reduce energy consumption, and has good application prospects.

Description

Technical Field

[0001] The present invention relates to the technical field of dust removal, and more particularly to a dust source tracking control method and system for a multi-degree-of-freedom robotic arm. Background Art

[0002] At present, many enterprises are seriously polluted by dust. Although traditional fog cannons can achieve some effect in dust removal, they are not widely used in enterprises due to serious waste of water resources, excessive spraying will affect workers' labor, and inconvenience in operation. Many factories such as brick and tile factories, cement factories, sand factories, quarries, coal yards, etc. have workshops for storing materials. There are often many vehicles in the workshops for moving, unloading and loading materials, and dust will be raised into the air during the operation.

[0003] At present, many workshops or dust-generating locations are equipped with fog cannons or pipe spray devices, but fog cannons or spray devices cannot be started for a long time. If they are turned on for a long time, water will accumulate in the workshop, the humidity in the air will be too high, which is not conducive to the operation of operators, and it also wastes water. Therefore, the fog cannon can only be started for a short time and then turned off manually. However, intermittently turning on the fog cannon to suppress dust will cause the dust concentration in the workshop to remain high, resulting in a large amount of dust pollution. Dust pollution will not only affect the workers' work and even affect the product quality, but also affect the health of the operators. High-concentration dust is very harmful to the health of the operators in the workshop, which is a very important issue. In addition, manually turning on and off the spray equipment also consumes manpower and increases human resource costs.

[0004] Therefore, how to automatically track the dust source, improve dust collection efficiency and save resources is a problem that technical personnel in this field need to solve urgently. Summary of the invention

[0005] In view of this, the present invention provides a multi-degree-of-freedom robotic arm dust source tracking control method and system to solve the problems existing in the background technology.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] On the one hand, a multi-degree-of-freedom robot arm dust source tracking control method is provided, comprising:

[0008] Build a multi-degree-of-freedom robotic arm end-point vision system;

[0009] Calibrate the built multi-degree-of-freedom robotic arm end vision system to obtain the vision system parameters;

[0010] By setting the motion mode of the multi-DOF robotic arm end vision system, continuous images can be acquired;

[0011] Extracting features from the continuous images to obtain feature points of the images;

[0012] Match and track the obtained feature points to form feature point trajectories;

[0013] Based on the obtained feature point trajectory, the continuous images are reconstructed in three dimensions to obtain the dust source tracking model;

[0014] Based on the dust source tracking model, tracking dust removal is performed.

[0015] Optionally, the multi-degree-of-freedom robotic arm end vision system includes a multi-degree-of-freedom robotic arm and a monocular camera; a monocular camera is arranged at the end of the multi-degree-of-freedom robotic arm.

[0016] Optionally, the calibration specifically includes: taking at least three or more calibration images at different positions and dividing them into checkerboards, and calculating all internal and external parameters of the monocular camera according to the nonlinear mapping relationship between the image coordinate system, the camera coordinate system and the world coordinate system.

[0017] Optionally, the corresponding motion mode is determined by an action instruction, and the determination of the action instruction specifically includes:

[0018] A set of preset actions are performed multiple times on the same or different test subjects to measure muscle movements, muscle potentials are obtained, electromyographic signals are formed, and the signals are divided into a first training set and a first test set according to a ratio. After feature extraction, the first action of the multi-degree-of-freedom robotic arm is obtained by analysis; video data of the same or different test subjects performing a set of preset actions multiple times is obtained, and action features are extracted as training data, wherein the action features are action sequence and action amplitude; machine learning is performed using the training data to generate a training model for training the first action, and a set of action instructions suitable for the multi-degree-of-freedom robotic arm to complete the preset action is formed.

[0019] Optionally, extracting features from continuous images specifically includes:

[0020] Perform edge extraction on continuous images to obtain edge images;

[0021] fusing the global image information contained in the continuous image and the edge image information contained in the edge image to obtain a fusion result, so as to determine the area including the target object in the continuous image;

[0022] Extracting features from the region to obtain global features and edge features;

[0023] The global features and the edge features are fused to obtain image feature points corresponding to the target object.

[0024] Optionally, fusing the global image information contained in the continuous image and the edge image information contained in the edge image to obtain a fusion result includes:

[0025] Convolution is performed at a first preset frequency on the global image information contained in the continuous image and the edge image information contained in the edge image through a parallel dual-path feature extraction network in the object detection network, so as to obtain reference image features corresponding to the continuous image and the edge image respectively as feature extraction results;

[0026] The reference image features are added, and the sum result is convolved with a second preset frequency to obtain a fusion result.

[0027] Optionally, the method for matching and tracking the feature points to form a feature point trajectory is:

[0028] Match the feature points of different images at the same shooting position, select the Hamming distance for matching, define different digits, and set a preset threshold. If it is less than the preset threshold, it means that the difference is relatively large and it is marked as a matching feature point. Otherwise, it is considered that there is interference between the two feature points and it is marked as a mismatch.

[0029] A random sampling consensus algorithm is used to remove feature points without matching;

[0030] Feature point tracking, based on the obtained feature point matching results, stores the matched feature points into the corresponding feature point queue to form the trajectory of the feature points.

[0031] Optionally, the specific process of the three-dimensional reconstruction is:

[0032] According to the obtained trajectory of feature points, the SFM method is used to obtain the camera parameters of the video key frame queue and the corresponding sparse 3D point cloud;

[0033] The block-based multi-view stereo vision algorithm PMVS is used to expand the sparse 3D point cloud as the extended seed point in the 3D space to obtain a dense 3D point cloud.

[0034] Poisson surface reconstruction is performed on dense 3D point cloud to obtain the dust source tracking model.

[0035] On the other hand, a multi-degree-of-freedom robotic arm dust source tracking control system is provided, comprising:

[0036] Visual system building module, building a multi-degree-of-freedom robotic arm end-point visual system;

[0037] The parameter acquisition module calibrates the multi-degree-of-freedom robotic arm end vision system to obtain the vision system parameters;

[0038] A continuous image acquisition module acquires continuous images by setting the motion mode of the multi-degree-of-freedom robotic arm end vision system;

[0039] A feature point extraction module performs feature extraction on the continuous image to obtain feature points of the image;

[0040] The feature point tracking module matches and tracks the acquired feature points to form feature point trajectories;

[0041] The tracking model generation module performs three-dimensional reconstruction of continuous images based on the obtained feature point trajectory to obtain the dust source tracking model;

[0042] The dust removal module performs tracking dust removal based on the dust source tracking model.

[0043] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a method and system for controlling dust source tracking of a multi-degree-of-freedom robotic arm. By building a multi-degree-of-freedom robotic arm end vision system and calibrating it, the visual system parameters are obtained. Then, by setting the motion mode of the robotic arm end vision system, continuous images are obtained, and feature extraction and matching of the images are performed to form feature point trajectories. Based on these feature point trajectories, three-dimensional reconstruction can be performed to obtain a dust source tracking model, thereby achieving accurate dust source tracking and dust removal operations. The present invention can achieve accurate tracking of dust sources and dust removal operations by using advanced visual systems and image processing technologies. This method can improve dust removal efficiency, reduce energy consumption, and has good application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0045] Figure 1 A flow chart of the method provided by the present invention;

[0046] Figure 2 This is a system structure diagram provided by the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] On the one hand, an embodiment of the present invention discloses a multi-degree-of-freedom robot arm dust source tracking control method, such as Figure 1 As shown, including:

[0049] Build a multi-degree-of-freedom robotic arm end-point vision system;

[0050] Calibrate the built multi-degree-of-freedom robotic arm end vision system to obtain the vision system parameters;

[0051] By setting the motion mode of the multi-DOF robotic arm end vision system, continuous images can be acquired;

[0052] Perform feature extraction on continuous images to obtain feature points of the images;

[0053] Match and track the obtained feature points to form feature point trajectories;

[0054] Based on the obtained feature point trajectory, the continuous images are reconstructed in three dimensions to obtain the dust source tracking model;

[0055] Based on the dust source tracking model, tracking dust removal is carried out.

[0056] In a specific embodiment, a multi-degree-of-freedom robotic arm end vision system includes a multi-degree-of-freedom robotic arm and a monocular camera; a monocular camera is arranged at the end of the multi-degree-of-freedom robotic arm.

[0057] In a specific embodiment, the calibration is specifically as follows: taking at least three or more calibration images at different positions and dividing them into checkerboards, and calculating all internal and external parameters of the monocular camera according to the nonlinear mapping relationship between the image coordinate system, the camera coordinate system and the world coordinate system.

[0058] Let the three-dimensional world coordinate point be P = [X, Y, Z, 1] T , the pixel coordinates of the two-dimensional camera plane are m = [u, v, 1] T , so the homography relationship from the calibration chessboard plane to the image plane is:

[0059] s·m=K[R,Y]P;

[0060] Where s is the scale factor, K is the internal parameter of the monocular camera, R is the rotation matrix, T is the translation vector, and H is the mapping matrix. The world coordinate system is constructed on the chessboard plane, and the chessboard plane is set as the plane with Z = 0, and we can get:

[0061]

[0062] Mapping matrix H = [h1, h2, h3] = λK[r1, r2, t];

[0063] Among them, h1, h2, h3 are three column vectors of the mapping matrix H, λ is the inverse of the scale factor s, r1, r2 are any two column vectors of the rotation matrix R, and t is the translation vector of the image corresponding to r1, r2;

[0064] The camera's intrinsic parameter matrix K is calculated using three images containing a checkerboard;

[0065] We get t = λK -1 h3.

[0066] The external parameter matrix can be obtained using a method similar to the internal parameter matrix.

[0067] In a specific embodiment, the corresponding motion mode is determined by the motion instruction, and the determination of the motion instruction specifically includes:

[0068] A set of preset actions are performed multiple times on the same or different test subjects to measure muscle movements, obtain muscle potentials, form electromyographic signals, and divide them into a first training set and a first test set according to a ratio. After feature extraction, the first action of the multi-degree-of-freedom robotic arm is obtained by analysis; video data of the same or different test subjects performing a set of preset actions multiple times is obtained, and action features are extracted as training data, wherein the action features are action sequence and action amplitude; machine learning is performed using the training data to generate a training model for training the first action, and a set of action instructions suitable for the multi-degree-of-freedom robotic arm to complete the preset action is formed.

[0069] In a specific embodiment, extracting features from continuous images specifically includes:

[0070] Perform edge extraction on continuous images to obtain edge images;

[0071] The global image information contained in the continuous image and the edge image information contained in the edge image are fused to obtain a fusion result, so as to determine the area including the target object in the continuous image;

[0072] Extract features from the region to obtain global features and edge features;

[0073] The global features and edge features are fused to obtain the image feature points corresponding to the target object.

[0074] The target object is dust; edge extraction is performed on the continuous image to obtain an edge image, including:

[0075] The gradient of the continuous image is calculated according to the preset convolution kernel to extract the edge image corresponding to the continuous image.

[0076] In a specific embodiment, the global image information contained in the continuous image and the edge image information contained in the edge image are fused to obtain a fusion result, including:

[0077] Convolution is performed at a first preset frequency on the global image information contained in the continuous image and the edge image information contained in the edge image through a parallel dual-path feature extraction network in the object detection network to obtain reference image features corresponding to the continuous image and the edge image respectively as feature extraction results;

[0078] The reference image features are added, and the sum result is convolved with a second preset frequency to obtain a fusion result.

[0079] Determine the area containing the target object in the continuous images, including:

[0080] The position information corresponding to the target object in the continuous images is identified according to the fusion result, and the area including the target object in the continuous images is determined according to the position information.

[0081] Extract features from the region to obtain global features and edge features, including:

[0082] Crop the area including the target object in the continuous image to obtain the target image corresponding to the area;

[0083] Extracting edge features corresponding to the target image through a first residual network including at least one first residual unit in the feature extraction network;

[0084] According to the bypass structure, the edge data of the first residual unit is superimposed on the output data of the corresponding second residual unit according to the color channel as the input data of the next second residual unit; wherein the feature extraction network includes a second residual network, and the second residual network includes at least one second residual unit; the bypass structure is used to connect the first residual unit and the corresponding second residual unit;

[0085] Residual calculation is performed on the output data of the second residual unit on which the edge data has been superimposed, so as to extract the global features corresponding to the target object.

[0086] Extracting edge features corresponding to the target image through a first residual network including at least one first residual unit in the feature extraction network includes:

[0087] Edge information of the target image is extracted through an edge information extraction layer in the feature extraction network, so that a first residual network including at least one first residual unit extracts edge features corresponding to the target object according to the edge information.

[0088] Perform feature fusion on global features and edge features, including:

[0089] Connect the global feature and the edge feature to obtain a first reference feature, where the dimension of the first reference feature is the sum of the dimensions of the global feature and the edge feature;

[0090] The first reference feature is converted into a dimension-reduced feature to obtain a second reference feature as an image feature point corresponding to the target object.

[0091] In a specific embodiment, the method for matching and tracking feature points to form feature point trajectories is:

[0092] Match the feature points of different images at the same shooting position, select the Hamming distance for matching, define different digits, and set a preset threshold. If it is less than the preset threshold, it means that the difference is relatively large and it is marked as a matching feature point. Otherwise, it is considered that there is interference between the two feature points and it is marked as a mismatch.

[0093] A random sampling consensus algorithm is used to remove feature points without matching;

[0094] Feature point tracking, based on the obtained feature point matching results, stores the matched feature points into the corresponding feature point queue to form the trajectory of the feature points.

[0095] In a specific embodiment, the specific process of 3D reconstruction is:

[0096] According to the obtained feature point trajectory, the SFM method is used to obtain the camera parameters of the video key frame queue and the corresponding sparse 3D point cloud;

[0097] The block-based multi-view stereo vision algorithm PMVS is used to expand the sparse 3D point cloud as the extended seed point in the 3D space to obtain a dense 3D point cloud.

[0098] Poisson surface reconstruction is performed on dense 3D point cloud to obtain the dust source tracking model.

[0099] On the other hand, a multi-degree-of-freedom robotic arm dust source tracking control system is provided, such as Figure 2 As shown, including:

[0100] Visual system building module, building a multi-degree-of-freedom robotic arm end-point visual system;

[0101] The parameter acquisition module calibrates the multi-degree-of-freedom robotic arm end vision system to obtain the vision system parameters;

[0102] A continuous image acquisition module acquires continuous images by setting the motion mode of the multi-degree-of-freedom robotic arm end vision system;

[0103] The feature point extraction module extracts features from continuous images to obtain feature points of the images;

[0104] The feature point tracking module matches and tracks the acquired feature points to form feature point trajectories;

[0105] The tracking model generation module performs three-dimensional reconstruction of continuous images based on the obtained feature point trajectory to obtain the dust source tracking model;

[0106] The dust removal module performs tracking dust removal based on the dust source tracking model.

[0107] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0108] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-degree-of-freedom robotic arm dust source tracking control method, characterized in that: include: Build a multi-degree-of-freedom robotic arm end-point vision system; Calibrate the built multi-degree-of-freedom robotic arm end vision system to obtain the vision system parameters; By setting the motion mode of the multi-DOF robotic arm end vision system, continuous images can be acquired; Extracting features from the continuous images to obtain feature points of the images; Match and track the obtained feature points to form feature point trajectories; Based on the obtained feature point trajectory, the continuous images are reconstructed in three dimensions to obtain the dust source tracking model; Based on the dust source tracking model, tracking dust removal is performed.

2. A multi-degree-of-freedom robot arm dust source tracking control method according to claim 1, characterized in that: The multi-degree-of-freedom robotic arm end visual system comprises a multi-degree-of-freedom robotic arm and a monocular camera; a monocular camera is arranged at the end of the multi-degree-of-freedom robotic arm.

3. A multi-degree-of-freedom robot arm dust source tracking control method according to claim 2, characterized in that: The calibration specifically includes: taking at least three or more calibration images at different positions and dividing them into checkerboards, and calculating all internal and external parameters of the monocular camera according to the nonlinear mapping relationship between the image coordinate system, the camera coordinate system and the world coordinate system.

4. The multi-degree-of-freedom robot arm dust source tracking control method according to claim 1, characterized in that: The corresponding motion mode is determined by the action instruction, and the determination of the action instruction specifically includes: A set of preset actions are performed multiple times on the same or different test subjects to measure muscle movements, muscle potentials are obtained, electromyographic signals are formed, and the signals are divided into a first training set and a first test set according to a ratio. After feature extraction, the first action of the multi-degree-of-freedom robotic arm is obtained by analysis; video data of the same or different test subjects performing a set of preset actions multiple times is obtained, and action features are extracted as training data, wherein the action features are action sequence and action amplitude; machine learning is performed using the training data to generate a training model for training the first action, and a set of action instructions suitable for the multi-degree-of-freedom robotic arm to complete the preset action is formed.

5. The method for tracking and controlling dust sources of a multi-degree-of-freedom robot arm according to claim 1, characterized in that: Feature extraction for continuous images specifically includes: Perform edge extraction on continuous images to obtain edge images; fusing the global image information contained in the continuous image and the edge image information contained in the edge image to obtain a fusion result, so as to determine the area including the target object in the continuous image; Extracting features from the region to obtain global features and edge features; The global features and the edge features are fused to obtain image feature points corresponding to the target object.

6. A multi-degree-of-freedom robot arm dust source tracking control method according to claim 5, characterized in that: The global image information contained in the continuous image and the edge image information contained in the edge image are fused to obtain a fusion result, including: Convolution is performed at a first preset frequency on the global image information contained in the continuous image and the edge image information contained in the edge image through a parallel dual-path feature extraction network in the object detection network, so as to obtain reference image features corresponding to the continuous image and the edge image respectively as feature extraction results; The reference image features are added, and the sum result is convolved with a second preset frequency to obtain a fusion result.

7. The method for tracking and controlling dust sources of a multi-degree-of-freedom robot arm according to claim 1, characterized in that: The method for matching and tracking feature points to form feature point trajectories is as follows: Match the feature points of different images at the same shooting position, select the Hamming distance for matching, define different digits, and set a preset threshold. If it is less than the preset threshold, it means that the difference is relatively large and it is marked as a matching feature point. Otherwise, it is considered that there is interference between the two feature points and it is marked as a mismatch. A random sampling consensus algorithm is used to remove feature points without matching; Feature point tracking, based on the obtained feature point matching results, stores the matched feature points into the corresponding feature point queue to form the trajectory of the feature points.

8. The method for tracking and controlling dust sources of a multi-degree-of-freedom robot arm according to claim 1, characterized in that: The specific process of the three-dimensional reconstruction is as follows: According to the obtained feature point trajectory, the SFM method is used to obtain the camera parameters of the video key frame queue and the corresponding sparse 3D point cloud; The block-based multi-view stereo vision algorithm PMVS is used to expand the sparse 3D point cloud as the extended seed point in the 3D space to obtain a dense 3D point cloud. Poisson surface reconstruction is performed on dense 3D point cloud to obtain the dust source tracking model.

9. A multi-degree-of-freedom robotic arm dust source tracking control system, characterized in that: A multi-degree-of-freedom robotic arm dust source tracking control method applied to any one of claims 1-8, comprising: Visual system building module, building a multi-degree-of-freedom robotic arm end-point visual system; The parameter acquisition module calibrates the multi-degree-of-freedom robotic arm end vision system to obtain the vision system parameters; A continuous image acquisition module acquires continuous images by setting the motion mode of the multi-degree-of-freedom robotic arm end vision system; A feature point extraction module performs feature extraction on the continuous image to obtain feature points of the image; The feature point tracking module matches and tracks the acquired feature points to form feature point trajectories; The tracking model generation module performs three-dimensional reconstruction of continuous images based on the obtained feature point trajectory to obtain the dust source tracking model; The dust removal module performs tracking dust removal based on the dust source tracking model.

Images