Motion control method, system and equipment of imaging robot, medium and product
By acquiring and reconstructing depth image sequences of high-voltage cables, point cloud data is generated and the swing trajectory with the highest confidence is predicted. The motion pose of the imaging robot is then adjusted, solving the problem of low motion control accuracy of the imaging robot when the high-voltage cable swings, and achieving stable imaging.
Patent Information
- Application Number
- CN202511741871.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-25
- Publication Date
- 2026-02-17
Smart Images
Figure CN121547692A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a motion control method, system, device, medium, and product for an imaging robot. Background Technology
[0002] The power system is an indispensable infrastructure of modern society, and high-voltage transmission lines within it bear the crucial responsibility of delivering electrical energy from power plants to end-users. In high-voltage transmission lines, tension clamps are key components connecting cables to power towers, and their quality directly affects the safe and stable operation of the power system. Therefore, regular inspections of tension clamps are necessary to promptly identify any problems.
[0003] Traditional manual inspection methods not only suffer from low efficiency and accuracy, but also pose risks such as working at heights and electric shock, which can easily lead to injuries or accidents to workers. Therefore, imaging robots have been introduced for inspection.
[0004] After the imaging robot connects to the high-voltage line, the high-voltage line will swing. After the high-voltage line swings, it is difficult for the imaging robot to take stable real-time pictures in time. The current motion control of the imaging robot has low precision, making it difficult to reliably stabilize the motion control of the imaging robot, which affects the accuracy of its fault inspection imaging. Summary of the Invention
[0005] In view of this, in order to solve the above-mentioned technical problems, the present invention provides a motion control method, system, device, medium and product for an imaging robot.
[0006] The first aspect of this invention provides a motion control method for an imaging robot, comprising:
[0007] Multiple depth images of the target imaging cable during its swinging process are acquired to form a depth image sequence;
[0008] Point cloud data is generated based on the depth image sequence, and the swinging process of the target imaging cable is reconstructed in three dimensions based on the point cloud data to obtain three-dimensional reconstructed data.
[0009] Based on the three-dimensional reconstruction data, the swing trajectory of the target imaging cable is predicted to obtain several candidate swing trajectory information, and the candidate swing trajectory information with the highest confidence is selected from all the candidate swing trajectory information as the target swing trajectory information.
[0010] Motion control information for the target imaging robot is generated based on the target predicted swing trajectory information, and the motion pose of the target imaging robot is adjusted according to the motion control information, wherein the motion control information is used to control the pose of the target imaging robot.
[0011] Preferably, the step of generating point cloud data based on the depth image sequence, and performing three-dimensional reconstruction of the swing process of the target imaging cable based on the point cloud data to obtain three-dimensional reconstruction data, further includes:
[0012] Based on the depth image sequence, a pixel matrix is constructed for each depth image; wherein each element in the pixel matrix corresponds to the depth value of each pixel in the depth image;
[0013] The confidence score and orientation information of each pixel in the pixel matrix are extracted using the Sobel detection operator.
[0014] The confidence level of each pixel in the pixel matrix is compared with a preset edge pixel confidence threshold, and multiple pixels with a confidence level greater than the preset edge pixel confidence threshold are identified as edge region pixels.
[0015] For each edge region pixel, a neighborhood region of a preset size is divided with the edge region pixel as the center. Within the neighborhood region, the number of other edge region pixels with the same direction information as the edge region pixel is counted. If the count of other edge region pixels is less than a preset threshold, the edge region pixel is determined to be an isolated point, and the isolated point is subjected to filtering and smoothing processing.
[0016] Preferably, the step of generating point cloud data based on the depth image sequence, and performing three-dimensional reconstruction of the swing process of the target imaging cable based on the point cloud data to obtain three-dimensional reconstruction data, further includes:
[0017] Edge information of each depth image in the depth image sequence is extracted based on the edge detection operator;
[0018] The edge information is refined based on nonmaximum suppression to obtain edge refinement information;
[0019] The depth image is thinned according to the edge thinning information, and then filtered and denoised.
[0020] Preferably, the step of generating point cloud data based on the depth image sequence, and performing three-dimensional reconstruction of the swing process of the target imaging cable based on the point cloud data to obtain three-dimensional reconstruction data includes:
[0021] Using a preset point cloud generation algorithm, each frame of the depth image in the depth image sequence is converted into corresponding point cloud data;
[0022] The point cloud data of all frames are registered and fused to obtain a three-dimensional point cloud dataset representing the swinging process of the target imaging cable.
[0023] Using a 3D reconstruction algorithm, 3D reconstruction data of the swinging process of the target imaging cable is generated based on the 3D point cloud dataset. The 3D reconstruction data is used to reflect the spatial morphology and positional changes of the target imaging cable during the swinging process.
[0024] Preferably, the step of predicting the swing trajectory of the target imaging cable based on the three-dimensional reconstruction data to obtain several candidate predicted swing trajectory information, and selecting the candidate predicted swing trajectory information with the highest confidence from all the candidate predicted swing trajectory information as the target swing trajectory information, includes:
[0025] The 3D reconstructed data is input into a pre-trained trajectory prediction model, which outputs several candidate predicted swing trajectories. Each candidate predicted swing trajectory includes the predicted swing trajectory and the corresponding confidence level. The pre-trained trajectory prediction model is obtained by training a machine learning network using 3D reconstructed data samples of imaging cables with historical swing trajectories.
[0026] All candidate predicted swing trajectory information are sorted in descending order of confidence level, and the candidate predicted swing trajectory information with the highest confidence level is selected as the target swing trajectory information; wherein, the target swing trajectory information is used to represent the swing trajectory of the target imaging cable in the future time period.
[0027] Preferably, the step of generating motion control information for the target imaging robot based on the target predicted swing trajectory information, and adjusting the motion pose of the target imaging robot according to the motion control information, wherein the motion control information is used to control the pose of the target imaging robot, including:
[0028] Based on the target predicted swing trajectory information, multiple point poses of the target imaging robot are matched in a preset trajectory pose mapping library; wherein, the preset trajectory pose mapping library contains the correspondence between various swing trajectories and the point poses of the imaging robot.
[0029] Based on the poses of multiple points of the target imaging robot, the change angle of the poses of adjacent points is obtained;
[0030] Determine the average angle of change of position based on the angle of change of all position poses;
[0031] Based on the sliding window and the average point change angle, the poses of multiple points are smoothed to obtain the poses of points where the point change angle is less than the average point change angle, which is then used as the smoothed poses of points.
[0032] The motion control information of the target imaging robot is generated based on the smoothed point pose, and the motion pose of the target imaging robot is adjusted based on the motion control information.
[0033] Secondly, the present invention also provides a motion control system for an imaging robot, comprising:
[0034] The image acquisition module is used to acquire multiple frames of depth images of the target imaging cable during its swinging process, forming a depth image sequence;
[0035] The three-dimensional reconstruction module is used to generate point cloud data based on the depth image sequence, and to perform three-dimensional reconstruction of the swing process of the target imaging cable based on the point cloud data to obtain three-dimensional reconstruction data.
[0036] The swing trajectory prediction module is used to predict the swing trajectory of the target imaging cable based on the three-dimensional reconstruction data, obtain several candidate predicted swing trajectory information, and select the candidate predicted swing trajectory information with the highest confidence from all the candidate predicted swing trajectory information as the target swing trajectory information.
[0037] A motion control module is used to generate motion control information for a target imaging robot based on the target predicted swing trajectory information, and to adjust the motion pose of the target imaging robot according to the motion control information, wherein the motion control information is used to control the pose of the target imaging robot.
[0038] Thirdly, the present invention also provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, which, when executed by the processor, causes the processor to perform the steps of the motion control method for the imaging robot as described in the first aspect.
[0039] Fourthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the steps of the motion control method for the imaging robot as described in the first aspect.
[0040] Fifthly, the present invention also provides a computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein, when the program instructions are executed by a computer, the computer performs the steps of the motion control method for the imaging robot as described in the first aspect.
[0041] As can be seen from the above technical solution, the present invention acquires multiple frames of depth images of the target imaging cable during its swing process, constructs a depth image sequence, generates point cloud data based on the depth image sequence, performs three-dimensional reconstruction of the swing process of the target imaging cable based on the point cloud data, predicts the swing trajectory of the target imaging cable based on the three-dimensional reconstruction data, and selects the candidate predicted swing trajectory information with the highest confidence from the candidate predicted swing trajectory information as the target swing trajectory information, thereby effectively improving the prediction accuracy of the swing trajectory and avoiding excessive deviation between the obtained swing trajectory and the actual trajectory. Furthermore, the present invention generates motion control information for the target imaging robot based on the target predicted swing trajectory information to adjust the motion pose of the target imaging robot, avoid interference from the swing of the high-voltage line to the real-time imaging robot, ensure the stability of the real-time imaging robot during operation, and improve the accuracy and reliability of the motion control of the imaging robot. Attached Figure Description
[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0043] Figure 1 An application environment diagram of a motion control method for an imaging robot provided in an embodiment of the present invention;
[0044] Figure 2 A flowchart of a motion control method for an imaging robot provided in an embodiment of the present invention;
[0045] Figure 3 This is a schematic diagram of the motion control system of an imaging robot provided in an embodiment of the present invention;
[0046] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0047] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0048] The motion control method for the imaging robot provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 101 communicates with server 102 via a network. A data storage system can store the data that server 102 needs to process. The data storage system can be integrated onto server 102 or placed in the cloud or on another network server. Terminal 101 or server 102 acquires multiple frames of depth images of the target imaging cable during its swinging process, forming a depth image sequence; point cloud data is generated based on the depth image sequence; the swinging process of the target imaging cable is reconstructed in three dimensions based on the point cloud data, resulting in three-dimensional reconstructed data; the swinging trajectory of the target imaging cable is predicted based on the three-dimensional reconstructed data, yielding several candidate predicted swinging trajectory information; the candidate predicted swinging trajectory information with the highest confidence is selected from all candidate predicted swinging trajectory information as the target swinging trajectory information; motion control information for the target imaging robot is generated based on the target predicted swinging trajectory information, and the motion pose of the target imaging robot is adjusted according to the motion control information, wherein the motion control information is used to control the pose of the target imaging robot.
[0049] Terminal 101 can be, but is not limited to, various personal computers, laptops, smartphones, and tablets.
[0050] Server 102 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.
[0051] like Figure 2 As shown, this application provides a motion control method for an imaging robot, which is applied to... Figure 1 Taking terminal 101 or server 102 as an example, the explanation includes the following steps S1 to S4. Wherein:
[0052] Step S1: Acquire multiple depth images of the target imaging cable during its swinging process to form a depth image sequence.
[0053] In this process, after the real-time imaging robot docks with the target imaging cable (such as an overhead line), it acquires multiple frames of depth images of the target imaging cable during its swinging process. These images are obtained through a depth sensor and can record the spatial position information of the cable at different times, thus forming a depth image sequence.
[0054] Step S2: Generate point cloud data based on the depth image sequence, and perform three-dimensional reconstruction of the swing process of the target imaging cable based on the point cloud data to obtain three-dimensional reconstruction data.
[0055] The process involves converting each frame of the depth image sequence into corresponding point cloud data, which contains the three-dimensional coordinate information of each point on the cable surface. Subsequently, the point cloud data from all frames undergo registration and fusion processing. The registration process aims to eliminate relative displacement and rotation differences between different frames, ensuring that all point cloud data are in the same coordinate system. The fusion process combines multiple frames of point cloud data into a complete dataset, resulting in a three-dimensional point cloud dataset that comprehensively reflects the swaying process of the target imaging cable.
[0056] Then, based on the three-dimensional point cloud dataset, three-dimensional reconstruction data of the target imaging cable swing process is generated. This three-dimensional reconstruction data can accurately present the spatial shape and positional changes of the target imaging cable during the swing process.
[0057] Step S3: Based on the three-dimensional reconstruction data, predict the swing trajectory of the target imaging cable to obtain several candidate predicted swing trajectory information, and select the candidate predicted swing trajectory information with the highest confidence from all candidate predicted swing trajectory information as the target swing trajectory information.
[0058] Among them, the candidate predicted swing trajectory information with the highest confidence is selected from all candidate predicted swing trajectory information as the target swing trajectory information, which effectively improves the prediction accuracy of the swing trajectory, avoids the obtained swing trajectory from deviating too much from the actual swing trajectory, and provides motion guidance for real-time imaging robots.
[0059] Step S4: Generate motion control information for the target imaging robot based on the target predicted swing trajectory information, and adjust the motion pose of the target imaging robot according to the motion control information. The motion control information is used to control the pose of the target imaging robot.
[0060] Specifically, predicting the target's swing trajectory provides the foundation for generating precise motion control information. Based on this motion control information, the robot's pose is adjusted to ensure accurate and stable movement along the predicted swing trajectory, thus achieving effective tracking and imaging of the target imaging cable.
[0061] It should be noted that, in this embodiment, multiple frames of depth images of the target imaging cable during its swing process are collected to form a depth image sequence. Point cloud data is generated based on the depth image sequence. The swing process of the target imaging cable is reconstructed in three dimensions based on the point cloud data. The swing trajectory of the target imaging cable is predicted based on the three-dimensional reconstruction data. The candidate predicted swing trajectory information with the highest confidence is selected from the candidate predicted swing trajectory information as the target swing trajectory information. This effectively improves the prediction accuracy of the swing trajectory and avoids excessive deviation between the obtained swing trajectory and the actual trajectory. Furthermore, motion control information for the target imaging robot is generated based on the target predicted swing trajectory information to adjust the motion pose of the target imaging robot, avoid interference from the swing of the high-voltage line to the real-time imaging robot, ensure the stability of the real-time imaging robot during operation, and improve the accuracy and reliability of the motion control of the imaging robot.
[0062] In some embodiments, to avoid isolated points in the image affecting the 3D reconstruction process, before generating point cloud data based on the depth image sequence and performing 3D reconstruction on the swinging process of the target imaging cable based on the point cloud data to obtain 3D reconstruction data, the method further includes:
[0063] Step S111: Construct a pixel matrix for each depth image based on the depth image sequence; wherein each element in the pixel matrix corresponds to the depth value of each pixel in the depth image.
[0064] The pixel matrix stores the depth information of each pixel. The rows of the matrix correspond to the height direction of the image, that is, the pixel position of the image in the vertical direction, and the columns of the matrix correspond to the width direction of the image, that is, the pixel position of the image in the horizontal direction.
[0065] Step S112: Extract the confidence and orientation information of each pixel in the pixel matrix using the Sobel detection operator.
[0066] In this embodiment, the Sobel detection operator is used to process the pixel matrix, enabling the extraction of the confidence level and orientation information for each pixel. The confidence level reflects the probability that the pixel belongs to the target imaging cable, while the orientation information indicates the direction of the cable edge at that pixel.
[0067] Specifically, the gradients of each pixel in the pixel matrix in preset directions (horizontal and vertical) are calculated using the Sobel operator, and the horizontal gradient G of each pixel is determined using the Sobel operator. x The vertical gradient G of each pixel is calculated using the Sobel operator to determine the vertical direction. y By G=|G x | 2 +|G y | 2Calculate the total gradient G for each pixel, use the total gradient G as the confidence level of the corresponding pixel, and compare the vertical gradient with the horizontal gradient to determine the direction of the maximum gradient as the orientation information of the pixel.
[0068] Step S113: Compare the confidence level of each pixel in the pixel matrix with the preset edge pixel confidence level threshold, and select multiple pixels with a confidence level greater than the preset edge pixel confidence level threshold as edge region pixels.
[0069] The preset edge pixel confidence threshold is a key parameter (e.g., 0.8 to 0.95) used to determine whether a pixel belongs to an edge region. By comparing the confidence of each pixel with this threshold, pixels with higher confidence can be selected, as these pixels are more likely to belong to the edge region of the target imaging cable.
[0070] Step S114: For each edge region pixel, divide a neighborhood region of a preset size with the edge region pixel as the center. Within the neighborhood region, count the number of other edge region pixels with the same orientation information as the edge region pixel. If the count of other edge region pixels is less than a preset threshold, then the edge region pixel is determined to be an isolated point, and the isolated point is filtered and smoothed.
[0071] The preset size is a suitable range (e.g., a pixel area of 3x3 to 7x7) derived from multiple experiments and empirical observations. Within the neighborhood, the number of other edge region pixels with the same orientation information as the central edge region pixel is counted to determine whether the central pixel is an isolated point. If the count is greater than or equal to a preset threshold (this threshold is set according to the actual application scenario and accuracy requirements, such as 5 to 10), then the pixel is considered to have enough pixels with similar orientations around it, meaning the pixel is not an isolated point; otherwise, it is determined to be an isolated point.
[0072] For pixels identified as isolated points, filtering and smoothing techniques (such as median filtering and mean filtering) are used to smooth them out, in order to eliminate any negative impact they may have on the 3D reconstruction process and ensure the accuracy and stability of the 3D reconstruction data.
[0073] In some embodiments, point cloud data is generated based on a depth image sequence, and the swinging process of the target imaging cable is reconstructed in three dimensions based on the point cloud data to obtain three-dimensional reconstructed data. This process further includes:
[0074] Step S121: Extract edge information of each depth image in the depth image sequence based on the edge detection operator.
[0075] Among them, the edge detection operator can be the Sobel operator to extract the edge information of each depth image in the depth image sequence.
[0076] Step S122: Refine the edge information based on nonmaximum suppression to obtain edge refinement information.
[0077] Non-maximum suppression (NVS) compares the values of neighboring pixels along the gradient direction of a pixel, retaining only the local maximum value to obtain finer and more accurate edge lines. Specifically, for each edge pixel, the values of its neighboring pixels along the gradient direction are checked. If the value of the pixel is not the maximum value in its neighborhood, it is suppressed (i.e., set to zero or the background value), retaining only the pixel with the maximum value along the gradient direction, thus obtaining edge refinement information.
[0078] Step S123: Thin the depth image according to the edge thinning information, and then perform filtering and noise reduction processing on the thinned depth image.
[0079] In this process, based on the edge refinement information, morphological operations (such as erosion and dilation) are used to refine the depth image in order to remove redundant pixels and make the edges clearer and more precise.
[0080] Subsequently, the refined depth image is filtered and denoised, for example, by using Gaussian filtering or multi-level median filtering, to eliminate noise and interference in the image, further improve image quality, and provide more accurate and reliable input data for the subsequent 3D reconstruction process.
[0081] In some embodiments, point cloud data is generated based on a depth image sequence, and the oscillation process of the target imaging cable is reconstructed in three dimensions based on the point cloud data to obtain three-dimensional reconstructed data, including:
[0082] Step S201: Using a preset point cloud generation algorithm, convert each frame of the depth image in the depth image sequence into corresponding point cloud data.
[0083] Among them, the point cloud generation algorithm can be an algorithm based on the principle of triangulation, which uses known camera position and pose information, combined with the depth value of pixels in the depth image, to calculate the coordinates of each pixel in three-dimensional space using triangulation relationships, and then generates point cloud data; or it can adopt a point cloud generation algorithm based on deep learning, which uses a large number of depth images and corresponding point cloud data as training samples to train a deep neural network model. This model can automatically learn the mapping relationship from depth images to point cloud data. When a new depth image is input, the model can quickly and accurately generate the corresponding point cloud data.
[0084] Step S202: Register and fuse the point cloud data of all frames to obtain a three-dimensional point cloud dataset that represents the swinging process of the target imaging cable.
[0085] Specifically, the generated point cloud data is divided into target point clouds according to the reference coordinate system. This involves transforming the point cloud data from the reference coordinate system to the target coordinate system to obtain the target point cloud, and dividing the point cloud of the preset viewpoint into the reference point cloud.
[0086] Then, coarse registration is performed based on the target point cloud and the reference point cloud using the RANSAC (Random Sample Consensus) algorithm. The RANSAC algorithm estimates the transformation parameters by randomly selecting sample point pairs and iteratively optimizes them to find the optimal registration parameters, thereby initially aligning the target point cloud with the reference point cloud. Next, fine registration is performed using the ICP (Iterative Closest Point) algorithm. The ICP algorithm iteratively finds the closest point pairs between the target point cloud and the reference point cloud and calculates the transformation relationship between them, so that the target point cloud gradually approaches the reference point cloud, ultimately achieving high-precision registration of all frame point cloud data.
[0087] After registration, the point cloud data of all frames are fused. A weighted average method can be used for data fusion, which assigns weights to the point cloud data of each frame and then performs a weighted average calculation to obtain the fused 3D point cloud dataset. This dataset can comprehensively and accurately reflect the spatial morphology and positional changes of the target imaging cable during its swinging process.
[0088] Step S203: Using a three-dimensional reconstruction algorithm, generate three-dimensional reconstruction data of the swing process of the target imaging cable based on the three-dimensional point cloud dataset. The three-dimensional reconstruction data is used to reflect the spatial shape and positional changes of the target imaging cable during the swing process.
[0089] Among them, the three-dimensional reconstruction algorithm can adopt the directed distance field reconstruction algorithm. The directed distance field assigns a directed distance value to each point in the three-dimensional space. This value represents the shortest distance from the point to the surface of the target imaging cable and its direction, thereby accurately describing the spatial shape of the cable during the swing process.
[0090] Specifically, for each point in the 3D point cloud dataset, its directed distance to the cable surface is calculated. These distance values constitute a continuous directed distance field. This directed distance field is used to reconstruct the 3D structure of each point, dividing the reconstruction space into several equal cubic spaces, each a voxel. This allows for the reconstruction of a 3D surface model of the target imaging cable during its swinging motion. This model not only includes the cable's geometric shape information but also reflects its dynamic changes during the swinging process. This 3D reconstruction data provides a more accurate and reliable foundation for subsequent swing trajectory prediction and motion control of the imaging robot.
[0091] In some embodiments, the swing trajectory of the target imaging cable is predicted based on 3D reconstructed data to obtain several candidate predicted swing trajectory information, and the candidate predicted swing trajectory information with the highest confidence is selected as the target swing trajectory information from all candidate predicted swing trajectory information, including:
[0092] Step S301: Input the 3D reconstruction data into the pre-trained trajectory prediction model, so that the pre-trained trajectory prediction model outputs several candidate predicted swing trajectory information, wherein each candidate predicted swing trajectory information includes the predicted swing trajectory and the corresponding confidence level; wherein the pre-trained trajectory prediction model is obtained by training the machine learning network with 3D reconstruction data samples of imaging cables of historical swing trajectories.
[0093] The pre-trained trajectory prediction model can employ various machine learning network structures, such as recurrent neural networks, gated recurrent units, or convolutional neural networks. These network structures can effectively process time-series data or data with spatial characteristics, thereby learning the inherent patterns and characteristics of the target imaging cable's swaying trajectory. During training, a large number of historical swaying trajectory imaging cable 3D reconstruction data samples need to be collected and labeled to indicate the true swaying trajectory information. Then, these labeled samples are input into the machine learning network for training. By continuously adjusting the network parameters, the error between the network's prediction results and the true labeled results is gradually reduced, ultimately yielding the trajectory prediction model.
[0094] When the current 3D reconstruction data is input into the pre-trained trajectory prediction model, the model will output several candidate predicted swing trajectory information based on its learned knowledge and experience. Each candidate information contains the predicted swing trajectory and the corresponding confidence level, so that the optimal target swing trajectory information, i.e. the target swing trajectory information, can be selected from them later.
[0095] Step S302: Sort all candidate predicted swing trajectory information in descending order of confidence level, and select the candidate predicted swing trajectory information with the highest confidence level as the target swing trajectory information; wherein, the target swing trajectory information is used to represent the swing trajectory of the target imaging cable in the future time period.
[0096] After acquiring all candidate predicted oscillation trajectory information, they are sorted in descending order based on the confidence level of each candidate. Confidence level is a key indicator of the reliability of the prediction result; a higher confidence level indicates a greater likelihood that the predicted trajectory matches the actual situation. This sorting method allows for the rapid identification of the candidate predicted oscillation trajectory information with the highest confidence level. Subsequently, the candidate information with the highest confidence level is selected as the target oscillation trajectory information.
[0097] In some embodiments, motion control information for the target imaging robot is generated based on the target predicted swing trajectory information, and the motion pose of the target imaging robot is adjusted according to the motion control information. The motion control information is used to control the pose of the target imaging robot, including:
[0098] Step S401: Match multiple point poses of the target imaging robot in a preset trajectory pose mapping library based on the target predicted swing trajectory information; wherein, the preset trajectory pose mapping library contains the correspondence between various swing trajectories and the point poses of the imaging robot.
[0099] Point poses refer to the poses of multiple points on the target imaging robot during its motion, and these point poses constitute the robot's motion path. The pre-defined trajectory pose mapping library is established by collecting and organizing the optimal point poses of the imaging robot under a large number of different swing trajectories, covering the correspondence between various possible swing trajectories and the corresponding point poses of the imaging robot. When the predicted swing trajectory information of the target is input, a search and matching process is performed in the trajectory pose mapping library to quickly find multiple corresponding point poses.
[0100] Step S402: Based on the poses of multiple points of the target imaging robot, obtain the change angle of the poses of adjacent points.
[0101] After acquiring the poses of multiple points on the target imaging robot, a detailed analysis of the poses of adjacent points is performed. By calculating the spatial vectors between the poses of adjacent points and using the vector angle formula, the change angle of the poses of adjacent points is accurately determined. This change angle reflects the magnitude and direction of the attitude adjustment required for the target imaging robot to move from one point to the next.
[0102] Step S403: Determine the average position change angle based on the change angles of all point poses.
[0103] This process involves summarizing the changes in pose angles of all adjacent points and calculating the arithmetic mean to precisely determine the average change angle. This average change angle comprehensively reflects the overall trend and average magnitude of the target imaging robot's posture adjustment throughout the entire motion process. This allows for further optimization of the motion control strategy, ensuring that the target imaging robot can move with a more stable and precise posture when tracking the swing trajectory of the target imaging cable.
[0104] Step S404: Based on the sliding window and the average point change angle, smooth the poses of multiple points to obtain the poses of points with a point change angle less than the average point change angle, and use this as the smoothed poses.
[0105] This method involves setting a fixed-size sliding window that slides sequentially across a sequence of point poses. Within each window, the angle of change in the pose of the included points is analyzed. The pose of the points within each window is then adjusted based on the average angle of change in these angles.
[0106] Specifically, the angle of change of pose of a point within the window is compared with the average angle of change of pose of the point. If the angle of change of pose of a certain point is greater than the average angle of change of pose of the point, the pose of that point is adjusted appropriately to reduce its angle of change and bring it closer to the average angle of change of pose. After sliding the window across the entire pose sequence of points, the poses of points with angles of change of pose less than the average angle of change of pose are finally obtained. These poses are the smoothed poses. The smoothed poses can effectively reduce the abrupt changes in attitude of the target imaging robot during its movement, making the motion trajectory of the target imaging robot smoother and more continuous, improving imaging quality and tracking accuracy, thereby improving the stability and imaging quality of the imaging robot when tracking the swinging trajectory of the target imaging cable.
[0107] Step S405: Generate motion control information for the target imaging robot based on the smoothed point pose, and adjust the motion pose of the target imaging robot based on the motion control information.
[0108] Specifically, based on the smoothed point-position pose, specific command information for the motion joints of the target imaging robot is generated, which can precisely control the robot's posture movement. This command information includes the robot's turning angle. Subsequently, this motion control information is transmitted in real time to the target imaging robot's motion control system. The control system, based on the received commands, precisely adjusts each motion joint and actuator of the target imaging robot, thereby achieving precise control of its motion pose. This ensures that the target imaging robot moves continuously with the smoothed point-position pose, closely tracking the swing trajectory of the target imaging cable to acquire high-quality imaging data.
[0109] Based on the same inventive concept, this application also provides a motion control system for an imaging robot that implements the motion control method of the imaging robot described above.
[0110] The solution provided by this system is similar to the solution described in the above method. Therefore, the specific limitations of one or more motion control system embodiments of imaging robots provided below can be found in the limitations of the motion control method of imaging robots described above, and will not be repeated here.
[0111] like Figure 3 As shown, this application embodiment provides a motion control system for an imaging robot, including:
[0112] Image acquisition module 100 is used to acquire multiple frames of depth images of the target imaging cable during the swinging process, forming a depth image sequence;
[0113] The 3D reconstruction module 200 is used to generate point cloud data based on the depth image sequence, and to perform 3D reconstruction of the swing process of the target imaging cable based on the point cloud data to obtain 3D reconstruction data.
[0114] The swing trajectory prediction module 300 is used to predict the swing trajectory of the target imaging cable based on the three-dimensional reconstruction data, obtain several candidate swing trajectory information, and select the candidate swing trajectory information with the highest confidence from all candidate swing trajectory information as the target swing trajectory information.
[0115] The motion control module 400 is used to generate motion control information for the target imaging robot based on the target predicted swing trajectory information, and to adjust the motion pose of the target imaging robot according to the motion control information. The motion control information is used to control the pose of the target imaging robot.
[0116] In some embodiments, the system further includes an outlier filtering module, used for:
[0117] Based on the depth image sequence, construct the pixel matrix of each depth image; where each element in the pixel matrix corresponds to the depth value of each pixel in the depth image;
[0118] The confidence and orientation information of each pixel in the pixel matrix are extracted using the Sobel detection operator.
[0119] The confidence level of each pixel in the pixel matrix is compared with a preset edge pixel confidence level threshold, and multiple pixels with a confidence level greater than the preset edge pixel confidence level threshold are identified as edge region pixels.
[0120] For each edge region pixel, a neighborhood region of a preset size is divided with the edge region pixel as the center. Within the neighborhood region, the number of other edge region pixels with the same orientation information as the edge region pixel is counted. If the count of other edge region pixels is less than a preset threshold, the edge region pixel is determined to be an isolated point, and the isolated point is filtered and smoothed.
[0121] In some embodiments, the system further includes an edge filtering module, configured to:
[0122] Edge information of each depth image in a depth image sequence is extracted based on edge detection operators;
[0123] Edge information is refined based on nonmaximum suppression to obtain edge refinement information;
[0124] The depth image is thinned based on the edge thinning information, and then filtered and denoised.
[0125] In some embodiments, the 3D reconstruction module 200 is used for:
[0126] Using a preset point cloud generation algorithm, each frame of depth image in the depth image sequence is converted into corresponding point cloud data;
[0127] The point cloud data of all frames are registered and fused to obtain a three-dimensional point cloud dataset that represents the swinging process of the target imaging cable.
[0128] Using a 3D reconstruction algorithm, 3D reconstruction data of the swinging process of the target imaging cable is generated based on a 3D point cloud dataset. The 3D reconstruction data is used to reflect the spatial morphology and positional changes of the target imaging cable during the swinging process.
[0129] In some embodiments, the swing trajectory prediction module 300 is used for:
[0130] The 3D reconstructed data is input into a pre-trained trajectory prediction model, which outputs several candidate predicted swing trajectories. Each candidate predicted swing trajectory includes the predicted swing trajectory and the corresponding confidence level. The pre-trained trajectory prediction model is obtained by training a machine learning network with 3D reconstructed data samples of imaging cables of historical swing trajectories.
[0131] All candidate predicted swing trajectory information is sorted in descending order of confidence level, and the candidate predicted swing trajectory information with the highest confidence level is selected as the target swing trajectory information; wherein, the target swing trajectory information is used to represent the swing trajectory of the target imaging cable in the future time period.
[0132] In some embodiments, the motion control module 400 is used for:
[0133] Based on the target's predicted swing trajectory information, multiple point poses of the target imaging robot are matched in a preset trajectory pose mapping library; wherein, the preset trajectory pose mapping library contains the correspondence between various swing trajectories and the point poses of the imaging robot.
[0134] Based on the poses of multiple points of the target imaging robot, obtain the angle of change of pose of adjacent points;
[0135] Determine the average angle of change of position based on the angle of change of all position poses;
[0136] The poses of multiple points are smoothed by combining the sliding window with the average point change angle. The point poses with a point change angle less than the average point change angle are used as the smoothed point poses.
[0137] The motion control information of the target imaging robot is generated based on the smoothed point pose, and the motion pose of the target imaging robot is adjusted based on the motion control information.
[0138] like Figure 4 As shown, this application provides an electronic device. The electronic device 10 includes a memory 20 and a processor 30. The memory 20 stores a computer program. When the computer program is executed by the processor 30, the processor 30 performs the steps of the motion control method for the imaging robot as described in the above embodiment.
[0139] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed, implements the steps of the motion control method for the imaging robot as described in the above embodiments.
[0140] This application provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, wherein when the program instructions are executed by a computer, the computer performs the steps of the motion control method for the imaging robot as described in the above embodiments.
[0141] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, electronic devices, computer storage media, and computer program products described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0142] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products or devices.
[0143] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0144] In the several embodiments provided by this invention, it should be understood that the disclosed systems, electronic devices, computer storage media, computer program products, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.
[0145] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0146] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0147] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for executing all or part of the steps of the methods described in the various embodiments of the present invention through a computer device (which may be a personal computer, a server, or a network device, etc.). The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.
[0148] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A motion control method for an imaging robot, characterized in that, include: Multiple depth images of the target imaging cable during its swinging process are acquired to form a depth image sequence; Point cloud data is generated based on the depth image sequence, and the swinging process of the target imaging cable is reconstructed in three dimensions based on the point cloud data to obtain three-dimensional reconstructed data. Based on the three-dimensional reconstruction data, the swing trajectory of the target imaging cable is predicted to obtain several candidate swing trajectory information, and the candidate swing trajectory information with the highest confidence is selected from all the candidate swing trajectory information as the target swing trajectory information. Motion control information for the target imaging robot is generated based on the target predicted swing trajectory information, and the motion pose of the target imaging robot is adjusted according to the motion control information, wherein the motion control information is used to control the pose of the target imaging robot.
2. The motion control method for the imaging robot according to claim 1, characterized in that, The process of generating point cloud data based on the depth image sequence, and then performing three-dimensional reconstruction of the swinging process of the target imaging cable based on the point cloud data to obtain three-dimensional reconstruction data, further includes: Based on the depth image sequence, a pixel matrix is constructed for each depth image; wherein each element in the pixel matrix corresponds to the depth value of each pixel in the depth image; The confidence score and orientation information of each pixel in the pixel matrix are extracted using the Sobel detection operator. The confidence level of each pixel in the pixel matrix is compared with a preset edge pixel confidence threshold, and multiple pixels with a confidence level greater than the preset edge pixel confidence threshold are identified as edge region pixels. For each edge region pixel, a neighborhood region of a preset size is divided with the edge region pixel as the center. Within the neighborhood region, the number of other edge region pixels with the same direction information as the edge region pixel is counted. If the count of other edge region pixels is less than a preset threshold, the edge region pixel is determined to be an isolated point, and the isolated point is subjected to filtering and smoothing processing.
3. The motion control method for the imaging robot according to claim 1 or 2, characterized in that, The process of generating point cloud data based on the depth image sequence, and then performing three-dimensional reconstruction of the swinging process of the target imaging cable based on the point cloud data to obtain three-dimensional reconstruction data, further includes: Edge information of each depth image in the depth image sequence is extracted based on the edge detection operator; The edge information is refined based on nonmaximum suppression to obtain edge refinement information; The depth image is thinned according to the edge thinning information, and then filtered and denoised.
4. The motion control method for the imaging robot according to claim 1, characterized in that, The step of generating point cloud data based on the depth image sequence, and performing three-dimensional reconstruction of the swing process of the target imaging cable based on the point cloud data to obtain three-dimensional reconstruction data includes: Using a preset point cloud generation algorithm, each frame of the depth image in the depth image sequence is converted into corresponding point cloud data; The point cloud data of all frames are registered and fused to obtain a three-dimensional point cloud dataset representing the swinging process of the target imaging cable. Using a 3D reconstruction algorithm, 3D reconstruction data of the swinging process of the target imaging cable is generated based on the 3D point cloud dataset. The 3D reconstruction data is used to reflect the spatial morphology and positional changes of the target imaging cable during the swinging process.
5. The motion control method for an imaging robot according to claim 1, characterized in that, The process of predicting the swing trajectory of the target imaging cable based on the three-dimensional reconstruction data yields several candidate predicted swing trajectory information. The candidate predicted swing trajectory information with the highest confidence level is then selected as the target swing trajectory information, including: The 3D reconstructed data is input into a pre-trained trajectory prediction model, which outputs several candidate predicted swing trajectories. Each candidate predicted swing trajectory includes the predicted swing trajectory and the corresponding confidence level. The pre-trained trajectory prediction model is obtained by training a machine learning network using 3D reconstructed data samples of imaging cables with historical swing trajectories. All candidate predicted swing trajectory information are sorted in descending order of confidence level, and the candidate predicted swing trajectory information with the highest confidence level is selected as the target swing trajectory information; wherein, the target swing trajectory information is used to represent the swing trajectory of the target imaging cable in the future time period.
6. The motion control method for an imaging robot according to claim 1 or 5, characterized in that, The process involves generating motion control information for the target imaging robot based on the predicted swing trajectory information, and adjusting the motion pose of the target imaging robot according to the motion control information. The motion control information is used to control the pose of the target imaging robot, including: Based on the target predicted swing trajectory information, multiple point poses of the target imaging robot are matched in a preset trajectory pose mapping library; wherein, the preset trajectory pose mapping library contains the correspondence between various swing trajectories and the point poses of the imaging robot. Based on the poses of multiple points of the target imaging robot, the change angle of the poses of adjacent points is obtained; Determine the average angle of change of position based on the angle of change of all position poses; Based on the sliding window and the average point change angle, the poses of multiple points are smoothed to obtain the poses of points where the point change angle is less than the average point change angle, which is then used as the smoothed poses of points. The motion control information of the target imaging robot is generated based on the smoothed point pose, and the motion pose of the target imaging robot is adjusted based on the motion control information.
7. A motion control system for an imaging robot, characterized in that, include: The image acquisition module is used to acquire multiple frames of depth images of the target imaging cable during its swinging process, forming a depth image sequence; The three-dimensional reconstruction module is used to generate point cloud data based on the depth image sequence, and to perform three-dimensional reconstruction of the swing process of the target imaging cable based on the point cloud data to obtain three-dimensional reconstruction data. The swing trajectory prediction module is used to predict the swing trajectory of the target imaging cable based on the three-dimensional reconstruction data, obtain several candidate predicted swing trajectory information, and select the candidate predicted swing trajectory information with the highest confidence from all the candidate predicted swing trajectory information as the target swing trajectory information. A motion control module is used to generate motion control information for a target imaging robot based on the target predicted swing trajectory information, and to adjust the motion pose of the target imaging robot according to the motion control information, wherein the motion control information is used to control the pose of the target imaging robot.
8. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, causes the processor to perform the steps of the motion control method for the imaging robot as described in any one of claims 1-6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the steps of the motion control method for the imaging robot as described in any one of claims 1-6.
10. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, wherein when the program instructions are executed by a computer, the computer performs the steps of the motion control method for the imaging robot as described in any one of claims 1-6.