Robot guiding method and system based on 3D vision

Abstract

The invention discloses a robot guiding method and system based on 3D vision, and the method comprises the steps: obtaining multi-view image data of a plurality of robots, and carrying out the analysis and processing of the multi-view image data to obtain initial position information; according to the initial position information, extracting position coordinates to obtain a three-dimensional model; analyzing spatial geometry and motion characteristics according to the three-dimensional model to obtain a motion track; comparing the motion tracks of the robots, if the motion parameters are consistent with preset parameters, generating a task allocation candidate set, and if the motion parameters are not consistent with the preset parameters, analyzing the sensing data of the multiple robots in real time, and correcting the motion parameters; carrying out constraint and verification on the corrected motion parameters, generating a collision-free path of each robot, and obtaining a cooperative action sequence; sending the coordination action sequence to each robot, and analyzing the real-time acquisition task execution state of each robot to obtain real-time task execution feedback information; according to the feedback information, environment obstacle information is obtained, the three-dimensional model is updated, and the updated three-dimensional model and the motion trail are obtained.

Description

Technical Field

[0001] This invention relates to the field of industrial vision technology, and in particular to a robot guidance method and system based on 3D vision. Background Technology

[0002] Currently, in fields such as intelligent manufacturing, logistics warehousing, and medical assistance, industrial vision technology serves as the core support for autonomous robot operation and collaborative control. Its performance directly determines the system's perception accuracy and decision-making efficiency. While the rapid development of visual intelligence has driven progress in robot vision guidance technology, existing solutions still have significant limitations in complex and dynamic scenarios.

[0003] In existing technologies, most solutions rely on single-viewpoint or localized information processing of environmental data, making it difficult to adapt to complex dynamic environments. For example, a single robot acquires target images and performs localization and tracking using its own onboard visual sensors. In multi-robot collaborative scenarios, each robot often perceives the environment independently, lacking a unified information integration and sharing mechanism. Furthermore, existing technologies struggle to handle complex dynamic scenarios with multiple targets and angles. Single-view data, due to its limited perspective, is prone to missing key details, leading to inaccurate target localization and insufficient completeness of the constructed 3D target model. In multi-robot collaborative operations, information isolation results in significant deviations in motion trajectory prediction, easily causing action conflicts or even task failure. Multi-view data fusion suffers from information redundancy and inconsistencies, and sensor noise and cross-viewpoint observation errors further reduce the accuracy of target perception, hindering the improvement of overall system performance.

[0004] In summary, existing technologies suffer from poor performance of robot guidance systems in complex dynamic environments. Summary of the Invention

[0005] This invention provides a robot guidance method and system based on 3D vision to solve the problem of poor performance of robot guidance systems in complex dynamic environments in the prior art.

[0006] In a first aspect, to solve the above-mentioned technical problems, the present invention provides a robot guidance method based on 3D vision, comprising: Acquire multi-view image data from multiple robots, analyze and process the multi-view image data to obtain preliminary position information; Based on the preliminary location information and combined with the multi-view image data, the robot's location information is extracted to obtain a three-dimensional model; The spatial geometric and motion characteristics of the three-dimensional model are analyzed to obtain the motion trajectory; The motion trajectories of multiple robots are compared to obtain the motion parameters of each robot; If the motion parameters are consistent with the preset motion parameters, then the motion trajectory is combined to generate a task assignment candidate set, and the task instructions for each robot are obtained. If the motion parameters are deviated, the perception data of multiple robots are analyzed in real time to correct the motion parameters and obtain the corrected motion parameters. The corrected motion parameters are constrained and verified to generate collision-free paths for each robot, resulting in a cooperative motion sequence. The coordinated action sequence is sent to each robot, and the real-time task execution status of each robot is analyzed to obtain real-time task execution feedback information. Based on the real-time task execution feedback information, environmental obstacle information is obtained, the three-dimensional model is updated, and the updated three-dimensional model and motion trajectory are obtained.

[0007] Preferably, the step of acquiring multi-view image data from multiple robots and analyzing and processing the multi-view image data to obtain preliminary position information includes: Acquire multi-view image data from multiple robots and perform timestamp calibration to obtain a synchronized multi-view image set; For the synchronized multi-view image set, each frame of the image is extracted and processed to obtain the feature point set of each view; If the number of feature points in each viewpoint is greater than a preset threshold, then visual tracking is performed on the feature points in each viewpoint to obtain preliminary position information.

[0008] Preferably, the step of extracting the robot's position information and obtaining a 3D model based on the preliminary position information and the multi-view image data includes: Acquire the robot's current pose data, correct the multi-view image data, and generate a first dataset containing the preliminary position information; Extract the robot's first edge features, first texture features and first key points from the first dataset, and combine them with the robot's current pose data to calculate the robot's coordinates in the coordinate system to obtain the first coarse spatial coordinates. If the first coarse spatial coordinates are deviated under multiple perspectives, the first key point is matched by fusing the multi-view image data to obtain the second optimized spatial coordinates. Error observation and data iteration are performed on the second optimized spatial coordinates to obtain the target three-dimensional model.

[0009] Preferably, the step of analyzing the spatial geometric features and motion features of the three-dimensional model to obtain the motion trajectory includes: Spatial geometric features of the centroid, principal axis of inertia, and curvature change point, as well as motion features of velocity, acceleration, and angular velocity, are extracted from the three-dimensional model to form a multi-dimensional state vector sequence. If the time series length of the multidimensional state vector sequence exceeds a preset time series length threshold, the time series length of the multidimensional state vector sequence is readjusted to obtain the first predicted trajectory data. Based on the first predicted trajectory data, trajectory features of continuous time periods are extracted and fitted to obtain the motion trajectory.

[0010] Preferably, if the motion parameters are consistent with preset motion parameters, then a task assignment candidate set is generated by combining the motion trajectory to obtain the task instructions for each robot, including: If the deviation of the motion parameters between multiple robots is within a preset deviation threshold, then the data timestamp is calibrated to generate a consistent set of motion parameters. Based on the consistent motion parameter set and the relative position of the motion trajectory, a task assignment candidate set is generated; Based on the task allocation candidate set and the priority sorting rules, a task instruction set for each robot is generated.

[0011] Preferably, if the motion parameters have deviations, the process of analyzing the perception data of multiple robots in real time to correct the motion parameters and obtain corrected motion parameters includes: Acquire perception data from multiple robots, calibrate the data acquisition timestamps of each robot, and generate a time-aligned perception data set; Based on the time-aligned perception data set, combined with the perception data of the multiple robots and the preset robot capability weights, the corrected target motion parameters are obtained using a weighted average method.

[0012] Preferably, the step of constraining and verifying the corrected motion parameters to generate collision-free paths for each robot and obtain a cooperative action sequence includes: Based on the corrected motion parameters and combined with the environmental map, the first path set is obtained by adjusting waypoints to reduce the motion distance. Based on the first path set, and using the environmental map and the location of dynamic obstacles, the path points are readjusted to obtain a collision-free second path set; For the second set of collision-free paths, the speed and acceleration are checked. If they exceed the preset speed and acceleration thresholds, the path points and action times are adjusted to obtain a third set of paths that meet the constraints. Based on the third path set, collision-free paths for each robot are generated, resulting in the cooperative action sequence.

[0013] Preferably, the step of sending the coordinated action sequence to each robot and analyzing the real-time task execution status of each robot to obtain real-time task execution feedback information includes: Decompose the cooperative action sequence, distribute task instructions to each robot, and obtain the first task sequence received by each robot; Based on the first task sequence, ensure that the timing of each robot's actions is consistent to obtain the first action coordination data; Based on the first motion coordination data, the task execution status of each robot is collected in real time, the status data is updated, compared with the preset motion optimization rules, the task completion rate is judged, and real-time task execution feedback information is obtained.

[0014] Preferably, the step of obtaining environmental obstacle information based on the real-time task execution feedback information, updating the 3D model, and obtaining the updated 3D model and motion trajectory includes: Based on the real-time task execution feedback information, the deviation data between the robot position and the preset robot position is extracted, and the deviation value is calculated through the coordinate difference. The system uses sensors to scan the dynamic environment, extracts obstacle information, and updates the environmental state to obtain the environmental obstacle information. Based on the deviation value and the environmental obstacle information, the three-dimensional model is reconstructed to obtain the updated three-dimensional model. The motion trajectory is obtained based on the updated 3D model and the deviation value.

[0015] Secondly, the present invention provides a robot guidance system based on 3D vision, comprising: The multi-view data processing module is used to acquire multi-view image data from multiple robots, analyze and process the multi-view image data, and obtain preliminary position information. The 3D modeling module is used to extract the robot's position information based on the preliminary position information and the multi-view image data, and obtain a 3D model. The trajectory prediction module is used to analyze the spatial geometric features and motion features of the three-dimensional model to obtain the motion trajectory; The parameter fusion module is used to compare the motion trajectories of multiple robots to obtain the motion parameters of each robot. The task allocation module is used to generate a task allocation candidate set by combining the motion parameters with the preset motion parameters if the motion parameters are consistent with the preset motion parameters, so as to obtain the task instructions for each robot. The parameter correction module is used to analyze the perception data of multiple robots in real time and correct the motion parameters if there is a deviation in the motion parameters, so as to obtain the corrected motion parameters. The path planning module is used to constrain and verify the corrected motion parameters, generate collision-free paths for each robot, and obtain a cooperative action sequence. The execution feedback module is used to send the cooperative action sequence to each robot, analyze the real-time task execution status of each robot, and obtain real-time task execution feedback information. The model update module is used to obtain environmental obstacle information based on the real-time task execution feedback information, update the three-dimensional model, and obtain the updated three-dimensional model and motion trajectory.

[0016] Thirdly, the present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement a 3D vision-based robot guidance method as described in any one of the preceding claims.

[0017] Fourthly, the present invention also provides a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute any one of the above-described 3D vision-based robot guidance methods.

[0018] Compared with the prior art, the present invention has the following beneficial effects: (1) This invention breaks through the information limitations of a single robot or single perspective by acquiring and analyzing multi-view image data from multiple robots; the fusion of multi-view data can cover a wider range of environments and reduce blind spots. At the same time, through timestamp calibration and feature point tracking, the synchronization and accuracy of the initial position information are ensured, solving the positioning deviation problem caused by limited perspective in the prior art. (2) This invention combines preliminary position information with multi-view image data to extract robot position information, and generates a high-precision three-dimensional model through pose correction, feature extraction and coordinate optimization; it not only integrates the complementarity of multi-source data, but also solves the problem of missing or distorted three-dimensional features in single-view modeling through error observation and iterative optimization, so that the model can more realistically reflect the spatial state of the robot and provide a reliable basis for trajectory analysis. (3) This invention compares the motion trajectories of multiple robots to extract motion parameters, thereby realizing the quantitative evaluation of the motion state between robots; it breaks the limitations of the existing technology where multiple robots perceive independently and lack a unified parameter benchmark. Through the horizontal comparison of trajectory features, it can accurately identify the motion differences of each robot (such as speed deviation and path offset), providing a clear basis for subsequent task allocation or parameter correction; when the motion parameters are consistent with the preset, it generates a task allocation candidate set in combination with the trajectory to ensure the matching of task instructions with robot motion state; compared with the fixed task allocation mode in the existing technology, this method can dynamically adjust the task division according to real-time trajectory features (such as position and speed) (such as robots closer to the target are given priority to perform tasks), improving the rationality of task allocation and execution efficiency; if there is a deviation in motion parameters, it can be corrected in real time by analyzing the perception data of multiple robots, solving the parameter drift problem caused by single robot sensor error or environmental interference; (4) The present invention updates the three-dimensional model and trajectory based on feedback information, realizing the adaptive adjustment of the system to environmental changes. In the prior art, the model is often fixed and difficult to cope with dynamic obstacles. However, by integrating environmental obstacle information in real time and dynamically updating the model and trajectory, the robot can continuously adapt to environmental changes, solving the problem of poor performance of robot guidance system in complex dynamic environments. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of a robot guidance method based on 3D vision provided in the first embodiment of the present invention; Figure 2 This is a schematic diagram of a robot guidance device based on 3D vision provided in the second embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Reference Figure 1 The first embodiment of the present invention provides a robot guidance method based on 3D vision, comprising the following steps: S11, acquire multi-view image data of multiple robots, analyze and process the multi-view image data to obtain preliminary position information; S12, Based on the preliminary position information and combined with the multi-view image data, extract the robot's position information to obtain a three-dimensional model; S13, Analyze the spatial geometric features and motion features of the three-dimensional model to obtain the motion trajectory; S14, compare the motion trajectories of multiple robots to obtain the motion parameters of each robot; S15, if the motion parameters are consistent with the preset motion parameters, then the motion trajectory is combined to generate a task assignment candidate set to obtain the task instructions for each robot; S16, If the motion parameters have a deviation, the perception data of multiple robots are analyzed in real time to correct the motion parameters and obtain the corrected motion parameters. S17, constrain and verify the corrected motion parameters, generate collision-free paths for each robot, and obtain a cooperative action sequence; S18, the cooperative action sequence is sent to each robot, the real-time task execution status of each robot is analyzed, and real-time task execution feedback information is obtained; S19, based on the real-time task execution feedback information, obtain environmental obstacle information, update the three-dimensional model, and obtain the updated three-dimensional model and motion trajectory.

[0022] In step S11, multi-view image data of multiple robots are acquired, and the multi-view image data is analyzed and processed to obtain preliminary position information.

[0023] In one feasible approach, acquiring multi-view image data from multiple robots and analyzing and processing the multi-view image data to obtain preliminary position information includes: Acquire multi-view image data from multiple robots and perform timestamp calibration to obtain a synchronized multi-view image set; For the synchronized multi-view image set, each frame of the image is extracted and processed to obtain the feature point set of each view; If the number of feature points in each viewpoint is greater than a preset threshold, then visual tracking is performed on the feature points in each viewpoint to obtain preliminary position information.

[0024] It should be noted that multiple robots acquire multi-view image data by being equipped with vision sensors. The sensor synchronization mechanism automatically aligns with pre-set timestamps, and a timestamp matching method is used to calibrate the image acquisition time, resulting in a synchronized multi-view image set. The multi-view image data refers to a collection of target images acquired by multiple robots from different spatial angles, obtained by the robot's vision sensors, and includes raw visual information such as the target's shape, texture, and spatial position. The vision sensors refer to devices such as high-definition cameras on the robots, used to capture two-dimensional image information of the target. Their acquisition rate is typically 30 frames per second, ensuring data continuity. The timestamp matching method involves setting a reasonable time deviation threshold to filter image frames acquired by multiple robots at approximately the same time, achieving time calibration. The timestamp matching method operates by using a unified reference time. Each robot's vision sensor timestamps each captured image frame with millisecond precision. A pre-defined, extremely small time window (e.g., ±5 milliseconds) is used as the standard for judging time synchronization. The timestamps of all robot images are iterated, and image frames whose timestamps fall within the pre-defined time window are identified as "synchronized frames." These frames are then combined into a multi-view image set. For the synchronized multi-view image set, a distributed processing framework is used to distribute image processing tasks to various computing nodes. Each node is responsible for processing image data from one viewpoint. Edge detection and feature point extraction algorithms are used to extract and process each image frame, obtaining the feature point set for each viewpoint. Edge detection here uses the Canny algorithm. By setting a low threshold of 50 and a high threshold of 150, the contour edges of the target object can be accurately extracted. The low threshold is used to detect weak edges, and the high threshold is used to preserve strong edges. The combination of the two achieves complete contour extraction. Feature point extraction uses the SIFT algorithm, which can generate a 128-dimensional feature descriptor with scale invariance. About 200 feature points (such as corner points of goods, texture abrupt change points, etc.) can be extracted from each frame of image. These feature point sets contain the key spatial information of the target and are the core basis for subsequent localization.

[0025] If the number of feature points in each viewpoint exceeds a preset threshold (e.g., 100 feature points), a Kalman filter algorithm is used to visually track the feature points in each viewpoint to determine the initial two-dimensional coordinates of the target in each viewpoint. For example, for a moving target (such as a marker point on an autonomous vehicle), the Kalman filter smooths the trajectory of the feature points through prediction and update steps, outputting the initial two-dimensional coordinates for each viewpoint, such as (120, 80) for viewpoint 1, (130, 85) for viewpoint 2, and (125, 90) for viewpoint 3, effectively reducing noise interference and improving tracking stability. Then, based on the initial two-dimensional coordinates, weights are assigned according to the quality of the images from each viewpoint (e.g., sharpness, feature point stability, sensor accuracy), with higher-quality views receiving larger weights. The initial two-dimensional coordinates (x, y) of each viewpoint are multiplied by their corresponding weights and summed to obtain the fused initial coordinates. Since there may be slight deviations in the multi-view coordinates (e.g., sensor noise, calculation errors), the coordinate results are optimized using the least squares method, iteratively adjusted to minimize the error, and the initial position information of the target is obtained.

[0026] In step S12, based on the preliminary position information and combined with the multi-view image data, the robot's position information is extracted to obtain a three-dimensional model.

[0027] In one feasible approach, the step of extracting the robot's position information and obtaining a 3D model based on the preliminary position information and the multi-view image data includes: Acquire the robot's current pose data, correct the multi-view image data, and generate a first dataset containing the preliminary position information; Extract the robot's first edge features, first texture features and first key points from the first dataset, and combine them with the robot's current pose data to calculate the robot's coordinates in the coordinate system to obtain the first coarse spatial coordinates. If the first coarse spatial coordinates are deviated under multiple perspectives, the first key point is matched by fusing the multi-view image data to obtain the second optimized spatial coordinates. Error observation and data iteration are performed on the second optimized spatial coordinates to obtain the target three-dimensional model.

[0028] It should be noted that the robot's current pose data is collected from multiple perspectives using a high-resolution visual sensor. The robot's IMU (Inertial Measurement Unit) and GPS modules are then used. The IMU is used to obtain the robot's motion state, such as acceleration and angular velocity, while GPS (Global Positioning System) provides geographical coordinates. The combination of the two yields the robot's attitude angles and position coordinates, which are the core sources of pose data with an accuracy of approximately 0.05 meters. Based on this, the multi-view image data is corrected. A checkerboard calibration board can be used for calibration. A 10x10 cm grid is pre-arranged in the experimental environment. Through camera intrinsic and extrinsic parameter calibration, lens distortion and viewing angle deviation are eliminated, thereby generating the first dataset containing preliminary position information. When extracting the robot's first edge features, first texture features, and first keypoints from the first dataset, the Canny operator is used to detect the first edge features. The image is first Gaussian smoothed, with low thresholds of 50 and high thresholds of 150 to extract clear boundaries of the robot's outline. The first texture features are analyzed using a Gabor filter, with the filter direction adjusted to 0°, 45°, and 90° to extract texture features from the robot's surface, enhancing its recognizability. The first keypoints, such as the robot's corner points, are extracted using the SIFT algorithm. Combined with the robot's current pose data, coordinate transformation maps the robot's local motion state to a unified three-dimensional Cartesian coordinate system, calculating the robot's coordinates in this system to obtain the first coarse spatial coordinates. The robot's pose data includes position information and attitude information. Position information typically represents the robot's relative displacement within its local coordinate system (with the robot's base as the origin), such as moving 2.5 meters along the X-axis, 1.3 meters along the Y-axis, and 0.5 meters along the Z-axis (height). Attitude information indicates the robot's spatial orientation, reflecting whether the robot is tilted or rotating. The robot's local pose is converted into coordinates in a three-dimensional Cartesian coordinate system through two steps: rotation correction and translation superposition. Since the robot may tilt or rotate, the orientation of the robot's local coordinates must first be corrected to align with the coordinate axes of the three-dimensional Cartesian coordinate system. The robot's local coordinates are displacements relative to its initial position; these are then superimposed with its initial installation position in the three-dimensional Cartesian coordinate system to obtain the first coarse spatial coordinates.

[0029] When the initial coarse spatial coordinates calculated by multiple robots from different perspectives deviate, it is necessary to find matching keypoints in the multi-view images based on the first keypoint extracted by the SIFT algorithm. These matching keypoints refer to the imaging of the first keypoint from different perspectives. Using these matching keypoints, the 3D spatial coordinates of the keypoints are calculated using triangulation. Triangulation involves solving for the 3D spatial coordinates of the matching keypoints using geometric calculations based on the optical center coordinates of the two perspectives and the pixel coordinates of the keypoint. For example, if the optical center of perspective 1 is O1 and the optical center of perspective 2 is O2, and the pixel coordinates of the paired point in the two perspectives are p1 and p2 respectively, the 3D coordinates (x, y, z) of the keypoint are obtained by solving for the intersection of the two light rays O1p1 and O2p2. The 3D coordinates of all matching keypoints are then integrated to obtain the robot's second optimized spatial coordinates.

[0030] The second optimization of spatial coordinates may still contain errors. Therefore, a reprojection error assessment is used to evaluate accuracy. The 3D coordinates are back-projected onto images from various viewpoints, and the distance between the back-projected points and the actual pixels is calculated. If the distance is greater than a preset value, it is considered to have a significant error. By constructing an error function, and using the sum of the squares of the distances between the back-projected points and the actual pixels of all keypoints as the objective, the 3D coordinates of the keypoints are iteratively adjusted to minimize the error, thus obtaining the target 3D model.

[0031] In step S13, the spatial geometric features and motion features of the three-dimensional model are analyzed to obtain the motion trajectory; In one feasible approach, the analysis of the spatial geometric and motion features of the three-dimensional model to obtain the motion trajectory includes: Spatial geometric features of the centroid, principal axis of inertia, and curvature change point, as well as motion features of velocity, acceleration, and angular velocity, are extracted from the three-dimensional model to form a multi-dimensional state vector sequence. If the time series length of the multidimensional state vector sequence exceeds a preset time series length threshold, the time series length of the multidimensional state vector sequence is readjusted to obtain the first predicted trajectory data. Based on the first predicted trajectory data, trajectory features of continuous time periods are extracted and fitted to obtain the motion trajectory.

[0032] It should be noted that when extracting spatial geometric features from the 3D model, all key points constituting the robot's geometric structure are extracted. Weights are assigned to key points in different parts based on the robot's physical structural characteristics (reflecting the mass proportion of that part). For example, the key point weight for the main body is set to 0.7; the key point weight for lightweight components (such as sensor housings) is set to 0.1. The coordinates of the centroid are obtained by the weighted average of the coordinates of all key points. The principal axis of inertia is calculated using the point cloud covariance matrix. The 3D coordinates of all key points are extracted from the robot's 3D model to form a point cloud set. The average position of the spatial coordinates of all points in the point cloud set is taken to obtain the point cloud centroid. A covariance matrix of the point cloud set relative to the point cloud centroid is constructed. Eigenvalue decomposition is performed on the covariance matrix to obtain eigenvectors, which represent the principal axis of inertia. The eigenvalue with the largest value corresponds to the principal orientation. Curvature abrupt change points are obtained by detecting changes in the surface normal vector. When the curvature change exceeds 0.2, it is recorded, which can identify critical corners of the robot structure. Regarding motion features, velocity, acceleration, and angular velocity are collected by the robot's IMU sensors. For example, the velocity is 5 m / s, the acceleration is 0.3 m / s², and the angular velocity is 0.08 radians / s. Z-score standardization is applied to each feature dimension, converting features of different units into dimensionless data with a mean of 0 and a standard deviation of 1. After standardization, all features are on the same order of magnitude. Integrating these geometric and motion features forms a multidimensional state vector sequence. If the time series length of the multidimensional state vector sequence exceeds a preset threshold (e.g., 50 time steps), the length needs to be readjusted, which can be done using a bidirectional LSTM network. The bidirectional LSTM can capture the dependencies between sequences in both forward and backward directions. Combined with an attention mechanism, weights are assigned to each time step, such as assigning a weight of 0.5 to the acceleration phase and 0.2 to the steady-state phase, thus obtaining the first predicted trajectory data, such as predicting the robot's displacement trajectory for the next 3 seconds. Based on the first predicted trajectory data, the sliding window method can be used to extract trajectory features over a continuous time period. The window size is set to 1.5 seconds and the step size is 0.3 seconds. Features such as local curvature and velocity changes are extracted. Then, the trajectory curve is fitted using the least squares method, such as fitting a quadratic curve, to obtain the motion trajectory. This trajectory can provide a basis for robot path planning.

[0033] In step S14, the motion trajectories of multiple robots are compared to obtain the motion parameters of each robot.

[0034] In one feasible approach, comparing the motion trajectories of multiple robots to obtain motion parameters for each robot includes: The position coordinate sequence is obtained from the motion trajectory, the velocity parameters and acceleration parameters at each time point are calculated, and a first velocity parameter set and a first acceleration parameter set are generated. The first velocity parameter set and the first acceleration parameter set are transmitted between multiple robots, the data acquisition timestamps of each robot are calibrated, and a second velocity parameter set and a second acceleration parameter set are generated. Based on the second set of velocity parameters and the second set of acceleration parameters, the velocity parameters and acceleration parameters are calculated using the weighted average method to obtain the set of motion parameters.

[0035] It should be noted that when obtaining the position coordinate sequence from the target's motion trajectory, a high-precision positioning device combining GPS and LiDAR can be used to collect coordinate points every 0.1 seconds over a certain period of time, forming sequences such as (x=1.5 m, y=0.8 m) and (x=1.7 m, y=0.9 m). The velocity and acceleration parameters at each time point are calculated using a differential method. The velocity is obtained by dividing the position difference between adjacent coordinate points by the 0.1-second time interval; for example, the velocity of the two points mentioned above is approximately 2.24 m / s. The acceleration is calculated by dividing the velocity difference between adjacent time points by the time interval, thus generating a first set of velocity parameters and a first set of acceleration parameters. When transmitting the first set of velocity parameters and the first set of acceleration parameters between multiple robots, the data acquisition timestamps of each robot are calibrated using a network time protocol, controlling the timestamp error within 1 millisecond to eliminate data inconsistencies caused by time deviations, generating a second set of velocity parameters and a second set of acceleration parameters. Based on the second set of velocity parameters and the second set of acceleration parameters, a weighted average method is used to calculate the velocity and acceleration parameters. For example, given the speed data of three robots as 3 m / s, 3.2 m / s, and 2.8 m / s, weights of 0.4, 0.3, and 0.3 are assigned to each robot's sensor accuracy, respectively, to calculate a unified speed parameter of 3 m / s. The acceleration parameters are processed in the same way to obtain the final set of motion parameters.

[0036] In step S15, if the motion parameters are consistent with the preset motion parameters, then a task assignment candidate set is generated by combining the motion trajectory to obtain the task instructions for each robot.

[0037] In one feasible approach, if the motion parameters are consistent with preset motion parameters, then a task assignment candidate set is generated based on the motion trajectory to obtain task instructions for each robot, including: If the deviation of the motion parameters between multiple robots is within a preset deviation threshold, then the data timestamp is calibrated to generate a consistent set of motion parameters. Based on the consistent motion parameter set and the relative position of the motion trajectory, a task assignment candidate set is generated; Based on the task allocation candidate set and the priority sorting rules, a task instruction set for each robot is generated.

[0038] It should be noted that determining whether the motion parameters of multiple robots are consistent can be achieved by comparing the deviation range of each robot's parameters. For example, if the preset deviation threshold is 0.5 m / s, when the speed parameters of three robots are 8.0 m / s, 8.2 m / s, and 7.9 m / s respectively, the motion parameters can be considered consistent because the deviation is within the threshold. In this case, a high-precision clock module (GPS-disciplined oscillator) and a dedicated network card supporting hardware timestamps are used to provide a stable clock reference and reduce system latency. Based on the PTP protocol, the error is attenuated by segmenting the boundary clock, dynamically shortening the synchronization period to adapt to the movement state. Kalman filtering is used to predict and compensate for network latency in real time, and IMU data is combined to correct clock drift caused by motion, controlling the error within 2 milliseconds. This controls the speed parameters of each robot to 8.1 m / s, generating a consistent motion parameter set containing information such as uniform speed and acceleration. When generating a candidate set of task assignments based on the consistent motion parameter set and the relative positions of the motion trajectories, GPS positioning is used to determine the positions of each robot and the target. The GPS data is then parsed to obtain the latitude and longitude coordinates of both robots in the WGS84 coordinate system. This is converted to local Cartesian coordinates (X, Y) via UTM projection, and the coordinate difference is calculated to obtain the relative positions of each robot and the target. The target position is obtained from the motion trajectory. If the target is 100 meters ahead, robot 1 is 90 meters away, robot 2 is 110 meters away, and robot 3 is 105 meters away. This relative position information is included in the candidate set to provide a basis for task assignment. When generating the task instruction set for each robot based on the candidate set, a priority ranking rule must be considered. This rule can be formulated based on the distance between the robot and the target, the current resource occupancy rate, etc. For example, robot 1, being closest to the target and with sufficient resources, has the highest priority and is assigned the dominant tracking task, with instructions including maintaining a speed of 8.1 m / s to approach the target. Robots 2 and 3, with lower priorities, are assigned auxiliary following tasks, with instructions to maintain a safe following distance. The safe distance needs to be dynamically adjusted based on the robot speed and scene complexity, calculated using the following formula: . Current speed (unit: m / s) The static safety distance is 5 meters by default to avoid low-speed collisions; k is the scenario coefficient (1.2 for straight driving, 1.8 for complex scenarios such as intersections to compensate for reaction time).

[0039] In complex scenarios such as intersections, the priority of robots approaching the target can be temporarily increased. A robot is considered to be approaching the target when the relative distance between them is less than 10 meters. The original task is immediately interrupted and the new instruction is switched only when the priority of the new task is two levels higher than the current task (e.g., from "assisted following" to "dominant tracking"). Lower priority transitions (e.g., from level 3 to level 2) wait for the original task to complete before switching. A 0.5-second buffer period is added during task switching, and a speed gradient adjustment (e.g., gradually increasing the following speed from 8.1 m / s to 9 m / s) avoids abrupt stops and starts, making task instructions more flexible and reasonable. When multiple robots simultaneously meet the condition of approaching the target, a comprehensive score is calculated based on a distance weight (60%) plus resource utilization (40%), and the robot with the highest score receives the task.

[0040] In step S16, if there is a deviation in the motion parameters, the perception data of multiple robots are analyzed in real time to correct the motion parameters and obtain the corrected motion parameters.

[0041] In one feasible approach, if the motion parameters deviate, the process involves real-time analysis of perception data from multiple robots to correct the motion parameters and obtain corrected motion parameters, including: Acquire perception data from multiple robots, calibrate the data acquisition timestamps of each robot, and generate a time-aligned perception data set; Based on the time-aligned perception data set, combined with the perception data of the multiple robots and the preset robot capability weights, the corrected target motion parameters are obtained using a weighted average method.

[0042] It should be noted that acquiring perception data from multiple robots relies on the sensors mounted on each robot, such as LiDAR and cameras. These sensors generate data such as environmental information and target position. The network time protocol (PTP) can be used to calibrate the data acquisition timestamps of each robot, controlling the time synchronization error to within 1 millisecond, thereby generating a time-aligned perception data set. When calculating the corrected target motion parameters based on the time-aligned perception data set, a weighted average method is used, combining the perception data from multiple robots and preset robot capability weights. The robot capability weights are allocated based on the performance of each robot's sensors; for example, robot 1, with higher sensor accuracy, is assigned a weight of 0.6, while robots 2 and 3 are assigned weights of 0.2 and 0.2 respectively. Taking target distance data as an example, the fusion calculation yields (50 × 0.6 + 49.8 × 0.2 + 50.2 × 0.2) = 50 meters, which is one of the corrected target motion parameters. If the fused parameters deviate from the reference parameters, the Kalman filter algorithm can be further combined with iterative optimization using the state transition matrix and observation matrix to make the corrected parameters more accurate, effectively reduce errors, and improve reliability.

[0043] In step S17, the corrected motion parameters are constrained and verified to generate collision-free paths for each robot, thus obtaining a cooperative action sequence.

[0044] In one feasible approach, constraining and verifying the modified motion parameters to generate collision-free paths for each robot and obtain a cooperative action sequence includes: Based on the corrected motion parameters and combined with the environmental map, the first path set is obtained by adjusting waypoints to reduce the motion distance. Based on the first path set, and using the environmental map and the location of dynamic obstacles, the path points are readjusted to obtain a collision-free second path set; For the second set of collision-free paths, the speed and acceleration are checked. If they exceed the preset speed and acceleration thresholds, the path points and action times are adjusted to obtain a third set of paths that meet the constraints. Based on the third path set, collision-free paths for each robot are generated, resulting in the cooperative action sequence.

[0045] It should be noted that when generating a collision-free second path set based on the first path set, a conflict detection algorithm is needed, combined with the environmental map and the positions of dynamic obstacles, to identify path intersections or points of overlap with obstacles. The conflict detection algorithm mainly refers to geometric collision detection, which uses the GJK algorithm to calculate the difference between the robot's geometric model and the obstacles, determining whether overlapping areas exist. This tool is used to identify potential collision risks in the path. It combines the environmental map and the real-time position information of dynamic obstacles to check each robot's path point in the first path set one by one. For example, it checks whether there are intersections between the paths of different robots. If the path points of robot R1 overlap with those of R2, it will be identified as a conflict point. When verifying the speed and acceleration of the second path set, preset thresholds (such as a maximum speed of 2 m / s and an acceleration of 0.5 m / s²) must be used. If a certain path requires acceleration from 0 to 2 m / s within 0.5 seconds, exceeding the acceleration threshold, the action time is adjusted to 1 second, and the path points are adjusted (e.g., changing (40,40) to (41,41)) to form a third path set that meets the constraints, ensuring smooth robot movement. When generating a cooperative action sequence based on the third path set, the corrected motion parameters must be combined to determine the position of each robot at different timestamps. For example, R1 is located at (10,10) at t=0 seconds and reaches (20,20) at t=2 seconds, while R2 is located at (15,15) at t=1 seconds. This is output as a timestamp and coordinate pair, while also considering task priority (e.g., prioritizing heavy object handling tasks for specific robots) to improve overall collaborative efficiency.

[0046] In step S18, the cooperative action sequence is sent to each robot, and the real-time task execution status of each robot is analyzed to obtain real-time task execution feedback information.

[0047] In one feasible approach, sending the cooperative action sequence to each robot and analyzing the real-time task execution status of each robot to obtain real-time task execution feedback includes: Decompose the cooperative action sequence, distribute task instructions to each robot, and obtain the first task sequence received by each robot; Based on the first task sequence, ensure that the timing of each robot's actions is consistent to obtain the first action coordination data; Based on the first motion coordination data, the task execution status of each robot is collected in real time, the status data is updated, compared with the preset motion optimization rules, the task completion rate is judged, and real-time task execution feedback is obtained.

[0048] It should be noted that when decomposing the collaborative action sequence, it is broken down into independent sub-tasks (such as handling and moving) using a distributed architecture based on ROS2 (Data Distribution Service, DDS). Task instructions are distributed to each robot according to a preset task priority strategy (handling heavy objects has higher priority than auxiliary movement tasks). To ensure the reliable and timely delivery of high-priority instructions, the priority mechanism of the communication framework (such as DEADLINE and LIVELINESS in ROS2 QoS policies) is used to place critical instructions (such as the handling instructions for R1) in a priority position in the transmission queue. Each robot receives and acknowledges its assigned sub-task instructions, forming its own first task sequence. To cope with possible network packet loss or instruction loss, a mechanism with ACK acknowledgment and timeout retransmission (such as TCP retransmission or application-layer custom retransmission protocols) is adopted to ensure reliable instruction arrival. For example, in a warehouse scenario, robot R1 is prioritized for handling heavy objects, and R2 is assigned auxiliary movement tasks. The message queue places R1's instructions at the top of the queue, forming the first task sequence received by each robot, ensuring the real-time and sequential distribution of instructions. Based on the first task sequence, a high-precision time synchronization protocol (preferably IEEE 1588 PTP protocol, secondarily NTP protocol) is used to ensure that the clocks of each robot system are highly consistent (target synchronization accuracy in the sub-millisecond to microsecond range). Based on this, a data synchronization protocol (based on shared memory, DDS publish / subscribe, or custom state synchronization messages) is used to send cooperative action instructions containing precise timestamps (accurate to milliseconds or higher). The start and duration of each robot's actions are coordinated to generate first action coordination data. While collecting task execution status data in real time based on the first action coordination data, sensors are used to acquire information such as the position and speed of each robot, continuously updating the information and comparing it with preset action optimization rules. These rules include multi-dimensional constraints, such as position deviation less than 0.5 meters and speed not exceeding 2 meters per second. When multiple constraints are violated simultaneously, such as a position deviation of 0.6 meters and a speed of 2.5 meters per second, a preset arbitration rule is used for processing. The arbitration rule can be based on priority, i.e., safety constraints take precedence over efficiency constraints, determining which deviation to process first. Finally, the task completion rate is determined based on the status data (e.g., R1 completion rate 95%, R2 completion rate 90%), generating real-time task execution feedback to provide a basis for subsequent adjustments. In step S19, environmental obstacle information is obtained based on the real-time task execution feedback information, and the 3D model is updated to obtain the updated 3D model and motion trajectory.

[0049] In one feasible approach, the step of acquiring environmental obstacle information based on the real-time task execution feedback, updating the 3D model, and obtaining the updated 3D model and motion trajectory includes: Based on the real-time task execution feedback, the deviation data between the robot position and the preset robot position is extracted, and the deviation value is calculated through the coordinate difference. The system uses sensors to scan the dynamic environment, extracts obstacle information, and updates the environmental state to obtain the environmental obstacle information. Based on the deviation value and the environmental obstacle information, the three-dimensional model is reconstructed to obtain the updated three-dimensional model. The motion trajectory is obtained based on the updated 3D model and the deviation value.

[0050] It should be noted that when extracting deviation data based on real-time task execution feedback, it is necessary to obtain the robot's current coordinates and the coordinates of the preset position, and then calculate the deviation value using the coordinate difference formula. For example, if a warehouse robot is preset to move from (0,0,0) to (10,5,2), and its actual position is (9.8,5.1,2), then the deviation value is (0.2,-0.1,0). To ensure accuracy, high-precision positioning sensors such as LiDAR are used to control the coordinate data error within 0.1 meters. When using sensors to scan the dynamic environment to extract obstacle information, such as in a warehouse environment, LiDAR can scan 1000 times per second to generate data, identifying the position and size of obstacles such as temporary goods and mobile robots, for example, detecting a shelf at coordinates (5,3,1). Then, the Kalman filter algorithm is used to fuse historical and current data to update the environmental state, for example, correcting the shelf position to (5.1,3.2,1), obtaining accurate environmental obstacle information. When reconstructing the 3D model based on the deviation value and environmental obstacle information, the sensors on the robot collect target-related images, and the 3D model is generated through parallax calculation. For example, the updated model may show a platform 0.5 meters high near the target, improving the robot's adaptability to complex environments.

[0051] To facilitate understanding of the present invention, some preferred embodiments of the present invention will be described in further detail below.

[0052] In this embodiment, taking the collaborative completion of a precision engine block assembly task by multiple industrial robots in an automotive parts assembly workshop as an example, the specific process of a robot guidance method and system based on 3D vision according to the present invention is as follows: First, multi-view image data from multiple industrial robots is acquired and processed to obtain preliminary position information. Four assembly robots in the workshop are each equipped with a high-resolution camera (30 frames per second) to acquire image data of the cylinder, bolt holes, and end effectors of the robotic arms from the front, back, left, and right angles of the cylinder assembly station. The system timestamps the image data using the Network Time Protocol (NTP) to control the synchronization error within 5 milliseconds, generating a synchronized multi-view image set. For this image set, the SIFT algorithm is used to extract feature points (such as cylinder edges and bolt hole centers) from each frame. When the number of feature points in a single view exceeds 200, the movement of these feature points is tracked to obtain preliminary position information for each robotic arm end effector and the cylinder (e.g., the end effector of robotic arm 1 is located at coordinates (500, 300, 150) mm, and the cylinder positioning reference hole is located at (600, 400, 100) mm).

[0053] Secondly, a 3D model is constructed based on the preliminary position information and multi-view image data. The pose data of each robot is acquired (e.g., the joint angles of robotic arm 1: axis 1 is 30°, axis 2 is 45°). Multi-view images are corrected using camera extrinsic parameters to generate a first dataset containing the preliminary positions. Cylinder features are extracted from the dataset, and the first coarse spatial coordinates of the cylinder in the workshop coordinate system are calculated using the robotic arm pose data. If the multi-view coordinate deviation exceeds 0.1mm, a second optimized spatial coordinate (600.02, 400.01, 100.00 mm) is obtained through feature point pairing (e.g., matching the same bolt hole key points observed by robotic arm 1 and robotic arm 2). The error is then optimized using iterative least squares (controlled within ±0.05mm) to generate a 3D model of the cylinder and robotic arm. Next, the spatial geometric and motion features of the 3D model are analyzed to obtain the motion trajectory. Geometric features such as the centroid coordinates (500, 300, 150 mm) of the robotic arm and the inertial principal axis direction of the end effector (along the Z-axis) are extracted from the 3D model, along with motion features such as the end effector velocity (50 mm / s) and acceleration (10 mm / s²), forming a multidimensional state vector sequence. When the sequence length exceeds 50 time steps (0.1 seconds per step), if it exceeds the preset time series length threshold (40 steps), the sequence length is adjusted using the sliding window method to obtain the first predicted trajectory data.

[0054] Then, the motion trajectories of multiple robots are compared to obtain motion parameters. The position coordinate sequence of each robotic arm is obtained from the trajectory, and the velocity parameters (e.g., robotic arm 1 has a velocity of 50 mm / s at 0.5 seconds, and robotic arm 2 has a velocity of 52 mm / s) and acceleration parameters at each time point are calculated to generate a first velocity parameter set and a first acceleration parameter set. When transmitting parameter sets between robots, the timestamps are calibrated to the same millisecond level to generate a second parameter set. The motion parameter set is then calculated using a weighted average method (based on robot accuracy weights: robotic arm 1 is 0.3, robotic arm 2 is 0.3, and the rest are 0.2).

[0055] If the motion parameters are consistent with the preset parameters (e.g., speed deviation ≤ 2mm / s), a candidate set of task assignments is generated based on the trajectory. According to the relative position of the robotic arm and the bolt hole (robotic arm 1 is the closest), a candidate set is generated: robotic arm 1 is responsible for tightening the main bolt, robotic arm 2 is responsible for auxiliary positioning, and robotic arms 3 and 4 are responsible for pre-assembling surrounding parts. Based on priority rules, a task instruction is generated: robotic arm 1 moves at a speed of 50mm / s to a position 5mm above the bolt hole.

[0056] If there are deviations in the motion parameters (e.g., the measured acceleration of robotic arm 3 is 15 mm / s², exceeding the preset threshold of 10 mm / s²), the perception data of multiple robots are analyzed in real time. Force sensor data (e.g., a force of 20 N at the end of robotic arm 3) and visual data are acquired from each robotic arm. Timestamps are calibrated to generate a time-aligned perception data set. Combined with preset robot capability weights (robotic arm 3 has a weight of 0.2 due to lower sensor accuracy), a weighted average method is used to correct its acceleration to 12 mm / s². The corrected motion parameters are then constrained and verified to generate a collision-free path and a sequence of cooperative actions. Based on the corrected parameters and the workshop environment map (including the positions of tooling fixtures), the path points are adjusted to reduce the movement distance, resulting in the first path set. Based on the environment map and the positions of dynamic obstacles (such as temporarily placed tools), the path points are readjusted (e.g., robotic arm 3 avoids the tool at coordinates (550, 350, 120) mm), resulting in the second collision-free path set. The speed and acceleration are verified (ensuring that they do not exceed 10 mm / s²), and the path points and action time of robotic arm 3 are adjusted (extending the original 0.2-second movement time to 0.3 seconds), resulting in the third path set. This leads to the generation of a coordinated action sequence (e.g., when robotic arm 1 tightens a bolt, robotic arm 2 remains in a fixed position).

[0057] The coordinated action sequence is sent to each robot, and feedback information is obtained by analyzing the real-time task execution status. The sequence is decomposed and instructions are distributed to ensure that the timing of each robotic arm's actions is consistent (e.g., when robotic arm 1 starts tightening, robotic arm 2 applies positioning force synchronously); the execution status is collected in real time (e.g., the tightening torque of robotic arm 1 reaches 30 N·m), compared with the preset rule (torque deviation ≤ 2 N·m), and the task completion rate is judged (robotic arm 1 completes 100%), and real-time feedback is obtained.

[0058] Based on feedback, environmental obstacle information is obtained, and the 3D model and motion trajectory are updated. The deviation between the actual position and the preset position of the robotic arm is extracted (e.g., the deviation of the end effector of robotic arm 2 is 0.03mm). The environment is scanned by LiDAR to find newly appearing parts boxes (coordinates (650, 450, 100)mm), and the environmental obstacle information is updated. The 3D model is reconstructed by combining the deviation value and obstacle information to obtain the updated model and motion trajectory (the subsequent movements of the robotic arm avoid the parts boxes).

[0059] Reference Figure 2 The second embodiment of the present invention provides a robot guidance system based on 3D vision, comprising: The multi-view data processing module is used to acquire multi-view image data from multiple robots, analyze and process the multi-view image data, and obtain preliminary position information. The 3D modeling module is used to extract the robot's position information based on the preliminary position information and the multi-view image data, and obtain a 3D model. The trajectory prediction module is used to analyze the spatial geometric features and motion features of the three-dimensional model to obtain the motion trajectory; The parameter fusion module is used to compare the motion trajectories of multiple robots to obtain the motion parameters of each robot. The task allocation module is used to generate a task allocation candidate set by combining the motion parameters with the preset motion parameters if the motion parameters are consistent with the preset motion parameters, so as to obtain the task instructions for each robot. The parameter correction module is used to analyze the perception data of multiple robots in real time and correct the motion parameters if there is a deviation in the motion parameters, so as to obtain the corrected motion parameters. The path planning module is used to constrain and verify the corrected motion parameters, generate collision-free paths for each robot, and obtain a cooperative action sequence. The execution feedback module is used to send the cooperative action sequence to each robot, analyze the real-time task execution status of each robot, and obtain real-time task execution feedback information. The model update module is used to obtain environmental obstacle information based on the real-time task execution feedback information, update the three-dimensional model, and obtain the updated three-dimensional model and motion trajectory.

[0060] It should be noted that the robot guidance system based on 3D vision provided in this embodiment of the invention is used to execute all the process steps of the power control method for subway emergency lighting in the above embodiment. The working principle and beneficial effects of the two are one-to-one, so they will not be described again.

[0061] This invention also provides an electronic device. The electronic device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, such as a data processing program for robot guidance based on 3D vision. When the processor executes the computer program, it implements the steps in the various embodiments of the robot guidance method based on 3D vision described above, for example... Figure 1 The step S11 shown. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in the above-described device embodiments, such as the data acquisition module.

[0062] For example, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0063] The electronic device may be a desktop computer, laptop, handheld computer, or smart tablet, etc. The electronic device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the above components are merely examples of electronic devices and do not constitute a limitation on the electronic device. It may include more or fewer components than described above, or combine certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc.

[0064] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the electronic device, connecting all parts of the electronic device via various interfaces and lines.

[0065] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0066] Wherein, if the modules / units integrated in the electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0067] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0068] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A robot guidance method based on 3D vision, characterized in that, include: Acquire multi-view image data from multiple robots, analyze and process the multi-view image data to obtain preliminary position information; Based on the preliminary location information and combined with the multi-view image data, the robot's location information is extracted to obtain a three-dimensional model; The spatial geometric and motion characteristics of the three-dimensional model are analyzed to obtain the motion trajectory; The motion trajectories of multiple robots are compared to obtain the motion parameters of each robot; If the motion parameters are consistent with the preset motion parameters, then the motion trajectory is combined to generate a task assignment candidate set, and the task instructions for each robot are obtained. If the motion parameters are deviated, the perception data of multiple robots are analyzed in real time to correct the motion parameters and obtain the corrected motion parameters. The corrected motion parameters are constrained and verified to generate collision-free paths for each robot, resulting in a cooperative motion sequence. The coordinated action sequence is sent to each robot, and the real-time task execution status of each robot is analyzed to obtain real-time task execution feedback information. Based on the real-time task execution feedback information, environmental obstacle information is obtained, the three-dimensional model is updated, and the updated three-dimensional model and motion trajectory are obtained.

2. The robot guidance method based on 3D vision according to claim 1, characterized in that, The process of acquiring multi-view image data from multiple robots, analyzing and processing the multi-view image data to obtain preliminary position information includes: Acquire multi-view image data from multiple robots and perform timestamp calibration to obtain a synchronized multi-view image set; For the synchronized multi-view image set, each frame of the image is extracted and processed to obtain the feature point set of each view; If the number of feature points in each viewpoint is greater than a preset threshold, then visual tracking is performed on the feature points in each viewpoint to obtain preliminary position information.

3. The robot guidance method based on 3D vision according to claim 1, characterized in that, The step of extracting the robot's position information and obtaining a 3D model based on the preliminary position information and the multi-view image data includes: Acquire the robot's current pose data, correct the multi-view image data, and generate a first dataset containing the preliminary position information; Extract the robot's first edge features, first texture features and first key points from the first dataset, and combine them with the robot's current pose data to calculate the robot's coordinates in the coordinate system to obtain the first coarse spatial coordinates. If the first coarse spatial coordinates have deviations under multiple perspectives, the first key point is matched by fusing the multi-view image data to obtain the second optimized spatial coordinates. Error observation and data iteration are performed on the second optimized spatial coordinates to obtain the target three-dimensional model.

4. The robot guidance method based on 3D vision according to claim 1, characterized in that, The analysis of the spatial geometric and motion features of the three-dimensional model to obtain the motion trajectory includes: Spatial geometric features of the centroid, principal axis of inertia, and curvature change point, as well as motion features of velocity, acceleration, and angular velocity, are extracted from the three-dimensional model to form a multi-dimensional state vector sequence. If the time series length of the multidimensional state vector sequence exceeds a preset time series length threshold, the time series length of the multidimensional state vector sequence is readjusted to obtain the first predicted trajectory data. Based on the first predicted trajectory data, trajectory features of continuous time periods are extracted and fitted to obtain the motion trajectory.

5. The robot guidance method based on 3D vision according to claim 1, characterized in that, If the motion parameters are consistent with the preset motion parameters, then a task assignment candidate set is generated based on the motion trajectory to obtain the task instructions for each robot, including: If the deviation of the motion parameters between multiple robots is within a preset deviation threshold, then the data timestamp is calibrated to generate a consistent set of motion parameters. Based on the consistent motion parameter set and the relative position of the motion trajectory, a task assignment candidate set is generated; Based on the task allocation candidate set and the priority sorting rules, a task instruction set for each robot is generated.

6. The robot guidance method based on 3D vision according to claim 1, characterized in that, If the motion parameters deviate, the sensory data from multiple robots are analyzed in real time to correct the motion parameters, resulting in corrected motion parameters, including: Acquire perception data from multiple robots, calibrate the data acquisition timestamps of each robot, and generate a time-aligned perception data set; Based on the time-aligned perception data set, combined with the perception data of the multiple robots and the preset robot capability weights, the corrected target motion parameters are obtained using a weighted average method.

7. The robot guidance method based on 3D vision according to claim 1, characterized in that, The process involves constraining and verifying the corrected motion parameters, generating collision-free paths for each robot, and obtaining a cooperative action sequence, including: Based on the corrected motion parameters and combined with the environmental map, the first path set is obtained by adjusting waypoints to reduce the motion distance. Based on the first path set, and using the environmental map and the location of dynamic obstacles, the path points are readjusted to obtain a collision-free second path set; For the second set of collision-free paths, the speed and acceleration are checked. If they exceed the preset speed and acceleration thresholds, the path points and action times are adjusted to obtain a third set of paths that meet the constraints. Based on the third path set, collision-free paths for each robot are generated, resulting in a cooperative action sequence.

8. The robot guidance method based on 3D vision according to claim 1, characterized in that, The step of sending the coordinated action sequence to each robot and analyzing the real-time task execution status of each robot to obtain real-time task execution feedback information includes: Decompose the cooperative action sequence, distribute task instructions to each robot, and obtain the first task sequence received by each robot; Based on the first task sequence, ensure that the timing of each robot's actions is consistent to obtain the first action coordination data; Based on the first motion coordination data, the task execution status of each robot is collected in real time, the status data is updated, compared with the preset motion optimization rules, the task completion rate is judged, and real-time task execution feedback information is obtained.

9. A robot guidance method based on 3D vision according to claim 1, characterized in that, The step of obtaining environmental obstacle information based on the real-time task execution feedback information, updating the 3D model, and obtaining the updated 3D model and motion trajectory includes: Based on the real-time task execution feedback information, the deviation data between the robot position and the preset robot position is extracted, and the deviation value is calculated through the coordinate difference. The system uses sensors to scan the dynamic environment, extracts obstacle information, and updates the environmental state to obtain the environmental obstacle information. Based on the deviation value and the environmental obstacle information, the three-dimensional model is reconstructed to obtain the updated three-dimensional model. Based on the updated 3D model and the deviation value, the updated motion trajectory is obtained.

10. A robot guidance system based on 3D vision, characterized in that, include: The multi-view data processing module is used to acquire multi-view image data from multiple robots, analyze and process the multi-view image data, and obtain preliminary position information. The 3D modeling module is used to extract the robot's position information based on the preliminary position information and the multi-view image data, and obtain a 3D model. The trajectory prediction module is used to analyze the spatial geometric features and motion features of the three-dimensional model to obtain the motion trajectory; The parameter fusion module is used to compare the motion trajectories of multiple robots to obtain the motion parameters of each robot. The task allocation module is used to generate a task allocation candidate set by combining the motion parameters with the preset motion parameters if the motion parameters are consistent with the preset motion parameters, so as to obtain the task instructions for each robot. The parameter correction module is used to analyze the perception data of multiple robots in real time and correct the motion parameters if there is a deviation in the motion parameters, so as to obtain the corrected motion parameters. The path planning module is used to constrain and verify the corrected motion parameters, generate collision-free paths for each robot, and obtain a cooperative action sequence. The execution feedback module is used to send the cooperative action sequence to each robot, analyze the real-time task execution status of each robot, and obtain real-time task execution feedback information. The model update module is used to obtain environmental obstacle information based on the real-time task execution feedback information, update the three-dimensional model, and obtain the updated three-dimensional model and motion trajectory.

Images