Multi-agent three-dimensional visual cooperative positioning method and system based on deep learning

Abstract

The application discloses a multi-agent three-dimensional visual cooperative positioning method and system based on deep learning, and relates to the technical field of image analysis. Firstly, depth prediction and error analysis are performed on each frame of image, the network learning rate is dynamically adjusted based on the re-projection error factor, and the depth map is updated, so that the depth estimation of the pixel sub-region is more accurate. Then, the fluctuation factor of the depth map iteration is analyzed to adaptively adjust the optimization step and the damping factor, and the pose re-optimization is performed among the multi-agents to enhance the stability and consistency of the respective trajectory estimation. Finally, the abnormal observation of the common view area is discriminated and removed, and the three-dimensional visual map of each agent is constructed based on the cleaned depth and pose data, so that a closed-loop process from local depth estimation to global cooperative optimization is realized. Not only the depth prediction accuracy and the pose estimation stability are improved, but also the overall positioning accuracy of the system in a complex environment is significantly enhanced through abnormal observation removal and multi-source information fusion.

Description

Technical Field

[0001] This invention relates to the field of image analysis technology, and in particular to a multi-agent 3D visual collaborative localization method and system based on deep learning. Background Technology

[0002] Multi-agent 3D visual collaborative localization, as an important research direction in the fields of intelligent robotics and autonomous driving, aims to achieve high-precision perception and localization of one's own position and attitude in complex environments through the collaborative cooperation between multiple intelligent agents such as drones or unmanned vehicles.

[0003] For example, CN120431181A discloses a decentralized multi-machine visual collaborative localization method and apparatus. This method is applied to a first target UAV. It includes: within a preset sliding window, when the currently acquired image frame is detected as a keyframe, updating the local initialization list of local feature points based on the current keyframe; performing feature matching between the image feature points corresponding to the keyframe received from the second target UAV and the updated list of local feature points; selecting successfully matched image feature points from the keyframe received from the second target UAV to generate an inter-UAV feature point list; the second target UAV is used to indicate any other UAV in the UAV cluster that is different from the first target UAV; after filtering target landmarks according to the updated list of local feature points and several inter-UAV feature point lists, updating the fusion state of the first target UAV and the position of the target landmarks to generate an updated state.

[0004] For example, CN119693460B discloses an array-type multi-vehicle cooperative visual positioning system and method for transporting oversized items. This system is used to synchronously locate multiple vehicles in a convoy transporting oversized items. It includes a positioning module and image markers. The convoy consists of multiple rows of vehicles distributed along a horizontal axis. Each row includes two vehicles placed side-by-side, with all vehicles facing the same direction. The front of a vehicle is considered the front, and the rear is considered the rear. Each vehicle has a positioning module at its front end, including a camera, an IMU, and a communication unit. The system uses the two vehicles in the front row to obtain the absolute pose, and the vehicles in subsequent rows are positioned relative to each other. This improves the coordination between vehicles, achieving accurate real-time positioning of multiple vehicles while enhancing the adaptability of the perception system in degraded environments such as tunnels and in scenarios with poor signal.

[0005] However, in the process of implementing the inventive technical solution in the embodiments of this application, it was found that the above-mentioned technology has at least the following technical problems:

[0006] Image data collected by intelligent agents is affected by factors such as differences in viewpoint, occlusion, changes in lighting, and interference from dynamic objects, which can lead to inconsistent depth estimation and increased pixel-level reprojection errors. This, in turn, causes oscillations and drifts during the camera pose optimization process, making it difficult for multiple intelligent agents to maintain globally consistent 3D reconstruction and accurate cooperative navigation in complex environments. Summary of the Invention

[0007] To address the technical problems existing in the prior art, this invention provides a multi-agent 3D visual collaborative localization method based on deep learning. The technical solution includes acquiring image data from each individual agent and importing it into a depth prediction network to obtain and analyze the depth maps of each agent, obtaining the depth reprojection error factor of the image data of each agent, determining the perceptual differences in the image data of each agent, dynamically adjusting the learning rate of the depth prediction network, and updating the depth maps of each agent.

[0008] The updated depth map is input into the pose regression branch for iteration. The iteration process is analyzed to obtain the fluctuation factor of the depth map iteration of each single agent. The iteration step size and damping factor are dynamically adjusted, and the pose of the multi-agent is further optimized.

[0009] Anomaly detection is performed on the shared visual region of each agent to obtain the anomaly detection results of each agent. Anomaly processing is then performed on the anomaly detection results of the shared visual region of each agent to construct a three-dimensional visual map of each agent.

[0010] Furthermore, the depth reprojection error factor of each single agent image data is obtained. The specific process is as follows: the depth map of each single agent is divided into pixel regions to obtain each pixel sub-region of the depth map of each single agent. The depth value variance, average pixel projection deviation, and average pixel deviation between the predicted depth and the actual observation when projected into adjacent frames are collected for each pixel sub-region of the depth map of each single agent. These are denoted as the mean value of pixel reprojection error.

[0011] The depth value variance of each pixel sub-region of the depth map of each single agent is compared with the depth value bounding variance stored in the database, the average pixel projection deviation is compared with the pixel projection bounding deviation stored in the database, and the mean pixel reprojection error is compared with the pixel reprojection bounding error stored in the database. Weighting coefficients are introduced to obtain the depth reprojection error factor of each single agent image data. The depth reprojection error factor of each single agent image data is used to quantify the accuracy and stability of depth prediction of each single agent in different pixel sub-regions.

[0012] Furthermore, the fluctuation factor of the depth map iteration of each single agent is obtained. The specific process is as follows: obtain the camera pose change, residual descent rate, pose Euclidean distance and depth reprojection error factor of the image data of each single agent in each two adjacent iterations during the iteration process.

[0013] The camera pose change of each single agent in each two adjacent iterations is compared with the camera pose change bounding value stored in the database, the residual descent rate is compared with the residual descent bounding rate stored in the database, and the pose Euclidean distance is compared with the global convergence reference value stored in the database. The proportion analysis is performed and weight coefficients and depth reprojection error factors of the image data of each single agent are introduced to obtain the fluctuation factor of the depth map iteration of each single agent. The fluctuation factor of the depth map iteration of each single agent is used to quantify the convergence stability and optimization effect in the current iteration process.

[0014] Furthermore, the iteration step size and damping factor are dynamically adjusted, and the pose of the multi-agent system is further optimized. The specific process is as follows: the fluctuation factor of the depth map iteration of each single agent is obtained and compared with the fluctuation factor range of the depth map iteration of the agent set in the database. If the fluctuation factor of the depth map iteration of a single agent is higher than the upper limit of the fluctuation factor range of the agent's depth map iteration, the iteration step size is increased and the damping factor is decreased, and the camera pose and 3D point coordinates of each agent are jointly optimized. If the fluctuation factor of the depth map iteration of a single agent is lower than the lower limit of the fluctuation factor range of the agent's depth map iteration, the iteration step size is decreased and the damping factor is increased, and the camera pose and 3D point coordinates of each agent are jointly optimized. If the fluctuation factor of the depth map iteration of a single agent is within the fluctuation factor range of the agent's depth map iteration, its iteration step size and damping factor values ​​are kept unchanged, and the camera pose and 3D point coordinates of each agent are directly jointly optimized.

[0015] Furthermore, anomaly detection and judgment are performed on the common viewing area of ​​each single agent. The specific process is as follows: obtain the common viewing area of ​​each single agent, divide each agent into several groups, extract the fluctuation factor of the depth map iteration of each single agent in each group and the average value of the fluctuation factor of the depth map iteration of each group, take the average value of the fluctuation factor of the depth map iteration of each group as the center, and take a range of twice the standard deviation as the normal range of the fluctuation factor of the depth map iteration of that group. If the fluctuation factor of the depth map iteration of a certain single agent exceeds the normal range of the fluctuation factor of the depth map iteration of its group, then the single agent is marked as the first abnormal agent, thereby obtaining each first abnormal agent.

[0016] Furthermore, the abnormal observation results of each single agent are obtained. The specific process is as follows: count the number of labels of the first abnormal agent in each group and compare it with the abnormal agent label number threshold stored in the database. If the number of labels of a certain first abnormal agent in its group is higher than or equal to the abnormal agent label number threshold, then the abnormal observation result of the first abnormal agent is set as an abnormal observation agent. Thus, each abnormal observation agent is obtained. If the number of labels of a certain first abnormal agent in each group is lower than the abnormal agent label number threshold, then it is not necessary to set the abnormal observation result of the first abnormal agent as an abnormal observation agent.

[0017] Furthermore, abnormal observation results in the shared viewing area of ​​each individual agent are processed by anomaly handling. Specifically, each abnormal observation agent is extracted, temporarily removed from the current shared viewing area, and its removal status is recorded in the anomaly index table. In subsequent iterations, the fluctuation factor of the depth map iteration of each abnormal observation agent is continuously monitored. When the fluctuation factor of the depth map iteration of each abnormal observation agent returns to the normal range, its observation data is re-integrated into the joint graph optimization.

[0018] Furthermore, a three-dimensional visual map of each agent is constructed. The specific process is as follows: based on the updated depth map of each single agent and the optimized camera pose and depth map, the three-dimensional points in the visible area are spatially reconstructed to form a unified multi-agent global three-dimensional visual map.

[0019] The second aspect of the present invention also provides a multi-agent 3D visual collaborative localization system based on deep learning, comprising: an adaptive depth optimization module, used to acquire image data of each single agent and import it into a depth prediction network to obtain depth maps of each single agent and analyze them, obtain depth reprojection error factors of image data of each single agent, determine perceptual differences of image data of each single agent, dynamically adjust the learning rate of the depth prediction network, and update the depth maps of each single agent.

[0020] The pose regression and steady-state adjustment module is used to input the updated depth map into the pose regression branch for iteration, analyze the iteration process, obtain the fluctuation factor of the depth map iteration of each single agent, dynamically adjust the iteration step size and damping factor, and further optimize the pose of multiple agents.

[0021] The collaborative observation anomaly handling and map building module is used to judge anomalies in the shared viewing area of ​​each individual agent, obtain the anomaly observation results of each individual agent, process the anomalies in the shared viewing area of ​​each individual agent, and build a three-dimensional visual map of each agent.

[0022] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0023] (1) This invention proposes a multi-agent 3D visual collaborative localization method based on deep learning. First, depth prediction and error analysis are performed on each frame of image. The network learning rate is dynamically adjusted and the depth map is updated based on the reprojection error factor to ensure more accurate depth estimation of pixel sub-regions. Then, the step size and damping factor are adaptively adjusted and optimized by analyzing the fluctuation factor of the depth map iteration, and pose re-optimization is performed among multiple agents to enhance the stability and consistency of their respective trajectory estimations. Finally, abnormal observations in the shared viewing area are identified and eliminated, and a 3D visual map of each agent is constructed based on the cleaned depth and pose data, realizing a closed-loop process from local depth estimation to global collaborative optimization. This not only improves the accuracy of depth prediction and the stability of pose estimation, but also significantly enhances the overall localization accuracy of the system in complex environments through abnormal observation elimination and multi-source information fusion.

[0024] (2) This invention effectively quantifies the accuracy and stability of depth prediction for each agent in different image regions by dividing the depth map of each agent into pixel-level regions and obtaining the average depth reprojection error factor of the image data of each agent. Based on this, the system sets a sliding monitoring window for reprojection error to dynamically track the error change trend of each agent, adaptively adjust the learning rate of the depth prediction network, and correct the prediction deviation of the network in different regions in real time, thereby optimizing its depth map update effect. This mechanism not only enhances the local sensitivity and global adaptability of depth prediction, but also makes the system more resistant to interference and more precise in the fusion and iterative optimization of multi-source visual information.

[0025] (3) This invention obtains the fluctuation factor of the depth map iteration of each single agent and dynamically adjusts the iteration step size and damping factor of the corresponding agent: when the fluctuation is large, the optimization convergence speed is improved by increasing the step size and decreasing the damping; when the fluctuation is small, the system stability is enhanced by decreasing the step size and increasing the damping; and when the fluctuation factor is in the normal range, the current optimization rhythm is maintained. This not only improves the adaptability and coordination of multiple agents in joint optimization, but also significantly enhances the adjustment ability of 3D vision collaborative localization driven by deep learning under different convergence states, which has a positive effect on improving the dynamic convergence efficiency, accuracy and collaborative consistency of the overall system.

[0026] (4) This invention analyzes the agents in the shared field of view of multiple agents by grouping them and uses the fluctuation factor of the depth map iteration as the basis for judgment. It constructs an abnormal observation identification mechanism based on statistical distribution, thereby strictly screening out the observation agents that have significant anomalies. This effectively suppresses the interference of local observation errors on the overall collaborative positioning results and improves the adaptive ability, error robustness and convergence of the deep learning-based multi-agent 3D vision collaborative system in dynamic environments. It is of great significance for building a high-precision and high-reliability agent collaborative positioning system. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a schematic diagram of the method provided in an embodiment of the present invention;

[0029] Figure 2 This is a flowchart of depth prediction and dynamic learning rate adjustment provided in an embodiment of the present invention;

[0030] Figure 3 This is a flowchart of the anomaly observation and processing provided in an embodiment of the present invention;

[0031] Figure 4 This is a schematic diagram of the system provided in an embodiment of the present invention. Detailed Implementation

[0032] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0033] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0034] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0035] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0036] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0037] like Figure 1 As shown, Figure 1 The proposed method is a multi-agent 3D visual collaborative localization method based on deep learning. It includes acquiring image data of each agent and importing it into a depth prediction network to obtain and analyze the depth map of each agent, obtaining the depth reprojection error factor of the image data of each agent, judging the perceptual differences of the image data of each agent, dynamically adjusting the learning rate of the depth prediction network, and updating the depth map of each agent.

[0038] It should be noted that the depth prediction network is an end-to-end self-supervised monocular depth estimation model. Its network structure typically adopts an encoder-decoder architecture. The encoder takes one or more frames of RGB image sequences as input and extracts multi-scale features through multiple convolutions. The decoder then maps these features back to a dense depth map with the same resolution as the input image. During the network training phase, the network parameters are optimized mainly by minimizing the reprojection consistency loss between adjacent frames and the depth smoothing regularization loss, without the need for additional depth annotation data. The output is the depth value of each pixel, which is used to provide accurate spatial scale information for subsequent geometric constraints in the 3D reconstruction and pose regression modules, thereby achieving high-precision estimation of the depth structure of complex scenes and supporting multi-agent cooperative localization and map construction.

[0039] The updated depth map is input into the pose regression branch for iteration. The iteration process is analyzed to obtain the fluctuation factor of the depth map iteration of each single agent. The iteration step size and damping factor are dynamically adjusted, and the pose of the multi-agent is further optimized.

[0040] It should be noted that the pose regression branch is a neural network module used to regress camera pose parameters from an image or its corresponding depth map. It typically employs a lightweight convolutional neural network structure to extract image or depth features and outputs the corresponding pose vector (containing translation and rotation information) through fully connected layers. Its input is the updated single-agent depth map or image and historical frame data, and its output is the camera pose of the current frame relative to the reference frame. The role of this branch is to provide high-frequency pose estimation results. By iteratively analyzing the changing trend and stability of the pose of adjacent frames in its output, it evaluates the convergence state of the entire optimization process and dynamically adjusts the step size and damping factor in the graph optimization based on the fluctuation factor of the depth map iteration, thereby enhancing the robustness and accuracy of pose estimation in multi-agent cooperative localization.

[0041] Anomaly detection is performed on the shared visual region of each agent to obtain the anomaly detection results of each agent. Anomaly processing is then performed on the anomaly detection results of the shared visual region of each agent to construct a three-dimensional visual map of each agent.

[0042] It should be noted that the depth map of each individual agent refers to the depth information map corresponding to each pixel obtained after the two-dimensional image collected by each individual agent (such as a drone or unmanned vehicle) is processed by a depth prediction network. Each pixel value in the depth map represents the distance of that pixel relative to the camera in three-dimensional space (usually in meters), reflecting the spatial structure of the scene and the distance distribution of objects, and is the basic data for realizing three-dimensional visual localization and reconstruction.

[0043] This application's depth prediction and dynamic learning rate adjustment are as follows: Figure 2 As shown, the depth reprojection error factor of each single agent image data is obtained. The specific process is as follows: the depth map of each single agent is divided into pixel regions to obtain each pixel sub-region of the depth map of each single agent. The depth value variance, average pixel projection deviation and the average pixel deviation between the predicted depth and the actual observation when projected into adjacent frames are collected for each pixel sub-region of the depth map of each single agent. These are denoted as the mean value of pixel reprojection error.

[0044] It should be noted that the pixel sub-regions of the depth map for each single agent refer to dividing the entire depth map into multiple small local regions (such as sliding windows of fixed size) for fine-grained evaluation of depth prediction quality. Depth value variance refers to the variance of the predicted depth values ​​of all pixels within this sub-region, reflecting the stability and consistency of depth estimation in that region. It is usually obtained by calculating the standard deviation after statistically analyzing the depth values ​​of all pixels within the region. The average pixel projection bias is calculated by combining the predicted depth within the sub-region with camera intrinsic parameters, projecting the pixels into 3D space, and then back onto the image plane. This is compared with the original pixels, and the average projection bias of all pixels is calculated, reflecting the consistency between depth prediction and spatial reconstruction. The average pixel deviation between the predicted depth projected into adjacent frames and the actual observation is calculated by using the depth prediction results of the current frame to project the pixels into the adjacent frame images through pose transformation, matching the pixel positions with the actual observed image pixels in the adjacent frames, and averaging the corresponding pixel deviations. This reflects the accuracy of depth prediction in cross-frame matching.

[0045] The depth value variance of each pixel sub-region of the depth map of each single agent is compared with the depth value bounding variance stored in the database, the average pixel projection deviation is compared with the pixel projection bounding deviation stored in the database, and the mean pixel reprojection error is compared with the pixel reprojection bounding error stored in the database. Weighting coefficients are introduced to obtain the depth reprojection error factor of each single agent image data. The depth reprojection error factor of each single agent image data is used to quantify the accuracy and stability of depth prediction of each single agent in different pixel sub-regions.

[0046] It should be noted that the pixel reprojection boundary error refers to the standard threshold range that the system pre-sets to measure the accuracy of depth prediction. It is used to represent the upper limit of the average pixel deviation that a pixel should maintain between itself and the actual observation when it is reprojected from one frame to an adjacent frame after depth prediction and pose transformation under ideal or acceptable conditions. It is used to help determine whether the error of a certain pixel sub-region in the cross-frame reprojection process is within the normal range.

[0047] It should be noted that the depth reprojection error factor for each single agent's image data is analyzed under the following conditions:

[0048] ;

[0049] In the formula, WC i Denotes the depth reprojection error factor of the image data of the i-th single agent, DE. ik Let DE represent the depth value variance of the k-th pixel sub-region of the depth map of the i-th single agent, and TY represent the depth value bounding variance. ikLet TY represent the average pixel projection deviation of the k-th pixel sub-region of the depth map of the i-th single agent, and ZC represent the pixel projection delimitation deviation. ik Let ZC represent the mean pixel reprojection error of the k-th pixel sub-region of the depth map of the i-th single agent, let A1 represent the weight coefficient corresponding to the variance of the depth values ​​stored in the database, let A2 represent the weight coefficient corresponding to the pixel projection deviation stored in the database, let A3 represent the weight coefficient corresponding to the mean pixel reprojection error stored in the database, let i represent the number of each single agent, i=1,2,3,...,n, where n represents the total number of single agents, and let k represent the number of each pixel sub-region, k=1,2,3,...,m, where m represents the total number of pixel sub-regions.

[0050] It should be noted that in the depth map of each single agent, the depth value variance of each pixel sub-region reflects the consistency level of depth prediction in that region. The larger the value, the more dispersed the prediction results are in the region, which can easily lead to significant pixel position shifts when projected into adjacent frames, i.e., an increase in average pixel projection deviation. When the average pixel projection deviation increases, the average pixel deviation between the predicted depth projected into adjacent frames and the actual observation also increases, indicating that the geometric consistency of the region deteriorates, and the system will be more sensitive to errors in these regions. Furthermore, the accumulation of prediction errors will cause the depth prediction network to converge more slowly in the sub-region, manifested as a continuous increase in depth value variance, forming a feedback loop.

[0051] It should be noted that the weighting coefficients corresponding to the depth value variance, pixel projection deviation, and mean pixel reprojection error are all stored in the database, and their values ​​are typically set between 0 and 1. For example, by constructing mapping tables between the depth value variance, pixel projection deviation, mean pixel reprojection error, and weighting coefficients, the real-time detected depth value variance, pixel projection deviation, and mean pixel reprojection error are input into the corresponding mapping tables in the database, thereby quickly obtaining the weighting coefficients corresponding to the depth value variance, pixel projection deviation, and mean pixel reprojection error, respectively.

[0052] It should be noted that the average pixel projection deviation between the predicted depth of each pixel sub-region of the depth map of each single agent and the actual observation when projected into adjacent frames is first calculated by back-projecting each pixel in the current frame into 3D space according to its predicted depth (using the camera's intrinsic parameters and the depth value corresponding to each pixel to map the 2D image coordinates back to the 3D coordinates in the camera coordinate system). Then, the 3D point is projected onto the image plane of the adjacent frame using the camera pose. The difference in pixel coordinates between the projected point and the corresponding observation point in the adjacent frame is calculated, and the average projection error of all pixels in the sub-region is taken to obtain the average pixel projection deviation of the region. The depth value variance is the statistical variance of the predicted depth values ​​of all pixels in the pixel sub-region, reflecting the dispersion of the depth values ​​in the region, and is used to measure the geometric complexity or instability of the region. The average pixel projection deviation refers to the average of the above pixel projection errors. As a quantitative indicator of the depth prediction accuracy of the region, it can reflect the overall consistency level between the predicted depth and the actual observation. The average pixel projection deviation is used to measure the local error of a single frame in cross-frame geometric projection, while the average pixel reprojection error focuses more on the comprehensive evaluation stability across multiple frames.

[0053] It should be noted that the depth value bounded variance refers to the upper limit of the depth value variance of each pixel sub-region statistically derived from a large-scale calibration or simulation dataset; the pixel projection bounded deviation is the statistical value of pixel deviation obtained by projecting the stereo or multi-view calibration data at the pixel level under known real 3D geometric conditions, to reflect the acceptable range of projection error under normal observation conditions; the pixel reprojection bounded error is determined by the statistical result of the reprojection error calculated by the monocular or multi-view depth estimation network on the validation set, taking the mean plus standard deviation of the reprojection error in the network's multi-scene testing as the tolerable reprojection error threshold.

[0054] Specifically, the learning rate of the depth prediction network is dynamically adjusted, and the depth maps of each agent are updated. The process is as follows: a sliding window for reprojection error monitoring is preset. The average depth reprojection error factor of the image data of each agent is obtained within the sliding window. The average depth reprojection error factor of the image data of each agent is compared with the preset range of average depth reprojection error factors in the database. If the average depth reprojection error factor of a certain agent image data is higher than the upper limit of the range, the learning rate is increased and the depth maps of each agent are updated. If the average depth reprojection error factor of a certain agent image data is lower than the lower limit of the range, the learning rate is decreased and the depth maps of each agent are updated. If the average depth reprojection error factor of a certain agent image data is within the range, the original learning rate is maintained and the depth maps of each agent are updated directly.

[0055] It should be noted that the average depth reprojection error factor of each agent's image data is obtained by statistically analyzing multiple depth reprojection error factors within a continuous time period or several consecutive frames within a preset reprojection error sliding monitoring window. The system calculates the current depth reprojection error factor in each frame or each iteration and stores it sequentially into the sliding window in chronological order. When the window is full, the earliest data is replaced by the new data, thus always maintaining a data set of a fixed length. By taking the arithmetic mean of all error factors in this set, the average error factor reflecting the overall depth prediction error level of the agent at the current moment is obtained, which is used to further evaluate its perception stability and depth prediction accuracy.

[0056] It's important to note that reprojection refers to mapping the 3D points obtained from depth prediction back from the camera coordinate system to the image plane based on camera intrinsic and extrinsic parameters. This is then compared with the corresponding pixel positions in the real image to measure the geometric deviation caused by the depth estimation. The reprojection error sliding monitoring window is a fixed-length historical error sequence buffer used to smooth and track the changing trend of depth reprojection errors over several consecutive frames or iterations, preventing single instantaneous fluctuations from excessively affecting the learning rate adjustment. The average depth reprojection error factor range is a preset upper and lower limit range reflecting the average reprojection error should be within during normal operation, ensuring the ability to distinguish between normal and abnormal conditions during online judgment. When the average reprojection error factor of an agent is higher than the upper limit of this range, it indicates that its depth prediction significantly deviates from the observed geometry, and the network's response speed should be increased. In this case, the learning rate will be increased by a small percentage (e.g., 10%~20% of the original learning rate) to increase the weight update step size and correct the depth error more quickly. Conversely, when the error factor is lower than the lower limit of the range, the learning rate will be decreased by the same amount to tighten the update step size and prevent over-adjustment. If the error factor falls within the normal range, the current learning rate is maintained, and the depth map is regenerated directly using the updated network parameters, thereby achieving a rapid response to high error regions and fine adjustment to low error regions.

[0057] It should be noted that after adjusting the learning rate, the system retrains the depth prediction network with the updated learning rate and performs forward propagation again on the input images of each agent to generate new depth maps. When the learning rate is increased, the network's gradient response to high-error regions intensifies, and the depth map is corrected more quickly in these regions, exhibiting a significantly smoother depth distribution that is closer to the real geometry. When the learning rate is decreased, the network update step size becomes smaller, and the output for low-error or converged regions is more detailed, avoiding excessive oscillations. In summary, by selectively increasing or decreasing the learning rate, the new depth map achieves significant improvements in projection consistency between adjacent frames and overall reprojection error.

[0058] The abnormal observation processing in this application is as follows: Figure 3 As shown, the fluctuation factor of the depth map iteration of each single agent is obtained. The specific process is as follows: obtain the camera pose change, residual descent rate, pose Euclidean distance and depth reprojection error factor of each single agent's image data in each two adjacent iterations during the iteration process.

[0059] The camera pose change of each single agent in each two adjacent iterations is compared with the camera pose change bounding value stored in the database, the residual descent rate is compared with the residual descent bounding rate stored in the database, and the pose Euclidean distance is compared with the global convergence reference value stored in the database. The proportion analysis is performed and weight coefficients and depth reprojection error factors of the image data of each single agent are introduced to obtain the fluctuation factor of the depth map iteration of each single agent. The fluctuation factor of the depth map iteration of each single agent is used to quantify the convergence stability and optimization effect in the current iteration process.

[0060] It should be noted that in the entire 3D vision joint optimization process, between two adjacent graph optimization iterations, each adjacent iteration refers to a new iteration formed after each parameter update in the iterative optimization algorithm. The system regards the result between the j-th iteration and the j-1-th iteration as a pair of adjacent iterations, and then compares and analyzes the camera pose change, residual descent rate, pose Euclidean distance and other indicators between these pairs of iterations to reflect the real-time convergence and stability of the current optimization step.

[0061] It should be noted that the camera pose change refers to the combined change in camera translation and rotation parameters between the current iteration and the previous iteration, measuring the magnitude of the pose update. The corresponding camera pose change threshold is a reference upper limit representing the normal fluctuation range of pose updates, obtained in advance through offline calibration or historical optimization experiments, used to determine whether the current update is too large or too small. The residual descent rate refers to the relative decrease in the optimization target projection error in each iteration, reflecting the convergence speed. Its corresponding residual descent threshold rate is also derived from offline or simulation data statistics, representing the typical rate at which the residual should decrease in each step under ideal convergence conditions. The pose Euclidean distance is the Euclidean distance calculated after converting the camera extrinsic parameters of the current frame and the previous frame into bit vectors, used to quantitatively measure the overall pose change. Its corresponding global convergence reference value is a small-amplitude limit that pose updates should maintain in global graph optimization, obtained through research on the algorithm's convergence behavior in numerous scenarios.

[0062] It should be noted that the fluctuation factor of the depth map iteration for each single agent is analyzed under the following conditions:

[0063] ;

[0064] In the formula, DWi XW represents the fluctuation factor of the depth graph iteration for the i-th single agent. ji X represents the change in camera pose of the i-th single agent in two consecutive iterations of the j-th iteration, XW represents the camera pose change bound, and CX represents the change in camera pose bound. ji Let CX represent the residual descent rate of the i-th single agent in two consecutive iterations of the j-th iteration, and TD represent the residual descent bounding rate. ji Let TD represent the Euclidean distance between the poses of the i-th single agent in two consecutive iterations of the j-th iteration, and WC represent the global convergence reference value. i B1 represents the depth reprojection error factor of the image data of the i-th single agent, B2 represents the weight coefficient corresponding to the camera pose change stored in the database, B3 represents the weight coefficient corresponding to the residual descent rate stored in the database, B4 represents the weight coefficient corresponding to the pose Euclidean distance stored in the database, i represents the weight coefficient corresponding to the depth reprojection error factor stored in the database, i represents the number of each single agent, i=1,2,3,...,n, n represents the total number of single agents, j represents the number of the two adjacent iterations between each iteration, j=1,2,3,...,t, t represents the total number of the two adjacent iterations.

[0065] It should be noted that the two adjacent iterations between each iteration refer to the iteration phases that connect the j-th iteration and the (j-1)-th iteration in the continuous optimization process, arranged in chronological or step order.

[0066] It should be noted that the change in camera pose of each agent in each two adjacent iterations directly reflects the magnitude of pose adjustment in the current optimization step. When the change in pose is large, it is usually accompanied by an accelerated rate of residual descent, indicating that the model is rapidly correcting errors. However, excessive change in pose may also lead to drastic fluctuations in the Euclidean distance of the pose, which in turn causes instability in the depth map projection, manifested as an increase in the depth reprojection error factor. Conversely, the increase in the depth reprojection error factor feeds back into the optimization process, slowing down the rate of residual descent and prompting the iteration step size and damping factor to be adaptively adjusted to suppress excessive oscillations, thereby reducing drastic changes in camera pose. As the error gradually converges, the change in pose and the Euclidean distance of the pose tend to stabilize, the depth reprojection error factor decreases, and the overall optimization enters a stable stage, achieving a dynamic balance between the rate of residual descent, pose change, and reprojection error, ensuring the convergence of the multi-agent 3D vision localization system.

[0067] It should be noted that the weighting coefficients corresponding to camera pose change, residual descent rate, pose Euclidean distance, and depth reprojection error factor are all stored in the database, and their values ​​are typically set between 0 and 1. For example, by constructing mapping tables between the camera pose change, residual descent rate, pose Euclidean distance, and depth reprojection error factor of each single agent's image data and their weighting coefficients, the real-time detected camera pose change, residual descent rate, pose Euclidean distance, and depth reprojection error factor of each single agent's image data are input into the corresponding mapping tables in the database, thereby quickly obtaining the weighting coefficients corresponding to the camera pose change, residual descent rate, pose Euclidean distance, and depth reprojection error factor, respectively.

[0068] Specifically, the iteration step size and damping factor are dynamically adjusted, and the pose of multiple agents is further optimized. The specific process is as follows: the fluctuation factor of the depth map iteration of each single agent is obtained and compared with the fluctuation factor range of the depth map iteration of the agent set in the database. If the fluctuation factor of the depth map iteration of a single agent is higher than the upper limit of the fluctuation factor range of the agent's depth map iteration, the iteration step size is increased and the damping factor is decreased, and the camera pose and 3D point coordinates of each agent are jointly optimized. If the fluctuation factor of the depth map iteration of a single agent is lower than the lower limit of the fluctuation factor range of the agent's depth map iteration, the iteration step size is decreased and the damping factor is increased, and the camera pose and 3D point coordinates of each agent are jointly optimized. If the fluctuation factor of the depth map iteration of a single agent is within the fluctuation factor range of the agent's depth map iteration, its iteration step size and damping factor values ​​are kept unchanged, and the camera pose and 3D point coordinates of each agent are directly jointly optimized.

[0069] It should be noted that the fluctuation factor range of depth map iteration is a numerical range used to measure whether the convergence stability and optimization effect of each single agent are within a reasonable range during the depth map iteration optimization process.

[0070] In one specific embodiment, the joint optimization process for the camera pose and 3D point coordinates of each agent involves the optimizer obtaining the translation and rotation increments for each camera and the adjustment vector for each 3D point coordinate based on the latest iteration step size and damping factor. Finally, the system applies these increments to the pose parameters and 3D point coordinates of each agent to complete a round of global update, thereby simultaneously refining the camera trajectory and scene reconstruction results of each agent while ensuring the consistency of multi-agent co-view.

[0071] In a specific embodiment, when the fluctuation factor of a single agent's depth map iteration exceeds the set fluctuation factor range, its iteration step size and damping factor need to be dynamically adjusted to improve the convergence and stability of the overall optimization: If the fluctuation factor is higher than the upper limit of the range, it indicates that the current iteration process is oscillating and the convergence is unstable. In this case, the deviation between the fluctuation factor and the upper limit is calculated, normalized, and multiplied by a preset adjustment coefficient. This negative adjustment amount is then subtracted from the current iteration step size (i.e., the step size is reduced), while the damping factor is increased to improve the system's ability to suppress oscillations. If the fluctuation factor is lower than the lower limit of the range, it indicates that the current iteration convergence is too slow or too conservative. In this case, the deviation between the fluctuation factor and the lower limit is normalized and multiplied by the adjustment coefficient. The step size is increased to accelerate the optimization process, while the damping factor is reduced to release the suppression constraint and improve the tracking ability of the target optimal solution. If the fluctuation factor is within the range, no adjustment is made, and the joint optimization of camera pose and 3D points is performed directly. The adjustment of the optimization step size is guided according to the error fluctuation to avoid oscillations caused by too fast convergence and to prevent getting stuck in local optima by too slow convergence, thus ensuring the robustness and efficiency of the multi-agent 3D vision collaborative localization system.

[0072] Specifically, anomaly detection and judgment are performed on the common viewing area of ​​each single agent. The specific process is as follows: obtain the common viewing area of ​​each single agent, divide each agent into several groups, extract the fluctuation factor of the depth map iteration of each single agent in each group and the average value of the fluctuation factor of the depth map iteration of each group, take the average value of the fluctuation factor of the depth map iteration of each group as the center, and take a range of twice the standard deviation as the normal range of the fluctuation factor of the depth map iteration of that group. If the fluctuation factor of the depth map iteration of a certain single agent exceeds the normal range of the fluctuation factor of the depth map iteration of its group, then the single agent is marked as the first abnormal agent, thereby obtaining each first abnormal agent.

[0073] It should be noted that the shared field of view (CLP) among the individual agents is identified by the system based on image data collected in a multi-agent collaborative perception scenario. This CLP is determined through feature extraction and matching algorithms, identifying regions where multiple agents simultaneously observe the same physical space from overlapping perspectives. Subsequently, the system performs joint triangulation on key feature points within the CLP region, generating preliminary locally dense or sparse 3D point cloud data. Using the intrinsic parameters of each agent's camera and relative pose information, the system maps the corresponding 3D structure of this region to the coordinate system of each individual agent, thus forming an initial 3D visual map prototype.

[0074] Specifically, the abnormal observation results of each single agent are obtained through the following process: the number of labels of the first abnormal agent in each group is counted and compared with the threshold of the number of abnormal agent labels stored in the database. If the number of labels of a certain first abnormal agent in its group is higher than or equal to the threshold of the number of abnormal agent labels, then the abnormal observation result of the first abnormal agent is set as an abnormal observation agent, thus obtaining each abnormal observation agent. If the number of labels of a certain first abnormal agent in each group is lower than the threshold of the number of abnormal agent labels, then it is not necessary to set the abnormal observation result of the first abnormal agent as an abnormal observation agent.

[0075] It should be noted that the threshold for the number of abnormal agent labels refers to a limit value that is pre-set for each group during offline calibration or historical data statistics to determine whether the agent's observations are continuously abnormal.

[0076] Specifically, abnormal observation results in the common field of view of each individual agent are handled by anomaly processing. The specific process is as follows: extract each abnormal observation agent, temporarily remove each abnormal observation agent from the current common field of view, record the removal status of each abnormal observation agent in the anomaly index table, and continuously monitor the fluctuation factor of the depth map iteration of each abnormal observation agent in subsequent iterations. When the fluctuation factor of the depth map iteration of each abnormal observation agent returns to the normal range, its observation data is re-integrated into the joint graph optimization.

[0077] It should be noted that the process of re-incorporating the observation data into the joint graph optimization is as follows: When the fluctuation factors of the depth map iterations of each anomalous observation agent fall back and stabilize within the normal range set by the system, the system first performs statistical verification using stable factor data within several consecutive sliding windows. The system records the depth map iteration fluctuation factors generated by the anomalous observation agents in the most recent iterations and fills these factor values ​​sequentially into a sliding window of a preset length. This fluctuation factor reflects the overall stability of the camera pose change amplitude, residual descent rate, and pose convergence distance during each round of depth map optimization. The system calculates the average value of the fluctuation factors in the sliding window and compares it with the normal stable range set in the database. Only when the average value remains within this range is the optimization state of the agent considered to have stabilized. Subsequently, the system reloads the snapshots of the observation data (including depth map, pose, 3D point cloud, and common-view information) saved when the agent was previously removed into the joint graph and performs a joint graph optimization on these data and the current observation data of other agents. During the optimization process, the data is incorporated into the corresponding positions in the graph structure, and its residual weights are initialized to low values ​​to avoid the impact of sudden changes.

[0078] Specifically, the process of constructing a 3D visual map for each agent is as follows: based on the updated depth map of each individual agent and the optimized camera pose and depth map, the 3D points in the visible area are spatially reconstructed to form a unified global 3D visual map for multiple agents.

[0079] It should be noted that the specific process of constructing the 3D visual map for each agent is as follows: First, the system extracts the updated depth map of each single agent and the corresponding optimized camera pose information. Based on these data, the pixels in each frame are back-projected to transform them into dense or semi-dense 3D point clouds. Then, the system transforms and fuses all 3D points in a unified coordinate system using the camera pose to ensure that 3D data from different agents and at different times can be aligned to the same global space. Finally, the system uses the co-view relationship to optimize the local alignment of overlapping areas to ensure the spatial consistency of observations among multiple agents, thereby constructing a unified, continuous, and structurally complete global 3D visual map for multiple agents.

[0080] like Figure 4 As shown, the second aspect of the present invention also provides a multi-agent 3D visual collaborative localization system based on deep learning, comprising: an adaptive depth optimization module, used to acquire image data of each single agent and import it into a depth prediction network, obtain depth maps of each single agent and analyze them, obtain depth reprojection error factors of image data of each single agent, determine perceptual differences of image data of each single agent, dynamically adjust the learning rate of the depth prediction network, and update the depth maps of each single agent.

[0081] The pose regression and steady-state adjustment module is used to input the updated depth map into the pose regression branch for iteration, analyze the iteration process, obtain the fluctuation factor of the depth map iteration of each single agent, dynamically adjust the iteration step size and damping factor, and further optimize the pose of multiple agents.

[0082] The collaborative observation anomaly handling and map building module is used to judge anomalies in the shared viewing area of ​​each individual agent, obtain the anomaly observation results of each individual agent, process the anomalies in the shared viewing area of ​​each individual agent, and build a three-dimensional visual map of each agent.

[0083] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0084] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. A computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. Available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. Semiconductor media can be solid-state drives.

[0085] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0086] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0087] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0088] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0089] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0090] In the embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0091] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0092] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0093] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0094] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multi-agent 3D visual cooperative localization method based on deep learning, characterized in that, The method includes: Image data of each agent is acquired and imported into the depth prediction network to obtain depth maps of each agent and analyze them. The depth reprojection error factor of the image data of each agent is obtained, the perceptual differences of the image data of each agent are judged, the learning rate of the depth prediction network is dynamically adjusted, and the depth maps of each agent are updated. The updated depth map is input into the pose regression branch for iteration. The iteration process is analyzed to obtain the fluctuation factor of the depth map iteration of each single agent. The iteration step size and damping factor are dynamically adjusted, and the pose of the multi-agent is further optimized. Anomaly detection is performed on the shared visual area of ​​each agent to obtain the anomaly detection results of each agent. Anomaly processing is performed on the anomaly detection results of the shared visual area of ​​each agent to construct a three-dimensional visual map of each agent. The depth reprojection error factor for the image data of each single agent is analyzed under the following conditions: ； In the formula, WC i Denotes the depth reprojection error factor of the image data of the i-th single agent, DE. ik Let DE represent the depth value variance of the k-th pixel sub-region of the depth map of the i-th single agent, and TY represent the depth value bounding variance. ik Let TY represent the average pixel projection deviation of the k-th pixel sub-region of the depth map of the i-th single agent, and ZC represent the pixel projection delimitation deviation. ik Let ZC represent the mean pixel reprojection error of the k-th pixel sub-region of the depth map of the i-th single agent, let A1 represent the weight coefficient corresponding to the variance of the depth values ​​stored in the database, let A2 represent the weight coefficient corresponding to the pixel projection deviation stored in the database, let A3 represent the weight coefficient corresponding to the mean pixel reprojection error stored in the database, let i represent the number of each single agent, i=1,2,3,...,n, where n represents the total number of single agents, and let k represent the number of each pixel sub-region, k=1,2,3,...,m, where m represents the total number of pixel sub-regions.

2. The multi-agent 3D visual collaborative localization method based on deep learning according to claim 1, characterized in that, The specific process for obtaining the depth reprojection error factor of each single agent's image data is as follows: The depth map of each single agent is divided into pixel regions to obtain each pixel sub-region of the depth map of each single agent. The depth value variance, average pixel projection deviation, and average pixel deviation between the predicted depth when projected into adjacent frames and the actual observation of each pixel sub-region of the depth map of each single agent are collected and denoted as the mean pixel reprojection error. The depth value variance of each pixel sub-region of the depth map of each single agent is compared with the depth value boundary variance stored in the database, the average pixel projection deviation is compared with the pixel projection boundary deviation stored in the database, and the mean pixel reprojection error is compared with the pixel reprojection boundary error stored in the database. Weighting coefficients are introduced to obtain the depth reprojection error factor of each single agent image data. The depth reprojection error factor of each single agent image data is used to quantify the accuracy and stability of depth prediction of each single agent in different pixel sub-regions.

3. The multi-agent 3D visual collaborative localization method based on deep learning according to claim 1, characterized in that, The process of dynamically adjusting the learning rate of the depth prediction network and updating the depth map of each agent is as follows: A preset reprojection error sliding monitoring window is used to obtain the depth reprojection average error factor of each agent's image data. The depth reprojection average error factor of each agent's image data is compared with the preset depth reprojection average error factor range in the database. If the depth reprojection average error factor of a certain agent's image data is higher than the upper limit of the depth reprojection average error factor range, the learning rate is increased and the depth map of each agent is updated. If the depth reprojection average error factor of a certain agent's image data is lower than the lower limit of the depth reprojection average error factor range, the learning rate is decreased and the depth map of each agent is updated. If the depth reprojection average error factor of a certain agent's image data is within the depth reprojection average error factor range, the original learning rate is maintained and the depth map of each agent is updated directly.

4. The multi-agent 3D visual collaborative localization method based on deep learning according to claim 1, characterized in that, The specific process for obtaining the fluctuation factor of the depth map iteration for each single agent is as follows: During the iteration process, obtain the camera pose change, residual descent rate, pose Euclidean distance, and depth reprojection error factor of each single agent's image data in each two adjacent iterations. The camera pose change of each single agent in each two adjacent iterations is compared with the camera pose change bounding value stored in the database, the residual descent rate is compared with the residual descent bounding rate stored in the database, and the pose Euclidean distance is compared with the global convergence reference value stored in the database. The proportion analysis is performed and weight coefficients and depth reprojection error factors of the image data of each single agent are introduced to obtain the fluctuation factor of the depth map iteration of each single agent. The fluctuation factor of the depth map iteration of each single agent is used to quantify the convergence stability and optimization effect in the current iteration process.

5. The multi-agent 3D visual collaborative localization method based on deep learning according to claim 1, characterized in that, The specific process of dynamically adjusting the iteration step size and damping factor, and re-optimizing the pose of the multi-agent system is as follows: The fluctuation factor of the depth map iteration for each agent is obtained and compared with the fluctuation factor range of the depth map iteration for agents set in the database. If the fluctuation factor of the depth map iteration for a certain agent is higher than the upper limit of the fluctuation factor range of the depth map iteration for the agent, the iteration step size is increased and the damping factor is decreased, and the camera pose and 3D point coordinates of each agent are jointly optimized. If the fluctuation factor of the depth map iteration for a certain agent is lower than the lower limit of the fluctuation factor range of the depth map iteration for the agent, the iteration step size is decreased and the damping factor is increased, and the camera pose and 3D point coordinates of each agent are jointly optimized. If the fluctuation factor of the depth map iteration for a certain agent is within the fluctuation factor range of the depth map iteration for the agent, its iteration step size and damping factor values ​​are kept unchanged, and the camera pose and 3D point coordinates of each agent are directly jointly optimized.

6. The multi-agent 3D visual collaborative localization method based on deep learning according to claim 1, characterized in that, The specific process for anomaly detection and judgment of the shared field of view of each individual agent is as follows: Obtain the common field of view of each agent, divide each agent into several groups, extract the fluctuation factor of the depth map iteration of each agent in each group and the average value of the fluctuation factor of the depth map iteration of each group, take the average value of the fluctuation factor of the depth map iteration of each group as the center, and take a range of twice the standard deviation as the normal range of the fluctuation factor of the depth map iteration of that group. If the fluctuation factor of the depth map iteration of a certain agent exceeds the normal range of the fluctuation factor of the depth map iteration of its group, then the agent is marked as the first abnormal agent, thus obtaining each first abnormal agent.

7. The multi-agent 3D visual collaborative localization method based on deep learning according to claim 6, characterized in that, The specific process for obtaining the abnormal observation results of each individual agent is as follows: The number of labels for the first anomalous agent in each group is counted and compared with the threshold number of anomalous agent labels stored in the database. If the number of labels for a first anomalous agent in its group is higher than or equal to the threshold number of anomalous agent labels, then the anomalous observation result of the first anomalous agent is set as an anomalous observation agent. This process is used to obtain each anomalous observation agent. If the number of labels for a first anomalous agent in each group is lower than the threshold number of anomalous agent labels, then it is not necessary to set the anomalous observation result of the first anomalous agent as an anomalous observation agent.

8. The multi-agent 3D visual cooperative localization method based on deep learning according to claim 7, characterized in that, The specific process for handling abnormal observations of the shared field of view of each individual agent is as follows: Extract each anomaly-observing agent and temporarily remove them from the current shared field of view. The removal status of each abnormal observation agent is recorded in the abnormal index table, and the fluctuation factor of the depth map iteration of each abnormal observation agent is continuously monitored in subsequent iterations. When the fluctuation factor of the depth map iteration of each abnormal observation agent returns to the normal range, its observation data is re-integrated into the joint graph optimization.

9. The multi-agent 3D visual collaborative localization method based on deep learning according to claim 1, characterized in that, The specific process for constructing the 3D visual map of each intelligent agent is as follows: Based on the updated depth maps of each individual agent and the optimized camera pose and depth maps, spatial reconstruction of 3D points within the visible area is performed to form a unified multi-agent global 3D visual map.

10. A system applying the deep learning-based multi-agent 3D visual cooperative localization method as described in any one of claims 1-9, comprising: The adaptive depth optimization module is used to acquire image data of each agent and import it into the depth prediction network to obtain and analyze the depth map of each agent. It obtains the depth reprojection error factor of the image data of each agent, judges the perceptual differences of the image data of each agent, dynamically adjusts the learning rate of the depth prediction network, and updates the depth map of each agent. The pose regression and steady-state adjustment module is used to input the updated depth map into the pose regression branch for iteration, analyze the iteration process, obtain the fluctuation factor of the depth map iteration of each single agent, dynamically adjust the iteration step size and damping factor, and further optimize the pose of multiple agents. The collaborative observation anomaly handling and map building module is used to judge anomalies in the shared viewing area of ​​each individual agent, obtain the anomaly observation results of each individual agent, process the anomalies in the shared viewing area of ​​each individual agent, and build a three-dimensional visual map of each agent.

Images