A method, system, terminal, and storage medium for constructing and updating real-scene 3D models based on UAV crowd-source perception.

Abstract

The application belongs to the technical field of computer and location service, and discloses a kind of based on unmanned aerial vehicle crowd source perception Real scene three-dimensional model construction and updating method, system, terminal and storage medium, comprising: obtaining unmanned aerial vehicle perception data;Based on unmanned aerial vehicle perception data, by the method that target detection and optical character recognition are combined, extract target feature and semantic information in current environment;Based on unmanned aerial vehicle perception data, utilize multi-sensor tight coupling technology to construct real scene three-dimensional local map in real time;According to the target feature and semantic information extracted, utilize key frame selection and semantic auxiliary constraint method to splice real scene three-dimensional local map, obtain real scene three-dimensional global map;Based on feature auxiliary global loop detection and pose graph optimization method, eliminate cumulative error in real scene three-dimensional global map, obtain updated real scene three-dimensional model.The application realizes the construction and timely update of real scene three-dimensional model, improves the efficiency and precision of construction and update.

Description

Technical Field

[0001] This invention relates to the fields of computer and location services technology, and in particular to a method, system, terminal and storage medium for constructing and updating a real-scene 3D model based on crowdsourced perception from unmanned aerial vehicles (UAVs). Background Technology

[0002] Real-world 3D models are digital spaces that realistically, three-dimensionally, and temporally reflect and express human production, living, and ecological spaces within a certain area. They can intuitively and accurately display the spatial structure of a city and support various spatial analyses and decision-making processes, possessing significant practical value. The rapid changes in the urban environment necessitate frequent updates to real-world 3D models to maintain their freshness and accuracy, posing a serious challenge to traditional methods of constructing and updating such models.

[0003] Currently, data acquisition for real-world 3D models primarily relies on satellite remote sensing, aerial remote sensing, or ground-based data collection. These methods are not only costly but also struggle to achieve large-scale and timely data updates. Furthermore, traditional real-world 3D model updates typically employ full reconstruction, resulting in time-consuming, labor-intensive, and unreliable data processing, failing to meet the real-time demands of dynamic urban changes. In today's context of rapid urban development, the limitations of traditional methods are particularly pronounced.

[0004] Therefore, the existing technology still suffers from low efficiency and low accuracy in construction and updating, and the technology needs further improvement. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method, system, terminal and storage medium for constructing and updating real-scene 3D models based on crowdsourced perception of UAVs, in order to solve the problems of low efficiency and low accuracy in the construction and updating of existing real-scene 3D models.

[0006] The technical solution adopted by this invention to solve the technical problem is as follows:

[0007] In a first aspect, the present invention provides a method for constructing and updating a real-scene 3D model based on crowd-sourced perception from unmanned aerial vehicles (UAVs), including:

[0008] Acquire UAV perception data; wherein, the UAV perception data includes: lidar point cloud data, visual image data, and inertial measurement data;

[0009] Based on the UAV perception data, target features and semantic information in the current environment are extracted by combining target detection and optical character recognition.

[0010] Based on the UAV perception data, a real-time 3D local map is constructed using multi-sensor tight coupling technology.

[0011] Based on the extracted target features and semantic information, the real-scene 3D local map is stitched together using keyframe selection and semantic-assisted constraint methods to obtain a real-scene 3D global map.

[0012] The feature-assisted global loop closure detection and pose graph optimization method eliminates the accumulated errors in the real-scene 3D global map and obtains the updated real-scene 3D model.

[0013] In one implementation, the step of extracting target features and semantic information in the current environment based on the UAV perception data, through a method combining target detection and optical character recognition, includes:

[0014] The target detection results are obtained by extracting target information from the visual image data using target detection technology; wherein, the target information includes: the bounding box, category label, and corresponding confidence score of the target;

[0015] The optical character recognition technology is used to identify the character regions in the visual image data, extract the character bounding box, character content and corresponding confidence score, and obtain the optical character recognition result.

[0016] Based on the target detection results and the optical character recognition results, feature filtering is performed to obtain the target features and semantic information in the current environment.

[0017] In one implementation, the step of performing feature filtering based on the target detection result and the optical character recognition result to obtain the target features and semantic information in the current environment includes:

[0018] The confidence level of the target detection results and the optical character recognition results is determined according to the preset confidence level filtering function, and the target detection results and optical character recognition results that meet the confidence level requirements are fused by the overlap of spatial positions to obtain the target features and semantic information in the current environment.

[0019] In one implementation, the step of constructing a real-world 3D local map in real time based on the UAV's perception data using multi-sensor tight coupling technology includes:

[0020] The lidar point cloud data and the visual image data are synchronously calibrated in space and time to establish a coordinate system transformation relationship between the two.

[0021] The lidar point cloud data is downsampled to obtain downsampled point cloud data;

[0022] Feature extraction is performed on the visual image data to obtain image features containing two-dimensional coordinate information;

[0023] The fusion objective function is determined based on the point cloud projection error, the image feature projection error, and the coordinate system transformation relationship.

[0024] Based on the fusion objective function, the inertial measurement data, the image features, and the downsampled point cloud data are fused using the multi-sensor tight coupling technology, and the real-scene 3D local map is generated through a sliding window mechanism.

[0025] In one implementation, the step of stitching together the real-scene 3D local map using keyframe selection and semantic-assisted constraint methods based on the extracted target features and semantic information to obtain a real-scene 3D global map includes:

[0026] Select key frames in the process of constructing the real-scene 3D local map;

[0027] The semantic information of the selected keyframes is matched, and the point cloud pose is matched based on the target features of the keyframes that are semantically matched to obtain the relative pose relationship between the keyframes.

[0028] The relative pose relationships between the corresponding real-scene 3D local maps are inferred based on the relative pose relationships between the keyframes.

[0029] Based on the relative pose relationships between the real-scene 3D local maps and the corresponding UAV flight trajectories, all the real-scene 3D local maps are stitched together to obtain the real-scene 3D global map.

[0030] In one implementation, the feature-assisted global loop closure detection and pose graph optimization method eliminates accumulated errors in the real-scene 3D global map to obtain an updated real-scene 3D model, including:

[0031] Based on the global loop closure detection method, the pose, corresponding target features and semantic information of each key frame are matched with the existing data in the real-scene 3D global map to identify the loop closure regions in the real-scene 3D global map.

[0032] The pose relationship of the target features in the loop closure region is used as a loop closure detection constraint and added to the pose graph optimization model;

[0033] Based on the pose graph optimization model, the real-scene 3D global map is globally optimized using the optimization objective function that minimizes the overall error, thereby eliminating the accumulated error in the real-scene 3D global map and obtaining the updated real-scene 3D model.

[0034] In one implementation, the step of globally optimizing the real-world 3D global map based on the pose graph optimization model using an optimization objective function that minimizes the overall error includes:

[0035] The optimization objective function is determined based on the relative pose relationship between keyframes, the loop closure detection constraint, and the corresponding weighting factor.

[0036] Adjust the pose of all keyframes in the real-world 3D global map according to the optimization objective function to eliminate the accumulated error in the real-world 3D global map.

[0037] Secondly, the present invention provides a system for constructing and updating a real-scene 3D model based on crowd-sourced perception from unmanned aerial vehicles (UAVs), comprising:

[0038] The data acquisition module is used to acquire UAV perception data; wherein, the UAV perception data includes: lidar point cloud data, visual image data, and inertial measurement data;

[0039] The target detection module is used to extract target features and semantic information in the current environment based on the UAV perception data by combining target detection with optical character recognition.

[0040] The local map construction module is used to construct a real-world 3D local map in real time based on the UAV's perception data and using multi-sensor tight coupling technology.

[0041] The local map stitching module is used to stitch together the real-scene 3D local map based on the extracted target features and semantic information, using keyframe selection and semantic auxiliary constraint methods, to obtain a real-scene 3D global map.

[0042] The real-scene 3D model update module is used to eliminate the accumulated errors in the real-scene 3D global map by using a feature-assisted global loop closure detection and pose graph optimization method, so as to obtain an updated real-scene 3D model.

[0043] Thirdly, the present invention provides a terminal, comprising: a processor and a memory, wherein the memory stores a program for constructing and updating a real-scene 3D model based on UAV crowd-source perception, and the program for constructing and updating a real-scene 3D model based on UAV crowd-source perception is executed by the processor to implement the operation of the method for constructing and updating a real-scene 3D model based on UAV crowd-source perception as described in the first aspect.

[0044] Fourthly, the present invention also provides a computer-readable storage medium storing a program for constructing and updating a real-scene 3D model based on UAV crowd-source perception. When executed by a processor, the program is used to implement the operation of the method for constructing and updating a real-scene 3D model based on UAV crowd-source perception as described in the first aspect.

[0045] The present invention, by employing the above technical solution, has the following effects:

[0046] This invention utilizes visual target detection, character recognition, and LiDAR point cloud fusion technologies to accurately calculate the location information of features. By establishing consistency constraints through perception results from different UAVs, it significantly improves the efficiency and accuracy of real-scene 3D mapping. Furthermore, by selecting keyframes and several preceding and following frames of point cloud data to construct local sub-maps, this invention effectively reduces the computational complexity of map stitching. Simultaneously, it combines the generalized ICP algorithm to achieve loop closure detection and error correction, optimizing the pose relationship and geometric consistency of the global map, significantly improving the efficiency and accuracy of 3D map construction. This invention is particularly suitable for map updates and optimization in rapidly changing urban environments. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0048] Figure 1 This is a flowchart of the method for constructing and updating a real-scene 3D model based on crowdsourced perception of unmanned aerial vehicles in this invention.

[0049] Figure 2 This is a schematic diagram illustrating data collection by multiple drones in a city, as described in this invention.

[0050] Figure 3 This is an overall flowchart of the method for constructing and updating a real-scene 3D model based on crowdsourced perception of UAVs in this invention.

[0051] Figure 4 This is a schematic diagram illustrating the fusion of target detection and character recognition in this invention.

[0052] Figure 5 This is a diagram of the multi-UAV data exchange architecture in this invention.

[0053] Figure 6 This is a schematic diagram of keyframe point cloud registration in this invention.

[0054] Figure 7This is a functional schematic diagram of the terminal in one implementation of the present invention.

[0055] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0057] Exemplary methods

[0058] Currently, data acquisition for real-world 3D models primarily relies on satellite remote sensing, aerial remote sensing, or ground-based data collection. These methods are not only costly but also struggle to achieve large-scale and timely data updates. Furthermore, traditional real-world 3D model updates typically employ full reconstruction, resulting in time-consuming, labor-intensive, and unreliable data processing, failing to meet the real-time demands of dynamic urban changes. In today's context of rapid urban development, the limitations of traditional methods are particularly pronounced.

[0059] To address the above-mentioned technical problems, this invention provides a method for constructing and updating a real-scene 3D model based on UAV crowdsourced perception. The method includes: acquiring UAV perception data; extracting target features and semantic information from the current environment based on the UAV perception data using a combination of target detection and optical character recognition; constructing a real-scene 3D local map in real time using multi-sensor tight coupling technology based on the UAV perception data; stitching the real-scene 3D local map together with keyframe selection and semantic-assisted constraint methods according to the extracted target features and semantic information to obtain a real-scene 3D global map; and eliminating accumulated errors in the real-scene 3D global map using feature-assisted global loop closure detection and pose graph optimization methods to obtain an updated real-scene 3D model. This invention achieves the construction and timely updating of real-scene 3D models, improving the efficiency and accuracy of construction and updating.

[0060] like Figure 1 As shown, this embodiment of the invention provides a method for constructing and updating a real-scene 3D model based on crowdsourced perception from unmanned aerial vehicles (UAVs), including the following steps:

[0061] Step S100: Acquire UAV perception data; wherein, the UAV perception data includes: lidar point cloud data, visual image data, and inertial measurement data;

[0062] In this embodiment, the proposed method for constructing and updating real-scene 3D models based on UAV crowdsourced perception fully utilizes the data collected by UAVs during their daily operations in the low-altitude economy. Unlike traditional methods that require satellite remote sensing or dedicated UAV deployment for data collection, this embodiment achieves large-scale data acquisition by crowdsourcing and utilizing the data generated by UAVs during routine tasks.

[0063] like Figure 2 As shown, this embodiment controls numerous drones to fly in the city, thereby collecting multi-source perception data required for the construction and updating of the real-scene 3D model. This extensive drone perception data includes: LiDAR point cloud data, visual image data, and inertial measurement data. By fusing these data, a high-precision real-scene 3D model is constructed, and a dynamic update algorithm is employed to achieve timely updates of the real-scene 3D model, ensuring the map's freshness and accuracy, and meeting the city's needs for real-scene 3D models.

[0064] like Figure 3 As shown, the method provided in this embodiment mainly includes four parts: visual feature recognition, local map construction, local map stitching, and global map optimization. The visual feature recognition part extracts permanent features (i.e., target features) and their semantic information through target detection and OCR (Optical Character Recognition), providing spatiotemporal correlation and constraints for local map construction. In the local map construction stage, multi-sensor fusion technology couples LiDAR point clouds, visual images, and inertial measurement data to generate a high-precision local map. In the local map stitching stage, the semantic information of the target features is used to guide keyframe alignment, and the ICP algorithm (a classic iterative optimization algorithm for point cloud registration) is combined to complete the high-precision stitching of the local map. Finally, global loop closure detection and pose graph optimization correct accumulated errors to ensure the accuracy and consistency of the global map.

[0065] like Figure 1 As shown, this embodiment of the invention provides a method for constructing and updating a real-scene 3D model based on crowdsourced perception from unmanned aerial vehicles (UAVs), including the following steps:

[0066] Step S200: Based on the UAV perception data, target features and semantic information in the current environment are extracted using a method that combines target detection and optical character recognition.

[0067] In this embodiment, a method for automatically recording visual features based on a combination of target detection and OCR character recognition is proposed for visual feature recognition. This method extracts multimodal feature information from video streams (i.e., visual image data) and records the detection results and related information (e.g., timestamps, confidence levels, and location information) in real time. This method can efficiently and automatically identify and record target objects and character information in complex environments, providing rich semantic input for subsequent multi-sensor data fusion and map construction.

[0068] Specifically, in one implementation of this embodiment, step S200 includes the following steps:

[0069] Step S201: Extract target information from the visual image data using target detection technology to obtain target detection results; wherein, the target information includes: the bounding box, category label, and corresponding confidence score of the target;

[0070] Step S202: The character regions in the visual image data are identified using optical character recognition technology, and the bounding boxes, character content, and corresponding confidence scores of the characters are extracted to obtain the optical character recognition results.

[0071] Step S203: Based on the target detection results and the optical character recognition results, feature filtering is performed to obtain the target features and semantic information in the current environment.

[0072] In this embodiment, during the visual feature recognition process that integrates target detection and character recognition, the input video frames are first processed. Target detection technology is used to extract target information in the scene, including the bounding boxes of the target objects. Category tags and confidence level The results of object detection can be described as follows:

[0073] ;

[0074] in, This indicates the number of objects detected in the current frame. Object detection algorithms can quickly and accurately detect objects in a scene, generating results with clear boundaries and category labels.

[0075] Then, optical character recognition (OCR) technology is used to identify the character regions in the input frame and extract the bounding boxes of the characters. Character content and confidence level The OCR recognition result can be represented as:

[0076] ;

[0077] in, This indicates the number of characters detected in the current frame. OCR technology can effectively supplement the semantic information deficiencies of object detection methods by detecting and decoding character regions.

[0078] To improve the accuracy of the recorded results, a confidence filtering function is introduced in this embodiment. The results of target detection and OCR are evaluated.

[0079] In one implementation of this embodiment, step S203 includes the following steps:

[0080] Step S203a: Determine the confidence level of the target detection result and the optical character recognition result according to the preset confidence level filtering function, and fuse the target detection result and the optical character recognition result that meet the confidence level requirements by the overlap of spatial positions to obtain the target features and semantic information in the current environment.

[0081] In this embodiment, the confidence screening function is defined as follows:

[0082] ;

[0083] in: The threshold for target detection confidence. The threshold for character recognition confidence. Only when the confidence level of target detection is [value missing]. Confidence in character recognition The current feature will only be recorded when it exceeds the corresponding threshold. and Information.

[0084] Furthermore, the target detection results and OCR results that meet the confidence requirements are fused using the Intersection over Union (IoU). The IoU is calculated as follows:

[0085] ;

[0086] The target object and character information are considered relevant when the following conditions are met:

[0087] ;

[0088] The fused set of features is represented as:

[0089] ;

[0090] in, The timestamp of the current frame. This indicates the number of multimodal features after fusion.

[0091] The timestamp is calculated using the current frame index and the video frame rate (FPS):

[0092] ;

[0093] Ultimately, the features that meet the criteria are recorded, such as... Figure 4 As shown, the information obtained after the fusion of object detection and character recognition includes: timestamp, object category, location, and character content.

[0094] like Figure 1 As shown, this embodiment of the invention provides a method for constructing and updating a real-scene 3D model based on crowdsourced perception from unmanned aerial vehicles (UAVs), including the following steps:

[0095] Step S300: Based on the UAV perception data, a real-time 3D local map is constructed using multi-sensor tight coupling technology.

[0096] In this embodiment, a method for constructing a real-world 3D local map based on tight coupling of multiple sensors is proposed during the local map construction stage. This method is specifically applied to high-precision real-world 3D mapping tasks for UAVs in low-altitude scenarios. By fusing multimodal data from LiDAR point cloud data, visual image data, and inertial measurement data, the UAV can generate high-precision local maps in real time in complex environments, providing crucial data support for navigation, path planning, and global map stitching.

[0097] Specifically, in one implementation of this embodiment, step S300 includes the following steps:

[0098] Step S301: Perform spatial and temporal synchronization calibration on the lidar point cloud data and the visual image data, and establish the coordinate system transformation relationship between the two.

[0099] Step S302: Downsample the lidar point cloud data to obtain downsampled point cloud data;

[0100] Step S303: Extract features from the visual image data to obtain image features containing two-dimensional coordinate information;

[0101] Step S304: Determine the fusion objective function based on the point cloud projection error, the image feature projection error, and the coordinate system transformation relationship;

[0102] Step S305: Based on the fusion objective function, the inertial measurement data, the image features, and the downsampled point cloud data are fused using the multi-sensor tight coupling technology, and the real-scene 3D local map is generated through a sliding window mechanism.

[0103] In this embodiment, during the drone's flight, the lidar is responsible for collecting dense point cloud data of the scene. The visual camera acquires environmental images in real time. First, in order to achieve the fusion of multi-sensor data, this embodiment performs spatial and temporal synchronous calibration on the lidar and camera, establishing a coordinate system transformation relationship between the two. Its representation is as follows:

[0104] ;

[0105] in, It is a rotation matrix; The translation vector describes the relative position and attitude of the lidar and camera in the UAV's body coordinate system.

[0106] During real-time data processing, the lidar point cloud data is downsampled using voxel filtering to remove redundant points and reduce computational load. The downsampled point cloud is represented as follows:

[0107] ;

[0108] At the same time, visual images Feature extraction is performed to generate a series of image feature points. , where each feature point It includes its two-dimensional coordinate information.

[0109] In this embodiment, based on the fusion objective function, a tightly coupled pose optimization framework is used to fuse these feature points with the LiDAR point cloud; wherein, the fusion objective function can be determined according to the point cloud projection error, the image feature projection error, and the coordinate system transformation relationship; specifically, the fusion objective function is defined as:

[0110] ;

[0111] in, For the first point cloud Three-dimensional points; The three-dimensional coordinates of the visual feature points; This is the pose transformation matrix for the current frame; and These represent the projection errors of point cloud and image feature points, respectively.

[0112] During drone flight, local map construction employs a sliding window mechanism to ensure real-time system performance while generating high-quality maps. Specifically, through the most recent... Intra-frame point cloud data and the corresponding pose Generate local map The formula is expressed as:

[0113] ;

[0114] The sliding window in this embodiment limits the data range of the local map, retaining only the key frame point cloud related to the current position of the UAV, which reduces computational complexity and improves the real-time update capability of the local map.

[0115] Furthermore, this embodiment also iteratively adjusts the pose estimation results within the sliding window through joint optimization of geometric constraints and feature matching. The goal of the sliding window mechanism is to minimize the accumulation of errors in the local map.

[0116] ;

[0117] Through the above optimization process, drones can construct local maps with geometric accuracy and semantic integrity in complex scenarios (such as those with large variations in lighting).

[0118] This embodiment achieves high-precision, low-computational-load local map construction in UAV 3D mapping tasks by employing a multi-sensor tightly coupled fusion and sliding window strategy. The generated local map not only provides high-quality basic data for global stitching and loop closure detection, but also directly supports UAV navigation, obstacle avoidance, and path planning applications, making it particularly suitable for urban low-altitude environments and complex indoor scenes.

[0119] like Figure 1 As shown, this embodiment of the invention provides a method for constructing and updating a real-scene 3D model based on crowdsourced perception from unmanned aerial vehicles (UAVs), including the following steps:

[0120] Step S400: Based on the extracted target features and semantic information, the real-scene 3D local map is stitched together using keyframe selection and semantic-assisted constraint methods to obtain a real-scene 3D global map.

[0121] In this embodiment, during the local map stitching stage, the extracted target features and semantic information are combined with the ICP algorithm to complete the high-precision stitching of the local map; wherein, the semantic information of the target features is used to guide keyframe alignment.

[0122] Specifically, in one implementation of this embodiment, step S400 includes the following steps:

[0123] Step S401: Select key frames in the process of constructing the real-scene 3D local map;

[0124] Step S402: Match the semantic information of the selected keyframes, and perform point cloud pose matching based on the target features of the keyframes that have successfully matched the semantics, so as to obtain the relative pose relationship between the keyframes.

[0125] Step S403: Calculate the relative pose relationship between the corresponding real-world 3D local maps based on the relative pose relationship between the keyframes.

[0126] Step S404: Based on the relative pose relationships between the real-scene 3D local maps and the corresponding UAV flight trajectories, all the real-scene 3D local maps are stitched together to obtain the real-scene 3D global map.

[0127] In this embodiment, during the local map stitching stage, to address the issue of insufficient constraints caused by the sparse LiDAR point cloud, target detection, OCR recognition, local map construction, and loop closure detection technologies are comprehensively used to perform semantically assisted map stitching, with features as the core. Figure 5 As shown, in this embodiment, a global database is established based on a multi-UAV data exchange architecture. On this basis, the real-scene 3D local map is stitched together using keyframe selection and semantic auxiliary constraint methods to obtain a real-scene 3D global map.

[0128] In a practical application scenario, based on multi-UAV data exchange, the process of stitching together a local map using selected keyframes is as follows:

[0129] First, permanent features and character information in the environment are extracted through object detection and OCR recognition. The feature recognition results include category, recognition confidence level, timestamp, time elapsed since the video started playing, and the feature's location in the image. During drone flight, once the recognition confidence levels of object detection and OCR reach a set threshold, the drone records the recognition results along with the aforementioned auxiliary information, providing semantic constraints for subsequent map stitching. (Timestamp) It is the core of the record, used to associate the spatiotemporal information of features with point clouds.

[0130] Secondly, a local map is constructed using a tightly coupled laser vision-inertial odometry system, and point cloud frames corresponding to specific time points are located based on recorded timestamps. (The text abruptly ends here, likely due to an incomplete sentence or missing information.) Within a range of 0.5 to 1 second before and after (0.5 seconds is used as an example in this embodiment, and the effective time range of this timestamp is 1 second), select the point cloud data of the first 5 frames and the last 5 frames, and fuse them into a keyframe. This strategy ensures that each keyframe contains sufficient information while reducing computational complexity.

[0131] Next, the lidar and camera are calibrated to obtain the pose relationship matrix between them. This refers to the transformation relationship between the LiDAR coordinate system and the camera coordinate system. After calibration, the camera can identify features in the image and calculate the relative position of those features within the image. Based on the coordinate transformations between the feature-image, image (camera)-LiDAR, and LiDAR-point cloud, the corresponding point cloud is found, and the spatial position of the feature within the point cloud is extracted by fitting a plane. Assuming the relative position information of the feature is... The equations for calculating the point cloud plane in the lidar coordinate system are as follows:

[0132] ;

[0133] in, This is the plane normal vector. After fitting the plane using the least squares method, the relative positional relationship between the lidar and the feature can be calculated. When different drones scan the same feature, this relative positional relationship can be recorded, and by utilizing the fixed nature of the feature, the relative relationship between the drone scanning trajectories at different times and locations can be deduced.

[0134] During the local map stitching stage, each UAV first extracts keyframes from the mapping process. Each keyframe carries semantic information (such as feature categories and OCR character information). When matching keyframes, the semantic information is checked first to see if they match. Only keyframes with consistent semantic information are registered using ICP (Iterative Closest Point) for the point cloud.

[0135] like Figure 6 As shown, Figure 6 A schematic diagram for registering point clouds of ICP keyframes; specifically, assuming two keyframe point clouds. and The initial pose relationship is determined by the positional relationship of the features, denoted as The goal of the ICP algorithm is to minimize the matching error between two frames of point clouds:

[0136] ;

[0137] in, The final relative pose between two keyframes. and For matching point cloud pairs. Initial values ​​provided at feature locations. Under the guidance of this, the ICP algorithm is more likely to converge to the global optimum.

[0138] Furthermore, by calculating the pose relationships between keyframes This allows us to deduce the relative pose relationships between different local maps. Assuming a local map... and The poses are respectively and The relative pose relationships between local maps are as follows:

[0139] ;

[0140] Different local maps and UAV trajectories can be stitched together to form a global map. Finally, the UAV's flight trajectory is extracted from each local map, and loop closure detection is performed using the Generalized ICP (GICP) algorithm. By introducing loop closure constraints, the pose of the global map is optimized, minimizing the overall mapping error. The objective function for global map optimization can be expressed as:

[0141] ;

[0142] in, Represents the set of constraint edges. This represents the relative pose relationship obtained through loop closure detection or feature constraints.

[0143] In summary, this embodiment combines visual recognition with LiDAR point cloud processing, utilizing the semantic information of features to provide auxiliary constraints for map stitching and global optimization, significantly improving the accuracy and robustness of map construction, while also solving the problem of insufficient constraint information caused by the sparseness of LiDAR point clouds.

[0144] like Figure 1 As shown, this embodiment of the invention provides a method for constructing and updating a real-scene 3D model based on crowdsourced perception from unmanned aerial vehicles (UAVs), including the following steps:

[0145] Step S500: The feature-assisted global loop closure detection and pose graph optimization method eliminates the accumulated error in the real-scene 3D global map, and obtains the updated real-scene 3D model.

[0146] In this embodiment, during the collaborative mapping process of multiple UAVs, due to the influence of accumulated errors, even if a global map is generated by stitching together local maps, there may still be significant overall pose offsets or local inconsistencies. To address this issue, this embodiment proposes an error correction method based on global loop closure detection. By identifying repeatedly visited regions (loops) in the map and applying additional constraints, combined with a global optimization model, the accumulated error is significantly reduced, improving the accuracy and consistency of the map.

[0147] Specifically, in one implementation of this embodiment, step S500 includes the following steps:

[0148] Step S501: Based on the global loop closure detection method, the pose and corresponding target features and semantic information of each key frame are matched with the existing data in the real-scene 3D global map to identify the loop closure regions in the real-scene 3D global map.

[0149] Step S502: The pose relationship of the target features in the loop closure region is used as a loop closure detection constraint and added to the pose graph optimization model.

[0150] In this embodiment, the core of the global loop closure detection lies in identifying repeatedly accessed targets in the same area at different time periods by using the drone's flight trajectory and the semantic information of features. During flight, the drone detects features in the environment in real time and records the feature's category, OCR recognition content, and its relative position in the image. After the global map is generated, the pose of each keyframe and its associated feature information are matched with existing data in the global map. Assuming a certain feature... At any moment and If an object is detected repeatedly by two drones or by the same drone, the pose relationship between the two detections can be expressed as:

[0151] ;

[0152] in, and These represent the poses of the feature at different times in the LiDAR coordinate system. This pose relationship is used to verify the geometric consistency of the two detections. If the error is within a set threshold, the loop closure detection is considered successful. The detection results are added as additional pose constraints to the optimization model of the global map.

[0153] Step S503: Based on the pose graph optimization model, the real-scene 3D global map is globally optimized using the optimization objective function that minimizes the overall error, thereby eliminating the accumulated error in the real-scene 3D global map and obtaining the updated real-scene 3D model.

[0154] In one implementation of this embodiment, step S503 includes the following steps:

[0155] Step S503a: Determine the optimization objective function based on the relative pose relationship between keyframes, the loop closure detection constraint, and the corresponding weight factor;

[0156] Step S503b: Adjust the pose of all keyframes in the real-world 3D global map according to the optimization objective function to eliminate the accumulated error in the real-world 3D global map.

[0157] In this embodiment, during the global optimization phase, the pose relationships of the map are modeled as a factor graph, where the poses of keyframes... These are the nodes in the graph, representing the relative poses between keyframes. Relationship between the feature object pose provided by loop closure detection These are the edges in the graph. The optimization objective is to minimize the overall error, including pose error between keyframes and error in loop closure detection constraints.

[0158] The objective function for global optimization is expressed as:

[0159] ;

[0160] in, This is the set of constraint edges between keyframes. The set of constraint edges provided for loop closure detection. This represents the weighting factor for loop closure detection constraints. By optimizing the above objective function, the poses of all keyframes are adjusted, correcting the accumulated error in the global map.

[0161] To further enhance the accuracy of the constraints, this embodiment also uses the point cloud information of the feature as a geometric supplement. A plane equation is generated by fitting the point cloud around the feature, described as follows:

[0162] ;

[0163] In global optimization, this planar constraint can be used to verify the geometric consistency of features in different detection methods, further enhancing the reliability of loop closure detection. Furthermore, the fitted feature plane is added as a fixed point to the optimization objective, providing supplementary constraints for global error correction.

[0164] In multi-drone collaborative scenarios, each drone independently constructs a local map, and then global loop closure detection is used to connect overlapping areas between different local maps. For the same feature detected by different drones, the feature's position and geometric constraints are uniformly incorporated into the global optimization to correct the relative poses between the local maps. Assuming local maps... and The poses are respectively and The relative pose relationship between the two is as follows:

[0165] ;

[0166] Ultimately, a consistent global map is generated through global optimization, correcting accumulated errors and improving map accuracy.

[0167] In summary, this embodiment combines semantic and geometric information through global loop closure detection and error correction, and introduces feature-assisted constraints, significantly enhancing the global consistency of the map. The semantic recognition and geometric fitting of features not only improve the reliability of loop closure detection but also provide strong support for global map optimization. This technical solution is particularly suitable for high-precision multi-UAV collaborative mapping in indoor and complex urban environments.

[0168] It is worth mentioning that the above solution in this embodiment can be modified in the following ways to adapt to different scenario requirements:

[0169] 1) The system's adaptability to complex environments can be enhanced by introducing more sensors (such as millimeter-wave radar or depth cameras). For example, combining the high penetration of millimeter-wave radar can solve the failure problem of visual sensors in low-light or strong occlusion environments, further improving the robustness of local map building and global positioning. In addition, visual and LiDAR data can be extended to dynamic scene analysis to adapt to the mapping and updating needs in dynamic target environments.

[0170] 2) In the local map stitching process of this embodiment, the keyframe matching method can be further optimized. For example, an optimization method based on local geometric features can be introduced during keyframe matching to improve the accuracy of point cloud registration by utilizing geometric information in the point cloud (such as corner points, edge points, or planar features). Furthermore, the joint matching of visual feature points (such as the SIFT scale-invariant feature transform algorithm, ORB feature point detection and matching algorithm, etc.) and point cloud feature points can be combined to further enhance the correlation between data from different sensors. In addition, a region-block matching strategy can be adopted to prioritize matching overlapping areas during local point cloud stitching, thereby reducing computational load and improving the efficiency and robustness of local map stitching.

[0171] Therefore, any improvements or modifications made based on the above description should fall within the protection scope of the embodiments of the present invention.

[0172] This embodiment achieves the following technical effects through the above technical solution:

[0173] This embodiment utilizes visual target detection, character recognition, and LiDAR point cloud fusion technology to accurately calculate the location information of features. It also establishes consistency constraints through the perception results of different UAVs, significantly improving the efficiency and accuracy of real-scene 3D mapping. This embodiment constructs local sub-maps by selecting keyframes and point cloud data from several frames before and after them, effectively reducing the computational complexity of map stitching. At the same time, it combines the generalized ICP algorithm to achieve loop closure detection and error correction, optimizing the pose relationship and geometric consistency of the global map, significantly improving the efficiency and accuracy of 3D map construction. It is particularly suitable for map updates and optimization in rapidly changing urban environments.

[0174] Exemplary device

[0175] Based on the above embodiments, the present invention also provides a system for constructing and updating a real-scene 3D model based on crowd-sourced perception of unmanned aerial vehicles, comprising:

[0176] The data acquisition module is used to acquire UAV perception data; wherein, the UAV perception data includes: lidar point cloud data, visual image data, and inertial measurement data;

[0177] The target detection module is used to extract target features and semantic information in the current environment based on the UAV perception data by combining target detection with optical character recognition.

[0178] The local map construction module is used to construct a real-world 3D local map in real time based on the UAV's perception data and using multi-sensor tight coupling technology.

[0179] The local map stitching module is used to stitch together the real-scene 3D local map based on the extracted target features and semantic information, using keyframe selection and semantic auxiliary constraint methods, to obtain a real-scene 3D global map.

[0180] The real-scene 3D model update module is used to eliminate the accumulated errors in the real-scene 3D global map by using a feature-assisted global loop closure detection and pose graph optimization method, so as to obtain an updated real-scene 3D model.

[0181] This embodiment achieves the following technical effects through the above technical solution:

[0182] This embodiment utilizes visual target detection, character recognition, and LiDAR point cloud fusion technology to accurately calculate the location information of features. It also establishes consistency constraints through the perception results of different UAVs, significantly improving the efficiency and accuracy of real-scene 3D mapping. This embodiment constructs local sub-maps by selecting keyframes and point cloud data from several frames before and after them, effectively reducing the computational complexity of map stitching. At the same time, it combines the generalized ICP algorithm to achieve loop closure detection and error correction, optimizing the pose relationship and geometric consistency of the global map, significantly improving the efficiency and accuracy of 3D map construction. It is particularly suitable for map updates and optimization in rapidly changing urban environments.

[0183] Based on the above embodiments, the present invention also provides a terminal, the principle block diagram of which can be as follows: Figure 7 As shown.

[0184] The terminal includes: a processor, a memory, an interface, a display screen, and a communication module connected via a system bus; wherein, the processor of the terminal provides computing and control capabilities; the memory of the terminal includes a computer-readable storage medium and internal memory; the computer-readable storage medium stores an operating system and computer programs; the internal memory provides an environment for the operation of the operating system and computer programs in the computer-readable storage medium; the interface is used to connect to external devices; the display screen is used to display relevant information; and the communication module is used to communicate with a cloud server or other devices.

[0185] When executed by the processor, this computer program is used to implement a method for constructing and updating real-world 3D models based on crowd-sourced perception from drones.

[0186] It will be understood by those skilled in the art that Figure 7 The schematic diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the terminal to which the present invention is applied. A specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0187] In one embodiment, a terminal is provided, comprising: a processor and a memory, the memory storing a program for constructing and updating a real-scene 3D model based on UAV crowd-sensing, the program being executed by the processor to implement the above-described method for constructing and updating a real-scene 3D model based on UAV crowd-sensing.

[0188] In one embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a program for constructing and updating a real-scene 3D model based on UAV crowd-sensing, which, when executed by a processor, is used to implement the above-described method for constructing and updating a real-scene 3D model based on UAV crowd-sensing.

[0189] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, database, or other media used in the embodiments provided by this invention can include both non-volatile and volatile memory.

[0190] In summary, this invention provides a method, system, terminal, and storage medium for constructing and updating a real-scene 3D model based on UAV crowdsourced perception. The method includes: acquiring UAV perception data; extracting target features and semantic information from the current environment based on the UAV perception data using a combination of target detection and optical character recognition; constructing a real-scene 3D local map in real time using multi-sensor tight coupling technology based on the UAV perception data; stitching the real-scene 3D local map together with keyframe selection and semantic-assisted constraint methods based on the extracted target features and semantic information to obtain a real-scene 3D global map; and eliminating accumulated errors in the real-scene 3D global map using feature-assisted global loop closure detection and pose graph optimization methods to obtain an updated real-scene 3D model. This invention achieves the construction and timely updating of real-scene 3D models, improving the efficiency and accuracy of construction and updating.

[0191] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for constructing and updating a real scene three-dimensional model based on crowd-sourced perception of unmanned aerial vehicles, characterized in that, The method comprises the following steps: acquiring unmanned aerial vehicle sensing data; wherein the unmanned aerial vehicle sensing data comprises laser radar point cloud data, visual image data and inertial measurement data; based on the unmanned aerial vehicle sensing data, extracting target feature objects and semantic information in the current environment by a method combining target detection and optical character recognition; based on the unmanned aerial vehicle sensing data, constructing a real scene three-dimensional local map in real time by using a multi-sensor tight coupling technology; wherein the multi-sensor tight coupling technology is that by fusing the laser radar point cloud data, the visual image data and the inertial measurement data, the unmanned aerial vehicle generates a high-precision local map in real time in a complex environment; according to the extracted target feature objects and semantic information, splicing the real scene three-dimensional local map by using a key frame selection and semantic auxiliary constraint method to obtain a real scene three-dimensional global map, comprising: selecting a key frame in the process of constructing the real scene three-dimensional local map; matching the semantic information of the selected key frame, and performing point cloud pose matching according to the target feature objects of the key frame whose semantic matching is successful to obtain the relative pose relationship between the key frames; eliminating the cumulative error in the real scene three-dimensional global map by using a feature object assisted global loop detection and pose graph optimization method to obtain an updated real scene three-dimensional model. 2.The UAV crowd-sourced perception based live 3D model construction and updating method of claim 1, wherein, The method of extracting target feature objects and semantic information in the current environment based on the unmanned aerial vehicle sensing data by combining target detection and optical character recognition comprises the following steps: extracting target object information in the scene from the visual image data by a target detection technology to obtain a target detection result; wherein the target object information comprises a bounding box, a category label and a corresponding confidence of the target object; extracting a bounding box, character content and a corresponding confidence of the character by recognizing the character region in the visual image data by an optical character recognition technology to obtain an optical character recognition result; performing feature screening according to the target detection result and the optical character recognition result to obtain the target feature objects and semantic information in the current environment. 3.The UAV crowd-sourced perception based live 3D model construction and updating method of claim 2, wherein, The feature screening according to the target detection result and the optical character recognition result to obtain the target feature objects and semantic information in the current environment comprises the following steps: determining the confidence of the target detection result and the optical character recognition result according to a preset confidence screening function, and fusing the target detection result and the optical character recognition result that meet the confidence requirement by the overlap degree of the spatial position to obtain the target feature objects and semantic information in the current environment. 4.The UAV crowd-sourced perception based live 3D model construction and update method of claim 1, wherein, The method of constructing a real scene three-dimensional local map in real time based on the unmanned aerial vehicle sensing data by using a multi-sensor tight coupling technology comprises the following steps: synchronizing and calibrating the laser radar point cloud data and the visual image data in space and time to establish a coordinate system conversion relationship therebetween; down-sampling the laser radar point cloud data to obtain down-sampled point cloud data; extracting features from the visual image data to obtain image features containing two-dimensional coordinate information; determining a fusion target function according to the point cloud projection error, the image feature projection error and the coordinate system conversion relationship; Based on the fusion target function, the multi-sensor tight coupling technology is used to fuse the inertial measurement data, the image features and the down-sampled point cloud data, and a sliding window mechanism is used to generate the real scene three-dimensional local map. 5.The UAV crowd-sourced perception based live 3D model construction and update method of claim 1, wherein, According to the extracted target features and semantic information, the real scene three-dimensional local map is spliced by using a key frame selection and semantic auxiliary constraint method to obtain a real scene three-dimensional global map, and the method further comprises: According to the relative pose relationship between the key frames, the relative pose relationship between the corresponding real scene three-dimensional local maps is calculated; According to the relative pose relationship between the real scene three-dimensional local maps and the corresponding unmanned aerial vehicle flight trajectory, all real scene three-dimensional local maps are spliced to obtain the real scene three-dimensional global map. 6.The UAV crowd-sourced perception based live 3D model construction and update method of claim 1, wherein, The feature-assisted global loop detection and pose graph optimization method eliminates the cumulative error in the real scene three-dimensional global map to obtain an updated real scene three-dimensional model, and the method comprises: Based on the global loop detection method, the pose of each key frame and the corresponding target features and semantic information are matched with the existing data in the real scene three-dimensional global map to identify the loop area in the real scene three-dimensional global map; The pose relationship of the target features in the loop area is taken as a loop detection constraint and added to a pose graph optimization model; Based on the pose graph optimization model, a minimum overall error optimization objective function is used to globally optimize the real scene three-dimensional global map to eliminate the cumulative error in the real scene three-dimensional global map and obtain the updated real scene three-dimensional model. 7.The UAV crowd-sourced perception based live 3D model construction and updating method of claim 6, wherein, Based on the pose graph optimization model, a minimum overall error optimization objective function is used to globally optimize the real scene three-dimensional global map, and the method comprises: Based on the relative pose relationship between the key frames, the loop detection constraint and the corresponding weight factor, the optimization objective function is determined; According to the optimization objective function, the poses of all key frames in the real scene three-dimensional global map are adjusted to eliminate the cumulative error in the real scene three-dimensional global map.

8. A system for constructing and updating a real-scene 3D model based on crowd-sourced perception from unmanned aerial vehicles (UAVs), characterized in that, It comprises: A data acquisition module is configured to acquire unmanned aerial vehicle perception data; wherein the unmanned aerial vehicle perception data comprises laser radar point cloud data, visual image data and inertial measurement data; A target detection module is configured to extract target features and semantic information in a current environment based on the unmanned aerial vehicle perception data by using a method combining target detection and optical character recognition; A local map construction module is configured to construct a real scene three-dimensional local map in real time based on the unmanned aerial vehicle perception data by using a multi-sensor tight coupling technology; wherein the multi-sensor tight coupling technology is to fuse laser radar point cloud data, visual image data and inertial measurement data to enable an unmanned aerial vehicle to generate a high-precision local map in real time in a complex environment; The local map stitching module is configured to stitch the real-scene three-dimensional local map according to the extracted target feature and semantic information, using a key frame selection and semantic auxiliary constraint method, to obtain a real-scene three-dimensional global map, including: selecting a key frame in the process of constructing the real-scene three-dimensional local map; matching semantic information of the selected key frame, and performing point cloud pose matching according to a target feature of a key frame for which semantic matching is successful, to obtain a relative pose relationship between the key frames; The real-scene three-dimensional model updating module is configured to eliminate accumulated errors in the real-scene three-dimensional global map based on a feature-assisted global loop detection and pose graph optimization method, to obtain an updated real-scene three-dimensional model.

9. A terminal, characterized by comprising: The real-scene three-dimensional model construction and updating method based on crowd-sourced perception of a UAV includes: A processor and a memory, the memory storing a real-scene three-dimensional model construction and updating program based on crowd-sourced perception of a UAV, the real-scene three-dimensional model construction and updating program based on crowd-sourced perception of a UAV being configured to implement operations of the real-scene three-dimensional model construction and updating method based on crowd-sourced perception of a UAV as claimed in any one of claims 1-7 when executed by the processor.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a real-scene three-dimensional model construction and updating program based on crowd-sourced perception of a UAV, the real-scene three-dimensional model construction and updating program based on crowd-sourced perception of a UAV being configured to implement operations of the real-scene three-dimensional model construction and updating method based on crowd-sourced perception of a UAV as claimed in any one of claims 1-7 when executed by the processor.

Images