Real-time map generation method, device and system and storage medium

Abstract

The invention discloses a real-time map generation method, device and system and a storage medium, and relates to the technical field of image processing. The method comprises the following steps: acquiring a first image frame, and processing the first image frame based on a frame screening mechanism to obtain a second image frame; inputting the second image frame into an ORB-SLAM2 system for thread processing to obtain camera pose information, ground pose parameters and a key frame, and processing the key frame based on an inferior frame rejection mechanism to obtain a third image frame; and integrating the camera pose information, the ground pose parameters and the third image frame into a Map2Dflution mapping system for processing, and generating a real-time map. According to the method and the device, positioning or mapping errors caused by image blurring are reduced, the processing flow is simplified, the calculation complexity is reduced, the influence of inferior frames on the mapping quality is avoided, and the accuracy and the stability of a map are improved.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular, relates to a real-time map generation method, device, system and storage medium. Background Art

[0002] As an important part of modern aviation technology, UAVs (Unmanned Aerial Vehicles) technology has shown wide application potential in many fields. UAVs have become an important tool for map making and spatial data collection due to their flexibility and cost-effectiveness. With the rapid development of sensor technology, computing power and image processing algorithms, real-time UAV map generation technology has become a hot topic of research. It can effectively provide geospatial data to support a variety of applications such as urban planning, environmental monitoring, and agricultural management.

[0003] The core of real-time UAV map generation technology is to quickly and accurately extract geographic information from images captured by UAVs and build maps that reflect the ground reality. The development of this technology relies on advanced technologies such as computer vision, machine learning, sensor fusion, and SLAM (Simultaneous Localization and Mapping).

[0004] However, existing technologies still have limitations in terms of real-time performance, accuracy, robustness, and adaptability to complex environments. In particular, the quality of map generation will be seriously affected in dynamically changing environments or under poor lighting conditions. Images captured during drone flight are easily affected by environmental factors such as lighting changes and excessive wind, resulting in reduced image clarity, which in turn affects the accuracy and reliability of map generation. At the same time, existing map generation methods have limitations in processing speed and are difficult to meet the needs of real-time or near-real-time application scenarios, especially in rapidly changing environments or emergency situations that require rapid response. Summary of the invention

[0005] The purpose of this application is to provide a real-time map generation method, device, system and storage medium, which integrates ORB-SLAM2 and Map2DFusion mapping technology to achieve efficient stitching and map construction of real-time images.

[0006] In order to achieve the above objectives, the solution of this application is:

[0007] In a first aspect, an embodiment of the present application provides a real-time map generation method, comprising:

[0008] Acquire the first image frame, and process the first image frame based on the frame screening mechanism to obtain the second image frame; input the second image frame into the ORB-SLAM2 system for thread processing to obtain camera pose information, ground pose parameters and key frames; based on the poor quality frame elimination mechanism, process the key frames to obtain the third image frame; integrate the camera pose information, ground pose parameters and the third image frame into the Map2DFusion mapping system for processing to generate a real-time map.

[0009] According to the above method of the embodiment of the present application, the following additional technical features may also be provided:

[0010] Furthermore, the frame screening mechanism includes performing edge enhancement on the first image frame through a Laplace operator to highlight key features in the first image frame, quantifying the clarity of the first image frame through a focus measurement method based on the variance after image processing, and screening out the first image frame whose clarity is greater than a preset threshold through a preset threshold to obtain a second image frame.

[0011] Furthermore, the Laplacian operator is applied to the gray value function of the first image frame, and the formula is as follows:

[0012]

[0013] in, represents the Laplace operator; f(x, y) represents the gray value function of the first image frame; and It represents the calculation of the second-order partial derivatives of the gray value function of the first image frame in two independent directions of x and y.

[0014] Furthermore, thread processing includes tracking thread, local mapping thread and closed-loop detection thread; the camera pose information is obtained through the tracking thread; the RANSAC algorithm is introduced in the local mapping thread to obtain the ground pose parameters; and the key frames are output through the closed-loop detection thread.

[0015] Furthermore, the poor quality frame elimination mechanism includes selecting three adjacent frames in the key frame for ORB feature point detection, performing feature matching on the three adjacent frames through feature descriptors, the three adjacent frames include the current frame, the next frame and the next frame, and performing feature point recognition on the three adjacent frames respectively to obtain the number of feature point matches in adjacent frames and the number of feature point matches in alternate frames, and comparing the number of feature point matches in adjacent frames with the number of feature point matches in alternate frames. If the number of feature point matches in adjacent frames is less than the number of feature point matches in alternate frames, the next frame is eliminated and the current frame and the next frame are saved; if the number of feature point matches in adjacent frames is not less than the number of feature point matches in alternate frames, the current frame and the next frame are saved; after the comparison is completed, the current frame is updated to the next frame, and all frames in the key frame are traversed to obtain the third image frame.

[0016] Furthermore, before edge enhancement is performed on the first image frame using the Laplace operator, it is determined whether the first image frame is a color image. If the first image frame is a color image, the color image is converted into a grayscale image; if the first image frame is a grayscale image, the grayscale image is maintained.

[0017] Furthermore, a first image frame is acquired through a monocular camera.

[0018] In a second aspect, an embodiment of the present application provides a real-time map generating device, comprising:

[0019] An image acquisition module, configured to acquire a first image frame;

[0020] The image processing module is configured to process the first image frame based on a frame screening mechanism to obtain a second image frame, input the second image frame into the ORB-SLAM2 system for thread processing, obtain camera pose information, ground pose parameters and key frames, and process the key frames based on a poor quality frame elimination mechanism to obtain a third image frame;

[0021] The map generation module is configured to integrate the camera posture information, the ground posture parameters and the third image frame into the Map2DFusion mapping system for processing to generate a real-time map.

[0022] In a third aspect, an embodiment of the present application provides a real-time map generation system, the system comprising a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the real-time map generation method provided in the first aspect of the embodiment of the present application.

[0023] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement the real-time map generation method as in the first aspect of the embodiment of the present application.

[0024] Compared with the prior art, the real-time map generation method provided by the embodiment of the present application has the following beneficial technical effects:

[0025] In the embodiment of the present application, the edge enhancement of the first image frame is performed by the Laplace operator, which helps to more accurately identify key features in subsequent processing; the image clarity is quantified based on the variance after image processing, and a preset threshold is used to screen to ensure that only image frames with high clarity are further processed, which effectively reduces the positioning or mapping errors caused by image blur; before edge enhancement, the color image is judged and converted into a grayscale image, which not only simplifies the processing flow, but also reduces the computational complexity. At the same time, grayscale images are often more effective in edge detection;

[0026] The embodiment of the present application significantly improves the processing speed by parallel processing of the tracking thread, local mapping thread and closed-loop detection thread of the ORB-SLAM2 system; the tracking thread estimates the camera pose in real time, the local mapping thread builds a local map and introduces the RANSAC algorithm to optimize the ground pose parameters, and the closed-loop detection thread is used to identify and correct the accumulated error to ensure the global consistency of the map;

[0027] The embodiment of the present application uses ORB feature point detection and matching to analyze the number of feature point matches between key frames, and intelligently removes poor quality frames caused by motion blur, underexposure, etc.; this method effectively avoids the impact of poor quality frames on the quality of map construction, and improves the accuracy and stability of the map;

[0028] The embodiment of the present application dynamically updates the current frame and traverses all key frames to ensure that each frame is processed reasonably. The final generated third image frame set is of higher quality, providing a solid foundation for subsequent map construction; the camera pose information, ground pose parameters and the third image frame are integrated into the Map2DFusion mapping system, and the powerful capabilities of the system are used for real-time map generation; this step not only makes full use of the output results of the previous steps, but also gives play to the advantages of the Map2DFusion system in map fusion and optimization, and finally generates a high-quality, real-time map. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram showing a flow chart of a real-time map generation method according to an embodiment of the present application is shown;

[0030] Figure 2 A structural block diagram of a real-time map generating device according to an embodiment of the present application is shown;

[0031] Figure 3 A structural block diagram of a computer device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0032] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. It is to be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some structures related to the present application are shown in the accompanying drawings, rather than all structures. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0033] The terms "including" and "having" and any variations thereof in this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0034] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0035] like Figure 1 As shown, the embodiment of the present application provides a real-time map generation method, comprising the following steps:

[0036] Step 101 , obtaining a first image frame, and processing the first image frame based on a frame screening mechanism to obtain a second image frame.

[0037] The embodiment of the present application obtains the first image frame through a monocular camera and uses the monocular camera as the image data source, which has the advantages of high flexibility and low cost. Although the monocular camera is inferior to the binocular or stereo camera in depth perception, it can still generate a high-quality real-time map through appropriate algorithms and processing procedures.

[0038] The frame screening mechanism includes edge enhancement of the first image frame by the Laplace operator to highlight the key features in the first image frame. The Laplace operator is a second-order differential operator used for edge detection of images. In the present embodiment, the Laplace operator is applied to the gray value function of the first image frame to highlight the key features in the image, such as edges, corners, etc. Edge enhancement helps to more accurately identify these features in subsequent processing, thereby improving the accuracy of camera pose estimation and map construction. The Laplace operator is applied to the gray value function of the first image frame, and the formula is as follows:

[0039]

[0040] in, represents the Laplace operator; f(x, y) represents the gray value function of the first image frame; and It represents the calculation of the second-order partial derivatives of the gray value function of the first image frame in two independent directions of x and y.

[0041] By combining the second-order partial derivative information in these two directions, the edge and texture features in the image can be accurately located. After applying the Laplacian operator, the edge area of ​​the image will show a significant bright or dark response in the result, while the non-edge area has a weaker response. This feature makes the Laplacian operator a powerful tool for image edge detection and detail enhancement.

[0042] Before applying the Laplacian operator, if the first image frame is a color image, it needs to be converted to a grayscale image first, because the Laplacian operator is usually applied to grayscale images, and grayscale images are more effective in edge detection. By converting the grayscale image, the processing flow is simplified, the computational complexity is reduced, and the accuracy of edge detection is improved.

[0043] The clarity of the first image frame is quantified by a focus measurement method based on the variance after image processing, and the first image frame whose clarity is greater than the preset threshold is screened out by a preset threshold to obtain a second image frame.

[0044] The focus measurement method based on the variance after image processing introduced in the embodiment of the present application evaluates the clarity of the image by quantitatively analyzing specific features in the image, such as edges, textures, and high-frequency components. The higher the value, the more clearly the details and edges in the image are discernible, indicating that there are rich detail changes in the image, thus indicating that the image has higher visual clarity; the lower the value, the fewer high-frequency components in the image, the less obvious the details and edges are, and the image looks blurry. This is caused by poor shooting conditions, lens blur, motion blur, or image compression.

[0045] In order to achieve automated image screening, the embodiment of the present application sets a clarity threshold, which is determined by statistical analysis based on a large number of image samples and manual evaluation results. Only when the focus measurement result exceeds this threshold will the image be automatically identified as clear by the system and subsequently saved and processed. This process not only improves the efficiency of image processing, but also reduces the impact of subjectivity by reducing manual intervention.

[0046] Step 102, input the second image frame into the ORB-SLAM2 system for thread processing, obtain camera pose information, ground pose parameters and key frames, and process the key frames based on the poor quality frame elimination mechanism to obtain the third image frame.

[0047] The thread processing of the embodiment of the present application includes a tracking thread, a local mapping thread and a closed-loop detection thread; the camera posture information is obtained through the tracking thread; the RANSAC algorithm is introduced into the local mapping thread to obtain the ground posture parameters; and the key frames are output through the closed-loop detection thread.

[0048] The tracking thread is a critical part of the ORB-SLAM2 system, which is responsible for estimating the camera's pose in real time when each new image frame arrives. Specifically, the tracking thread receives the second image frame from the camera or other image source; it uses the known feature points in the previous keyframe to track the feature points in the current frame, and implements feature matching algorithms, such as matching ORB feature descriptors, to find the corresponding relationship between the feature points in the current frame and the previous keyframe.

[0049] If a sufficient number of feature points are successfully tracked, the tracking thread uses these correspondences to estimate the current pose of the camera by minimizing the reprojection error, i.e. the difference between the actual position of the feature points in the image and the position predicted based on the camera pose.

[0050] The tracking thread also evaluates the success of tracking. If the number of tracked feature points is too small or the tracking quality is poor, for example, due to rapid motion, lighting changes, or occlusion, the tracking fails. If tracking fails or other preset conditions are met, such as the camera moves a sufficient distance or rotates a sufficient angle, the tracking thread will notify the local mapping thread to create a new keyframe. The creation of a keyframe usually involves adding the information of the current frame, including feature points, descriptors, and camera pose, to the map, and performing further optimization and construction of the local map. If tracking is successful, the tracking thread will update the camera pose information and pass it to other threads, such as the local mapping thread and the closed loop detection thread, for further use.

[0051] It should be noted that in this embodiment, the "previous key frame" mentioned refers to the image frame that has been identified as a "key frame" by the system before the currently processed image frame. The key frame is an image frame specially selected by the ORB-SLAM2 system to build and maintain the environment map, containing rich visual information, such as sufficient feature points, and these feature points are evenly distributed in space.

[0052] During feature point tracking, the ORB-SLAM2 system uses known feature points in previous keyframes, including their positions and descriptors, to try to find corresponding feature points in the current frame. This is done through feature matching algorithms, such as using the ORB (Oriented FAST and Rotated BRIEF) feature descriptor for matching. The ORB feature descriptor is an algorithm for image feature point detection and description. It can quickly generate feature point descriptors and allow feature point matching between different images.

[0053] The matching process looks for feature points in the current frame that are most similar to the feature point descriptors in the previous key frame, thereby establishing feature point correspondences between the current frame and the previous key frame. These correspondences are then used to estimate the camera's pose, i.e., the camera's position and orientation in space; in ORB-SLAM2, the selection of key frames is based on a series of criteria, such as the degree of overlap between images, the number and quality of feature points, and consistency with existing maps. Once a frame is selected as a key frame, it is added to the map and used as a reference in subsequent tracking and mapping processes. Therefore, in this embodiment, a "previous key frame" refers to an image frame that precedes the current frame in time and has been identified by the system as being of great value to map construction.

[0054] In the local mapping thread, ORB-SLAM2 not only builds and maintains a local map, but also introduces the RANSAC algorithm to optimize the points in the map and estimate the ground pose parameters. In this embodiment, the RANSAC algorithm is used to perform plane fitting on the point cloud data in ORB-SLAM2. As a widely recognized parameter optimization technology, RANSAC is particularly suitable for accurately estimating model parameters from data containing noise. Through the iterative process of the algorithm, a set of points is successfully randomly extracted from the point cloud, the best fitting plane is calculated, and its error is evaluated. After multiple rounds of iterations, the plane model with the smallest error is selected to accurately derive the normal vector and specific position of the ground. This method not only improves the accuracy of map splicing, but also enhances the visualization of the map.

[0055] The loop detection thread is responsible for detecting and correcting the accumulated pose errors. It does this by identifying similarities between the current keyframe and the previous keyframe, which helps to find loops, where the camera returns to a previously visited location. When a loop is detected, the loop detection thread calculates a pose transformation to correct all pose estimates from the starting point to the loop point.

[0056] After obtaining the key frame, the implementation of this application introduces a poor-quality frame elimination mechanism to further improve data quality and system performance. By selecting three adjacent frames in the key frame for ORB feature point detection, feature matching is performed on the three adjacent frames through feature descriptors. The three adjacent frames include the current frame, the next frame, and the next-next frame. Feature point recognition is performed on the three adjacent frames respectively to obtain the number of feature point matches in adjacent frames and the number of feature point matches in alternate frames. The number of feature point matches in adjacent frames and the number of feature point matches in alternate frames are compared. If the number of feature point matches in adjacent frames is less than the number of feature point matches in alternate frames, the next frame is eliminated and the current frame and the next-next frame are saved; if the number of feature point matches in adjacent frames is not less than the number of feature point matches in alternate frames, the current frame and the next frame are saved; after the comparison is completed, the current frame is updated to the next frame, and all frames in the key frame are traversed. The final frame set is the third image frame. These frames have been screened to remove potential poor-quality frames, thereby improving the accuracy and efficiency of subsequent processing.

[0057] In this way, the ORB-SLAM2 system is not only able to estimate the camera pose and build an environment map in real time, but also to further optimize its performance and data quality through a poor-quality frame rejection mechanism.

[0058] Step 103: Integrate the camera posture information, ground posture parameters and the third image frame into the Map2DFusion mapping system for processing to generate a real-time map.

[0059] Specifically, first make sure you have a dataset containing a series of images taken by the camera, which will be used in the Map2DFusion mapping process. At the same time, you need to make sure that the dataset contains the camera pose information and the ground pose parameters.

[0060] Map2DFusion usually uses configuration files, such as config.cfg, to specify various parameters in the map building process, including camera type, camera parameters, such as resolution and internal parameters, initial GPS coordinates, etc. The camera pose information is usually obtained through a SLAM (Simultaneous Localization and Mapping) system, and the ORB-SLAM2 system is used in the embodiment of this application. These pose information include the position (translation) and attitude (rotation) of the camera, usually expressed in the form of quaternions or rotation matrices.

[0061] In the ORB-SLAM2 system, the pose information of all key frames is saved by modifying the source code or using the provided API and formatted into the format required by Map2DFusion. The ground pose parameters specify the position and direction of the ground in the map. In the Map2DFusion system, these parameters are used to adjust the vertical direction and horizontal plane of the map to ensure that the generated map matches the actual ground. These parameters are manually set or automatically obtained based on the ground estimate provided by the ORB-SLAM2 system.

[0062] The camera pose information is integrated with the corresponding image frames, the timestamp of each image frame is matched with its corresponding pose information, and they are provided as input to the Map2DFusion system. In the Map2DFusion system, these image frames and pose information will be used to construct local maps and global maps, and to optimize and fuse maps.

[0063] In the Map2DFusion system, the generation of real-time maps is a continuous process. The system continuously updates the map based on the input image frames and pose information, and optimizes and fuses it to eliminate accumulated errors. The Map2DFusion system will eventually output the generated real-time map, usually a two-dimensional raster map or point cloud map, which contains all the key features and structural information in the environment.

[0064] The output map can be used in a variety of application scenarios, such as navigation, path planning, and obstacle detection. Furthermore, in order to better understand and analyze the generated map, visualization tools such as Rviz can be used to display the real-time update of the map. This helps to verify the performance and accuracy of the Map2DFusion system and find potential problems and improvements.

[0065] The embodiments of the present application also conducted a series of ablation experiments, comparing the method provided by the present application with a method that only performs a frame screening mechanism or a poor-quality frame removal mechanism, and an original method without any improvement.

[0066] The data sets used for testing in this application are phantom3-village and phantom3-freeway. These two data sets contain rich scene elements and dynamic changes, which can effectively evaluate and demonstrate the performance of the system in different environments. At the same time, two processing methods are also implemented in the data sets in the embodiments of this application: one is to apply Gaussian blur technology to some images, and the other is to insert irrelevant frames in the data sets. Then, the performance of the system is evaluated through comparative experiments.

[0067] 5, 10, and 15 images were selected for Gaussian blur, and 5 irrelevant frames were inserted to simulate the image blur and mis-shooting encountered by drones in actual shooting. When the phantom3-village dataset was selected, its data indicators are shown in Tables 1 to 3 below:

[0068] Table 1

[0069]

[0070] Table 2

[0071]

[0072] Table 3

[0073]

[0074] Through data analysis, it can be found that the frame screening mechanism performs better in removing low-quality frames, and its effect is better than the poor-quality frame removal mechanism. At the same time, the reduction in the number of key frames directly leads to a reduction in map construction time, which further confirms the importance of key frame selection methods in improving system efficiency. In the process of map construction using the original unoptimized method, there are significant defects in the fineness of the map. These problems are mainly attributed to the distortion of the images captured by the drone during flight, which in turn affects the quality of the final map construction.

[0075] After the introduction of the frame filtering mechanism, its map construction performance has been significantly improved. This mechanism successfully removes distorted and blurred images, thereby significantly improving the integrity and continuity of the map and effectively solving the fault problem that existed before. The method of building the map with only the poor-quality frame removal mechanism did not meet expectations despite the implementation of the poor-quality frame removal mechanism. This mechanism performs frame filtering based on the number of feature point matches between adjacent frames. It mainly plays a role in improving the performance of high-quality frames, but has limited improvement on the overall mapping effect. Therefore, faults still occurred during the mapping process.

[0076] After adding the frame screening mechanism and the low-quality frame elimination mechanism on the basis of the original method, through the synergy of these two mechanisms, the method of this application has achieved a significant improvement in the mapping effect. Compared with the original method, the improved method can generate a smoother, continuous and non-fractured map. In addition, due to the elimination of low-quality frames, the accuracy of mapping has also been significantly improved. This improvement not only improves the quality and accuracy of mapping, but also enhances the robustness and reliability of the system.

[0077] When the phantom3-freeway dataset is selected, its data indicators are shown in Tables 4 to 6:

[0078] Table 4

[0079]

[0080] Table 5

[0081]

[0082]

[0083] Table 6

[0084]

[0085] Through data analysis, both the frame filtering mechanism and the low-quality frame elimination mechanism can exclude low-quality frames, but the frame filtering mechanism has a better elimination effect. At the same time, as the number of key frames decreases, the time required for the map construction process is also reduced accordingly, which not only confirms the key role played by the key frame selection method in improving the overall performance, but also provides an empirical basis for further performance optimization.

[0086] Taking into account that the slight difference in the number of blurred frames does not significantly change the mapping results, the embodiment of the present application selects a case with 10 blurred frames and 5 irrelevant frames to demonstrate, in order to facilitate a deeper analysis and understanding of the mapping process. When the dataset phantom3-freeway was selected, when the original method was used for the map creation task, there was an obvious lack of detail expression in the map, the fracture problem at the seams was particularly prominent, and the road splicing did not appear smooth enough. These defects can be mainly traced back to the distortion of the images captured by the drone when performing flight missions, which ultimately had a negative impact on the quality of map construction.

[0087] After the introduction of the frame filtering mechanism, the efficiency and quality of map construction have been significantly improved. This mechanism effectively eliminates distorted or unclear images, significantly enhances the integrity and coherence of the map, and successfully solves the fault problem that appeared previously. However, in some specific areas of the map, the accuracy of road connections needs to be further optimized. Despite these minor issues, overall, its performance is worthy of recognition.

[0088] After the introduction of the low-quality frame removal mechanism, although the mapping quality has improved, it still has not fully achieved the expected goal. This mechanism selects frames by evaluating the number of feature point matches between adjacent frames, but this method cannot completely eliminate all low-quality frames. It performs well in improving the performance of high-quality frames, but has limited effect on improving the overall mapping effect. Therefore, the fault problem still exists during the mapping process.

[0089] The introduction of frame screening and poor-quality frame removal mechanisms based on the original method significantly improves the results of map generation. Compared with the original method, the improved method can create smoother, continuous and seamless maps. In addition, by removing poor-quality frames, the accuracy of the map is also significantly improved. These improvements not only improve the quality and accuracy of map generation, but also enhance stability and credibility.

[0090] The real-time map generation method provided in the embodiment of the present application shows excellent smoothness and high image quality in orthophoto generation. In the comparative analysis, the original method and the software PTGui and PSG 3D respectively show problems of uneven edges and road cracks. In comparison, the mapping effect of the present application in these areas is better, achieving smooth transitions and connections, and at the same time being more refined and accurate in detail accuracy.

[0091] In terms of real-time performance, this application has shown significant efficiency advantages. Through comparative analysis, this application selected the phantom3-village dataset as the test benchmark. In the actual test process, the original method took 31 seconds to build the orthophoto, while this application took 32 seconds to build the orthophoto, which is much lower than other existing technologies. PTGui software took 10 minutes and 28 seconds on the same dataset, while PSG 3D software took 25 minutes and 42 seconds to complete the same task.

[0092] In the comparative analysis, the software PTGui and PSG 3D had obvious roughness or misalignment problems at road connections, and the original method also showed road alignment problems. In comparison, the mapping effect of this application is better in these areas, achieving smooth transitions and connections, and more refined and accurate in detail accuracy.

[0093] This application selected the phantom3-freeway dataset as the test benchmark. In terms of real-time performance, the method of this application showed a significant efficiency advantage. Through comparative analysis, in the actual test process, the original method took 33 seconds to build an orthophoto, while this application took 34 seconds to build an orthophoto, which is much lower than the time taken by other existing technologies. PTGui software took 12 minutes and 07 seconds on the same dataset, while PSG 3D software took 30 minutes and 23 seconds to complete the same task.

[0094] like Figure 2 As shown, the embodiment of the present application provides a real-time map generation device, including an image acquisition module 201, an image processing module 202 and a map generation module 203, wherein:

[0095] The image acquisition module 201 is configured to acquire a first image frame;

[0096] The image processing module 202 is configured to process the first image frame based on a frame screening mechanism to obtain a second image frame, input the second image frame into the ORB-SLAM2 system for thread processing, obtain camera pose information, ground pose parameters and key frames, and process the key frames based on a poor quality frame elimination mechanism to obtain a third image frame;

[0097] The map generation module 203 is configured to integrate the camera posture information, the ground posture parameters and the third image frame into the Map2DFusion mapping system for processing to generate a real-time map.

[0098] The real-time map generation device in the embodiment of the present application can be a computer device, or a component in the computer device, such as an integrated circuit or a chip. The computer device can be a terminal, or a device other than a terminal. Exemplarily, the computer device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted computer device, a mobile Internet device (Mobile Internet Device, MID), an ultra-mobile personal computer (Ultra-Mobile Personal Computer, UMPC), a netbook or a personal digital assistant (Personal Digital Assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (Personal Computer, PC), etc., which is not specifically limited in the embodiment of the present application.

[0099] The real-time map generation device provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented in the embodiment of the real-time map generation method are not described here.

[0100] The present application also provides a computer device, such as Figure 3 As shown, the computer device includes a processor 301 and a memory 302. The memory 302 stores programs or instructions that can be run on the processor 301. When the program or instruction is executed by the processor 301, the various steps of the above-mentioned real-time map generation method are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0101] It should be noted that the computer device in the embodiment of the present application includes the above-mentioned mobile computer device and non-mobile computer device.

[0102] The memory 302 can be used to store software programs and various data. The memory 302 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 302 may include a volatile memory or a non-volatile memory, or the memory 302 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 302 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0103] The processor 301 may include one or more processing units; optionally, the processor 301 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 301.

[0104] The embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned real-time map generation method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0105] The embodiment of the present application also provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned real-time map generation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0106] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0107] The embodiment of the present application also provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned real-time map generation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0108] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0109] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A real-time map generation method, characterized in that: The method comprises: Acquire a first image frame, and process the first image frame based on a frame screening mechanism to obtain a second image frame; Input the second image frame into the ORB-SLAM2 system for thread processing to obtain camera pose information, ground pose parameters and key frames, and process the key frames based on a poor quality frame elimination mechanism to obtain a third image frame; The camera posture information, ground posture parameters and the third image frame are integrated into the Map2DFusion mapping system for processing to generate a real-time map.

2. The real-time map generation method according to claim 1, characterized in that: The frame filtering mechanism includes: The first image frame is edge enhanced by using the Laplace operator to highlight key features in the first image frame, the clarity of the first image frame is quantified by a focus measurement method based on the variance after image processing, and the first image frame with a clarity greater than the preset threshold is screened out by a preset threshold to obtain a second image frame.

3. The real-time map generation method according to claim 2, characterized in that: The Laplace operator is applied to the gray value function of the first image frame, and the formula is as follows: in, represents the Lagrange operator; f(x, y) represents the gray value function of the first image frame; and It represents the calculation of the second-order partial derivatives of the gray value function of the first image frame in two independent directions of x and y.

4. The real-time map generation method according to claim 1, characterized in that: The thread processing includes a tracking thread, a local mapping thread and a closed-loop detection thread; the camera posture information is obtained through the tracking thread; the RANSAC algorithm is introduced into the local mapping thread to obtain the ground posture parameters; and the key frame is output through the closed-loop detection thread.

5. The real-time map generation method according to claim 1, characterized in that: The poor quality frame elimination mechanism includes: Select three adjacent frames in the key frame for ORB feature point detection, perform feature matching on the three adjacent frames through feature descriptors, the three adjacent frames include a current frame, a next frame and a next-next frame, perform feature point recognition on the three adjacent frames respectively, and obtain the number of feature point matches of adjacent frames and the number of feature point matches of alternate frames, compare the number of feature point matches of adjacent frames with the number of feature point matches of alternate frames, if the number of feature point matches of adjacent frames is less than the number of feature point matches of alternate frames, remove the next frame and save the current frame and the next-next frame; if the number of feature point matches of adjacent frames is not less than the number of feature point matches of alternate frames, save the current frame and the next frame; after the comparison is completed, update the current frame to the next frame, traverse all frames in the key frame, and obtain the third image frame.

6. The real-time map generation method according to claim 2, characterized in that: Before edge enhancement is performed on the first image frame by using the Laplace operator, it is determined whether the first image frame is a color image. If the first image frame is a color image, the color image is converted into a grayscale image; if the first image frame is a grayscale image, the grayscale image is maintained.

7. The real-time map generation method according to claim 1, characterized in that: The first image frame is acquired through a monocular camera.

8. A real-time map generating device, characterized in that: The device comprises: An image acquisition module, configured to acquire a first image frame; The image processing module is configured to process the first image frame based on a frame screening mechanism to obtain a second image frame, input the second image frame into the ORB-SLAM2 system for thread processing, obtain camera pose information, ground pose parameters and key frames, and process the key frames based on a poor quality frame elimination mechanism to obtain a third image frame; The map generation module is configured to integrate the camera posture information, ground posture parameters and the third image frame into the Map2DFusion mapping system for processing to generate a real-time map.

9. A real-time map generation system, the system comprising a processor and a memory, wherein a computer program is stored in the memory, characterized in that: The computer program is loaded and executed by the processor to implement the real-time map generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, wherein a computer program is stored in the storage medium, characterized in that: When the computer program is executed by a processor, it is used to implement the real-time map generation method according to any one of claims 1 to 7.

Images