Point cloud map repairing method, electronic device and storage medium

Abstract

The application discloses a point cloud map repairing method, an electronic device and a storage medium, and the method comprises the following steps: determining a repairing key frame in a map according to a current state of each frame; taking the repairing key frame and / or a frame related to the repairing key frame as a to-be-repaired key frame; and performing an optimization operation on the to-be-repaired key frame to obtain a repaired point cloud map. According to the application, the repairing key frame in the map is determined according to the current state of each frame, and then the repairing key frame is optimized, so that the repairing of local ghosting and misplacement of the point cloud map can be realized, and the quality of the point cloud map is ensured.

Description

Technical Field

[0001] This invention relates to the field of point cloud map technology, and in particular to a point cloud map repair method, electronic device, and storage medium. Background Technology

[0002] In the field of autonomous driving, Level 4 and above autonomous driving are generally considered computer-driven solutions. Level 4 autonomous driving requires intelligent perception, localization, decision-making, and planning technologies to achieve advanced autonomous driving functions, and these modules all rely on data input from high-definition maps. High-definition maps are generally considered indispensable for Level 4 autonomous driving.

[0003] High-precision maps are typically produced based on laser point cloud data. High-precision map acquisition vehicles use onboard laser sensors, along with data from their inertial navigation and satellite navigation, to construct a 3D point cloud of the target road. Then, map annotators mark traffic elements such as road surfaces, curbs, intersections, and traffic lights on the point cloud or a bird's-eye view of it.

[0004] Throughout the production process, accurately and efficiently constructing large-scale point cloud data is an unavoidable technical challenge in high-precision map production. Because it is difficult to find real-world 3D ground truth values, the point cloud quality of high-precision maps is difficult to guarantee effectively. Therefore, the acquired high-precision maps need to be verified, and errors need to be corrected.

[0005] Existing high-precision map verification methods focus on standardized data collection beforehand and verification testing afterward. However, if problems arise during the verification phase, there is still a lack of feedback and correction mechanisms from the back end to the front. This leads to issues with the final point cloud map... Figure 1 If a problem occurs, the entire production process must be restarted from scratch, making map quality control extremely cumbersome. Common issues include: in a specific area of ​​the map, due to jitter and instability in satellite navigation signals, the point cloud may become distorted or interlaced, leading to positioning problems or information corruption across the entire map. The larger the map area and the more complex the scene, the higher the probability of this problem occurring. Maps used by L4 passenger vehicles, in particular, can easily reach hundreds of kilometers in distance, making them prone to problems at individual intersections. Currently, it is difficult to handle all possible map problems through automated algorithms; for example, the local misalignment or overlap issues that may exist in large-scale maps remain unresolved. Summary of the Invention

[0006] The embodiments of the present invention aim to solve at least one of the above-mentioned technical problems. The embodiments of the present invention can correct problems such as local ghosting and misalignment in point cloud maps.

[0007] In a first aspect, embodiments of the present invention provide a point cloud map repair method, comprising: a keyframe repair step: in response to an inspector adjusting the pose of a keyframe in a visual operation interface, updating the pose of the keyframe to obtain a repaired keyframe, wherein the visual operation interface loads a point cloud map to be repaired, and the visual operation interface allows the inspector to modify the data of any keyframe; and a loop closure detection and optimization step: in response to the inspector selecting a loop closure marking instruction and an optimization instruction in the visual operation interface, using the repaired keyframe and the pose of the repaired keyframe as a manual loop closure, and initiating loop closure detection and optimization within a preset range area of ​​the repaired keyframe.

[0008] In a second aspect, embodiments of the present invention provide an electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the point cloud map repair methods described above.

[0009] Thirdly, embodiments of the present invention provide a storage medium storing one or more programs including execution instructions, the execution instructions being readable and executable by electronic devices (including but not limited to computers, servers, or network devices, etc.) to perform any of the point cloud map repair methods described above.

[0010] Fourthly, embodiments of the present invention also provide a computer program product, the computer program product including a computer program stored on a storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, causing the computer to perform any of the above-described point cloud map repair methods.

[0011] This invention addresses issues such as local ghosting and misalignment in point cloud maps by identifying keyframes to be repaired in the point cloud map, applying strong loop closure constraints to these keyframes, or performing small-scale loop closure detection on the area near two selected keyframes to be repaired, removing the influence of other factors in that area, and performing factor graph optimization on the entire map. This ensures the quality of the point cloud map. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of an embodiment of the point cloud map repair method of the present invention; Figure 2 This is a flowchart of another embodiment of the point cloud map repair method of the present invention; Figure 3 A flowchart illustrating another embodiment of the point cloud map repair method of the present invention. Figure 4 A software operation interface diagram of a point cloud map repair method provided in an embodiment of the present invention; Figure 5 This is a keyframe restoration diagram of a point cloud map restoration method provided in an embodiment of the present invention; Figure 6 This is a newly added manually constrained pose graph in a point cloud map repair method provided in an embodiment of the present invention; Figure 7 This is a block diagram of a point cloud map repair device provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of an embodiment of the electronic device of the present invention. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0016] Those skilled in the art will recognize that embodiments of this application can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0017] For ease of understanding, the technical terms used in this application are explained below: The term "mobile device" as used in this application includes, but is not limited to, vehicles with six levels of autonomous driving technology, L0-L5, as defined by the Society of Automotive Engineers International (SAE International) or the Chinese national standard "Classification of Driving Automation for Automobiles".

[0018] In some embodiments, the mobile device may be a vehicle or robot having various functions, such as: (1) Passenger carrying function, such as family cars, buses, etc.; (2) Cargo carrying function, such as ordinary trucks, box trucks, trailer trucks, enclosed trucks, tank trucks, flatbed trucks, container trucks, dump trucks, special structure trucks, etc.; (3) Tool functions, such as logistics delivery vehicles, automated guided vehicles (AGVs), patrol vehicles, cranes, hoists, excavators, bulldozers, loaders, road rollers, loaders, off-road engineering vehicles, armored engineering vehicles, sewage treatment vehicles, sanitation vehicles, vacuum trucks, floor scrubbers, water sprinkler trucks, sweeping robots, food delivery robots, shopping guide robots, lawnmowers, golf carts, etc. (4) Entertainment functions, such as recreational vehicles, amusement park automatic driving devices, balance bikes, etc.; (5) Special rescue functions, such as fire trucks, ambulances, power repair vehicles, and engineering emergency rescue vehicles.

[0019] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0020] This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, elements, data structures, etc., that perform a specific task or implement a specific abstract data type. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0021] In this invention, terms such as "module," "device," and "system" refer to relevant entities applied to a computer, such as hardware, combinations of hardware and software, software, or software in execution. More specifically, for example, an element can be, but is not limited to, a process running on a processor, a processor, an object, an executable element, an execution thread, a program, and / or a computer. Furthermore, an application program or script running on a server, and the server itself, can also be an element. One or more elements may be in an execution process and / or thread, and elements may be localized on a single computer and / or distributed across two or more computers, and may be run on various computer-readable media. Elements can also communicate via local and / or remote processes based on signals having one or more data packets, for example, signals from data interacting with another element in a local system, a distributed system, and / or interacting with other systems via signals over a network on the Internet.

[0022] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0023] This invention provides a point cloud map repair method that can be applied to electronic devices. The electronic device can be a computing device such as a computer with strong computing power; this invention is not limited to this. The goal of loop closure detection is: if the camera actually passes through the same location, then our estimated trajectory should also pass through the same location. Specifically, if we detect that the camera has passed through the same location, we can add constraints between the current pose and the camera pose several frames ago. These constraints, represented by a Lie group, should be very close to the identity matrix. By adding such an edge (pose constraint), the drift caused by the intermediate frames can be directly "smoothed out."

[0024] Specifically, in the mapping phase of autonomous driving, we typically instruct the data collection vehicle to circle a given area a certain number of times to cover the entire data collection range. Suppose we extract features at the front end, then ignore the feature points, and optimize the entire trajectory using a pose graph at the back end. The front end only provides local pose constraints, such as x1-x2, x2-x3, etc. However, because the estimation of x1 has errors, and x2 is determined based on x1, and x3 is determined by x2, and so on, the errors accumulate, causing the optimization result at the back end to gradually become inaccurate. In this application scenario, we should ensure that the optimized trajectory is consistent with the actual location. When we actually pass through the same location, the estimated trajectory must also pass through the same point.

[0025] While the backend can estimate the maximum a posteriori error, the saying "a good model can't withstand bad data" applies. With only adjacent keyframe data, there's not much we can do, and we can't eliminate accumulated errors. However, the loop closure detection module can provide constraints with a longer time interval than adjacent frames: for example, pose transformations between x1 and x100. Why are there constraints between them? This is because we perceive that the camera has passed through the same location and collected similar data. The key to loop closure detection is how to effectively detect this phenomenon. If we can successfully detect it, we can provide more effective data for the backend pose graph, leading to better estimations, especially a globally consistent estimate. Since the pose graph can be viewed as a point mass-spring system, loop closure detection is equivalent to adding an extra spring to the image, improving system stability. Readers can also intuitively imagine that the loop closure "pulls" the edge with accumulated error to the correct position—if the loop itself is correct.

[0026] Please refer to Figure 1 This illustrates a point cloud map repair method provided by an embodiment of the present invention.

[0027] like Figure 1 As shown, in step 101, the repair keyframes in the map are determined based on the current state of each frame. In step 102, the repair keyframe and / or frames related to the repair keyframe are taken as keyframes to be repaired; In step 103, the keyframes to be repaired are optimized to obtain the repaired point cloud map.

[0028] In this embodiment, for step 101, the point cloud map repair device determines the repair key frame in the map based on the current state of each frame. For example, the map may be affected by factors such as the collection route, GPS signal, and road overlap, which may cause point cloud misalignment. If it is found that a certain frame has an offset or misalignment, this key frame can be identified as the repair key frame in the visual operation interface, and the selected point cloud will be highlighted. The point cloud map repair device can load the point cloud map according to the time sequence of map collection and display the point cloud map construction process.

[0029] Then, for step 102, the point cloud map repair device will use the repair key frame and / or the frame related to the repair key frame as the key frame to be repaired. For example, if there is an offset or misalignment in a single frame data, the single frame can be selected as the key frame to be repaired in the visualization interface, and the position and orientation of the key frame to be repaired can be adjusted. If there is misalignment and ghosting in the map, two ghosting key frames can be selected as key frames to be repaired, and then manually marked as loop closure areas.

[0030] Finally, for step 103, the point cloud map repair device performs optimization operations on the key frames to be repaired to obtain the repaired point cloud map. For example, it applies strong loop closure constraints to the key frames to be repaired, or performs small-scale loop closure detection on the area near the two selected key frames to be repaired, then removes the influence of other factors in the area and performs factor map optimization of the whole map.

[0031] The method in this embodiment determines the repair keyframes in the map based on the current state of each frame, and then optimizes the repair keyframes, thereby achieving the repair of problems such as local ghosting and misalignment in point cloud maps and ensuring the quality of point cloud maps.

[0032] In the method of the above embodiments, determining the repair keyframes in the map based on the current state of each frame includes: If the current state of the current frame is misaligned, the current frame is identified as the key frame for repair. For example, if there is an offset or misalignment in a single frame of data, the single frame can be selected as the key frame to be repaired in the visualization interface, and the position and orientation of the key frame to be repaired can be adjusted.

[0033] The method in this embodiment can achieve local repair of point cloud maps by determining repair key frames.

[0034] In the method of the above embodiments, optimizing the keyframes to be repaired to obtain the repaired map includes: In response to the user's pose adjustment of the keyframe to be repaired, the pose of the keyframe to be repaired is updated. For example, if the user finds that a certain frame is offset or misaligned, the user can fine-tune this keyframe in the visual operation interface to adjust the position and orientation of the keyframe. In a specific example, only one keyframe needs to be operated for a region, instead of fine-tuning each keyframe.

[0035] The method in this embodiment optimizes the point cloud map repair algorithm by adjusting and updating the pose of the key frame to be repaired, thereby making the operation simpler.

[0036] In some optional embodiments, determining the repair keyframes in the map based on the current state of each frame includes: If the current state of the current frame is overlapping, then the current frame is determined as the key frame for repair. The step of using the repair keyframe and / or frames related to the repair keyframe as the keyframe to be repaired includes: Keyframes with a similarity higher than a preset threshold to the keyframes to be repaired are identified as frames related to the keyframes to be repaired. The keyframes to be repaired and the frames related to the keyframes to be repaired are designated as keyframes to be repaired.

[0037] For example, if there is ghosting on the map, two ghosting keyframes can be selected as keyframes to be repaired, and then manually marked as loop regions.

[0038] Please refer to Figure 2 The flowchart illustrates another point cloud map repair method provided in an embodiment of this application. This flowchart primarily defines the steps of the above embodiment's process of "optimizing the keyframes to be repaired to obtain the repaired point cloud map".

[0039] like Figure 2 As shown, in step 201, in response to the user marking the keyframe to be repaired as a loop, loop detection optimization is initiated within a preset range of the keyframe to be repaired; In step 202, the similarity of the keyframes to be repaired is added to the entire map as a manual constraint, and the entire map is optimized based on at least one manual constraint.

[0040] In this embodiment, for step 201, the point cloud map repair device responds to the user marking loop closures in the key frame to be repaired and initiates loop closure detection optimization within a preset range of the key frame to be repaired. For example, if there is an offset or misalignment in a single frame of data, the key frame to be repaired can be fine-tuned in the visual operation interface to adjust the position and orientation of the key frame. If there is misalignment and ghosting in the map, two key frames to be repaired and their orientations can be selected as artificial loop closures, and loop closure detection can be initiated in the surrounding area.

[0041] Then, for step 202, the point cloud map repair device adds the similarity of the keyframes to be repaired as a manual constraint to the entire map, and optimizes the entire map based on at least one manual constraint. For example, it obtains the user's optimization instructions after adding new loop constraints to the point cloud map through the visual operation interface. After obtaining the user's optimization instructions, the marked manual loops are added to the point cloud map optimization problem, and then the whole map is optimized and refreshed. In a specific embodiment, the user can add new loop constraints to the point cloud map through the visual operation interface to achieve shape correction of local areas. If there is ghosting on multiple points of the map, the ghosting keyframes of multiple points can be repeatedly fine-tuned until the ghosting disappears. The visual operation interface can display the point cloud map after the keyframe repair step, the point cloud map after the loop detection optimization step, and the area with added manual loops when displaying the point cloud map.

[0042] The method in this embodiment starts loop closure detection optimization within a preset range of the keyframes to be repaired. Then, the similarity of the keyframes to be repaired is added to the entire map as a manual constraint, and the entire map is optimized based on at least one manual constraint. This can achieve the repair of problems such as local ghosting and misalignment in point cloud maps, thus ensuring the quality of point cloud maps.

[0043] Please refer to Figure 3 The flowchart illustrates another point cloud map repair method provided in an embodiment of this application. This flowchart mainly describes the process... Figure 2 The flowchart further defines the steps of the process of “point cloud map optimization adopts factor graph optimization, and loop closure detection optimization is started within the preset range of the key frame to be repaired”, wherein the map optimization adopts factor graph optimization.

[0044] like Figure 3 As shown, in step 301, based on the results of loop closure detection, the similarity of the keyframe to be repaired is used as an artificial constraint to apply an artificial loop closure detection factor to the point cloud map in order to control the shape of the point cloud within the preset range. In step 302, the influence of other factors within the preset range area is removed, and factor graph optimization is performed only based on the artificial loop closure detection factor.

[0045] In this embodiment, for step 301, the point cloud map repair device, based on the results of loop closure detection, applies artificial loop closure detection factors to the point cloud map as artificial constraints to control the point cloud shape within a preset range, using the similarity of the key frames to be repaired as an artificial constraint. For example, in map optimization algorithms, in order to comprehensively consider the influence of various sensors, factor graph optimization is used as a theoretical tool for map building optimization algorithms. Furthermore, in order to ensure the shape of the map's intersection areas, the loop closure detection algorithm applies many loop closure detection factors to the map to control the point cloud shape of these areas.

[0046] Then, for step 302, the point cloud map repair device removes the influence of other factors within the preset range area and performs factor graph optimization based only on manually detected loop closure factors. For example, if the overlapping area of ​​the point cloud map is not obvious, loop closure constraints can be applied to the repair key frame given by the user, and a small-scale loop closure detection can be performed near the repair key frame. After that, the influence of other factors in the area near the repair key frame is removed, and finally, factor graph optimization of the entire map is performed.

[0047] The method in this embodiment removes the influence of other factors in the region and performs factor graph optimization of the entire map, thereby repairing local areas of the point cloud map and ensuring the quality of the point cloud map.

[0048] Please refer to Figure 4The diagram shows the software operation interface of a point cloud map repair method provided in an embodiment of the present invention.

[0049] like Figure 4 As shown, the software can perform the following functions: 1. Visualize point cloud maps to facilitate user viewing of mapping results. It can display the entire map construction process in chronological order of data collection; additionally, it can dynamically load and unload large-scale maps, enabling the loading of maps for extremely large scenes.

[0050] 2. Users can view single-frame sensor data of the map. Since a map is composed of many keyframe data, users can easily analyze the mapping situation in the actual scene by viewing single-frame data.

[0051] 3. If inspectors find any offset or misalignment in a single frame of data, they can fine-tune the keyframe in the data panel on the left, adjusting its position and orientation. This is a way to manually modify single-frame data.

[0052] 4. If map misalignment or ghosting is detected, two ghosting keyframes can be selected and manually marked as loop closure areas. This is an interactive loop closure annotation function, which is also the main function described in this invention.

[0053] Please refer to Figure 5 The diagram shows a keyframe restoration diagram of a point cloud map restoration method provided in an embodiment of the present invention.

[0054] like Figure 5 As shown, when the data collection vehicle was collecting data from the left and right roads, a misalignment of the point cloud occurred in the vegetation area in the center of the road due to poor GPS signal. In this example, the two central lines should have overlapped, but the mapping algorithm had an error, causing the map point cloud to shift by approximately 10-20 centimeters. The point cloud map repair process is as follows: Step 1: Select one of the keyframes, such as the gray area on the left side of the image before correction, and make adjustments; during the adjustment, the software will highlight the currently selected point cloud in white. Step 2: Adjust the point cloud to a satisfactory position, such as... Figure 5 As shown on the right, there will be no ghosting in these two keyframes, but the changes are limited to these two keyframes. Step 3: Click "Mark Loopback". The software will then treat these two keyframes and their poses as artificial loopbacks and initiate loopback detection in the surrounding area. Step 4: Click "Execute Optimization". The software will add the manual constraints you just added to the map optimization problem, perform full map optimization, and refresh the pose of the entire map.

[0055] Step 5: Repeat the above steps until the map no longer shows obvious ghosting.

[0056] Please refer to Figure 6 This illustrates a newly added artificially constrained pose graph in a point cloud map repair method provided by an embodiment of the present invention.

[0057] like Figure 6 As shown, the operator added three manual constraints to the large map. Through interaction with the software, new loop constraints were added to the map, achieving shape correction of local areas.

[0058] Please refer to Figure 7 The diagram shows a schematic representation of a point cloud map repair device according to an embodiment of the present invention.

[0059] like Figure 7 As shown, the point cloud map repair device 700 includes a status determination module 710, an expansion module 720, and a repair module 730.

[0060] The state determination module 710 is configured to determine the repair key frame in the map based on the current state of each frame; the extension module 720 is configured to take the repair key frame and / or frames related to the repair key frame as the key frame to be repaired; and the repair module 730 is configured to perform optimization operations on the key frame to be repaired to obtain the repaired point cloud map.

[0061] It should be understood that Figure 7 The modules and references described in the document Figure 1 , Figure 2 and Figure 3 The steps described in the text correspond to those in the method described above. Therefore, the operations, features, and corresponding technical effects described above also apply to the method described in the text. Figure 7 The various modules in the document will not be described in detail here.

[0062] It is worth noting that the modules in the embodiments of this disclosure are not intended to limit the solutions of this disclosure. For example, the state determination module can be described as a module that determines the repair keyframes in the map based on the current state of each frame. In addition, related functional modules can also be implemented by hardware processors. For example, the state determination module can also be implemented by a processor, which will not be elaborated here.

[0063] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of combined actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, as some steps can be performed in other orders or simultaneously according to the present invention. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention. In the above embodiments, the descriptions of each embodiment have their own emphasis; for parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0064] In some embodiments, the present invention provides a non-volatile computer-readable storage medium storing one or more programs including execution instructions, which can be read and executed by electronic devices (including but not limited to computers, servers, or network devices, etc.) to perform any of the point cloud map repair methods described above.

[0065] In some embodiments, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform any of the above-described point cloud map repair methods.

[0066] In some embodiments, the present invention also provides an electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a point cloud map repair method.

[0067] Figure 8 This is a schematic diagram of the hardware structure of an electronic device for performing a point cloud map repair method according to another embodiment of this application, as shown below. Figure 8 As shown, the device includes: One or more processors 810 and memory 820, Figure 8 Take the 810 processor as an example.

[0068] The device for performing the point cloud map repair method may also include an input device 830 and an output device 840.

[0069] The processor 810, memory 820, input device 830, and output device 840 can be connected via a bus or other means. Figure 7 Taking the example of a connection between China and Israel via a bus.

[0070] The memory 820, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the point cloud map repair method in the embodiments of this application. The processor 810 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 820, thereby implementing the point cloud map repair method in the above-described method embodiments.

[0071] The memory 820 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the point cloud map repair device. Furthermore, the memory 820 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 820 may optionally include memory remotely located relative to the processor 810, and these remote memories can be connected to the point cloud map repair device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0072] The input device 830 can receive input digital or character information, and generate signals related to user settings and function control of the point cloud map repair device. The output device 840 may include a display device such as a display screen.

[0073] The one or more modules are stored in the memory 820, and when executed by the one or more processors 810, they perform the point cloud map repair method in any of the above method embodiments.

[0074] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0075] The electronic devices in this application embodiments exist in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0076] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include: PDAs, MIDs, and UMPCs, etc.

[0077] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0078] (4) Other airborne electronic devices with data interaction capabilities, such as vehicle-mounted systems installed on vehicles.

[0079] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0080] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A point cloud map repair method, comprising: Based on the current state of each frame, determine the key frames for repair in the map; The repair keyframe and / or frames related to the repair keyframe are designated as keyframes to be repaired. The keyframes to be repaired are optimized to obtain a repaired point cloud map; wherein, in response to the user marking the keyframes to be repaired as loops, loop detection optimization is initiated within a preset range of the keyframes to be repaired; the similarity of the keyframes to be repaired is added to the entire map as a manual constraint, and the entire map is optimized based on at least one manual constraint; The point cloud map optimization employs factor graph optimization. The step of initiating loop closure detection optimization within a preset range of the keyframes to be repaired includes: based on the results of loop closure detection, applying an artificial loop closure detection factor to the point cloud map as an artificial constraint to control the point cloud shape within the preset range, using the similarity of the keyframes to be repaired as an artificial constraint; removing the influence of other factors within the preset range, and performing factor graph optimization solely based on the artificial loop closure detection factor.

2. The method according to claim 1, characterized in that, The process of determining the keyframes for map repair based on the current state of each frame includes: If the current state of the current frame is misaligned, the current frame is determined as the key frame for repair.

3. The method according to claim 2, characterized in that, The keyframes to be repaired are optimized to obtain the repaired map, which includes: In response to the user's pose adjustment of the keyframe to be repaired, the pose of the keyframe to be repaired is updated.

4. The method according to claim 1, characterized in that, The process of determining the keyframes for map repair based on the current state of each frame includes: If the current state of the current frame is overlapping, the current frame is determined as the key frame for repair. The step of using the repair keyframe and / or frames related to the repair keyframe as the keyframe to be repaired includes: Keyframes with a similarity higher than a preset threshold to the repair keyframe are identified as frames related to the repair keyframe. The repair keyframe and the frames related to the repair keyframe are designated as keyframes to be repaired.

5. A point cloud map restoration device, comprising: The status determination module is configured to determine the repair keyframes in the map based on the current status of each frame. The extension module is configured to use the repair keyframe and / or frames related to the repair keyframe as keyframes to be repaired. The repair module is configured to perform optimization operations on the keyframes to be repaired to obtain a repaired point cloud map. Specifically, in response to a user marking a loop closure on the keyframes to be repaired, loop closure detection optimization is initiated within a preset range of the keyframes to be repaired. The similarity of the keyframes to be repaired is added as a manual constraint to the entire map, and the entire map is optimized based on at least one manual constraint. The point cloud map optimization employs factor graph optimization. Initiating loop closure detection optimization within the preset range of the keyframes to be repaired includes: based on the loop closure detection results, applying a manual loop closure detection factor to the point cloud map as a manual constraint to control the point cloud shape within the preset range; removing the influence of other factors within the preset range, and performing factor graph optimization only based on the manual loop closure detection factor.

6. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-4.

7. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-4.

8. A computer program product comprising a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions that, when executed by a computer, cause the computer to perform the steps of the method according to any one of claims 1 to 4.

Images