Image frame pose optimization method and related products

Abstract

The present application discloses a method for optimizing the pose of an image frame and related products, relating to the field of autonomous driving technology. The method comprises obtaining an image frame at a current time point; the image frame is an image frame collected by a vehicle sensor at different time points during the autonomous driving process; optimizing the image frame at the current time point based on a pose optimizer to obtain an initial optimized image frame; determining whether the image frame at the current time point is a key frame; if the image frame at the current time point is not a key frame, outputting an optimized pose of the initial optimized image frame; if the image frame at the current time point is a key frame, optimizing the initial optimized image frame based on the pose optimizer to obtain a key optimized image frame, and outputting an optimized pose of the key optimized image frame. The present application only performs secondary optimization when a key frame exists, thereby reducing the time and computational complexity of graph optimization and improving the efficiency of pose optimization.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a method for optimizing the posture of an image frame and related products. Background Art

[0002] With the continuous development of autonomous driving technology, map positioning modules are playing an increasingly important role in autonomous driving. In recent years, with the continuous improvement of chip computing power, the back-end technology of map positioning modules has gradually shifted from traditional filtering solutions to more advantageous graph optimization solutions.

[0003] Graph optimization constructs a graph of historical poses and related influencing factors, optimizing the pose through node and edge constraints. While graph optimization has the advantage of optimizing historical poses, it also suffers from computational complexity and high computing power consumption. Current graph optimization solutions operate incrementally when constructing a graph, continuously adding nodes and constraints over time. This results in a growing number of nodes and constraints, increasing optimization time and making it only suitable for offline mapping tasks with low real-time requirements. Another solution is to construct a sliding window with real-time considerations. While this solution reduces the amount of computation to a certain extent, it requires a high window length setting and marginalized computing also affects the computational load. For high-frequency output requirements, such as 100Hz, this solution still places a heavy burden on chips with lower computing power due to the need to maintain the sliding window and perform a large amount of computation.

[0004] Therefore, how to reduce the time and computational complexity of graph optimization so as to quickly complete pose iterative optimization has become a technical problem that needs to be solved urgently. Summary of the Invention

[0005] The Summary of the Invention introduces a series of simplified concepts that will be further described in the Detailed Description of the Invention. The Summary of the Invention of this application is not intended to limit the key features and essential technical features of the claimed technical solution, nor is it intended to determine the scope of protection of the claimed technical solution.

[0006] In a first aspect, the present application proposes a method for optimizing the pose of an image frame, comprising:

[0007] Acquire an image frame at the current time point; the image frame is an image frame collected by the vehicle sensor at different time points during the autonomous driving process;

[0008] Optimizing the image frame at the current time point based on a posture optimizer to obtain an initial optimized image frame;

[0009] Determining whether the image frame at the current time point is a key frame;

[0010] When the image frame at the current time point is not the key frame, outputting the optimized pose of the initial optimized image frame;

[0011] When the image frame at the current time point is the key frame, the initial optimized image frame is optimized based on the pose optimizer to obtain a key optimized image frame, and the optimized pose of the key optimized image frame is output.

[0012] In a feasible implementation manner, the pose optimizer is used to optimize the image frame at the current time point to obtain an initial optimized image frame, including:

[0013] Determine whether the image frame at the current time point is the first frame;

[0014] If the image frame at the current time point is the first frame, setting the predicted pose of the image frame at the current time point as the first node to be optimized;

[0015] Constructing a first error edge based on the real-time dynamic positioning pose of the image frame at the current time point and the predicted pose of the image frame at the current time point;

[0016] constructing a second error edge based on the single-frame matching pose of the image frame at the current time point and the predicted pose of the image frame at the current time point;

[0017] The value of the first node to be optimized is adjusted until the sum of the first error edge and the second error edge reaches a first preset threshold, and the optimization is completed to obtain the initial optimized image frame.

[0018] In a feasible implementation manner, the optimizing the image frame at the current time point based on the pose optimizer to obtain the initial optimized image frame further includes:

[0019] If the image frame at the current time point is not the first frame, setting the optimized pose of the image frame at the previous time point as a fixed node;

[0020] Setting the predicted pose of the image frame at the current time point as the second node to be optimized;

[0021] constructing a third error edge between the fixed node and the second node to be optimized based on the relative pose between the image frame at the current time point and the image frame at the preset time point;

[0022] constructing a fourth error margin based on the real-time dynamic positioning pose of the image frame at the current time point and the predicted pose of the image frame at the current time point;

[0023] constructing a fifth error edge based on the single-frame matching pose of the image frame at the current time point and the predicted pose of the image frame at the current time point;

[0024] The value of the second node to be optimized is adjusted until the sum of the third error edge, the fourth error edge, and the fifth error edge reaches a second preset threshold, and the optimization is completed to obtain the initial optimized image frame.

[0025] In a feasible implementation manner, determining whether the image frame at the current time point is a key frame includes:

[0026] Respectively obtaining the time difference and position distance between the image frame at the current time point and the previous key frame;

[0027] If both the time difference and the position distance satisfy corresponding preset values, the image frame at the current time point is determined to be the key frame.

[0028] In a feasible implementation manner, when the image frame at the current time point is the key frame, optimizing the initial optimized image frame based on the pose optimizer to obtain the key optimized image frame includes:

[0029] If the image frame at the current time point is the key frame, adding the initial optimized image frame to the optimized sliding window;

[0030] constructing a correlation error edge in the optimized sliding window;

[0031] The initial optimized image frame is optimized based on the associated error edge until the constraint of the associated error edge reaches a third preset threshold, thereby obtaining the key optimized image frame.

[0032] In a feasible implementation manner, constructing the associated error edge in the optimized sliding window includes:

[0033] Constructing a correlation error edge between a predicted position and an actual position of the image frame at the current time point based on the single-frame matching position of the image frame at the current time point and the real-time dynamic positioning position of the image frame at the current time point;

[0034] Based on the relative poses between different key frames in the optimized sliding window, associated error edges between the key frames based on the relative poses are constructed.

[0035] In a feasible implementation manner, before optimizing the image frame at the current time point based on the pose optimizer, the method further includes:

[0036] The pose optimizer is initialized and input parameters and output parameters are set; the input parameters include relative pose, predicted pose, single-frame matching pose and real-time dynamic positioning pose; the output parameters include optimized pose.

[0037] In a second aspect, the present application proposes a system for optimizing the pose of an image frame, which is applied to the method for optimizing the pose of an image frame described in any of the above embodiments, comprising:

[0038] An image acquisition module is used to acquire image frames at the current time point; the image frames are image frames acquired by the vehicle sensors at different time points during the autonomous driving process;

[0039] An initial optimization module, configured to optimize the image frame at the current time point based on a posture optimizer to obtain an initial optimized image frame;

[0040] A key frame determination module, configured to determine whether the image frame at the current time point is a key frame;

[0041] A first optimized pose output module, configured to output the optimized pose of the initial optimized image frame when the image frame at the current time point is not the key frame;

[0042] The second optimized posture output module is used to optimize the initial optimized image frame based on the posture optimizer when the image frame at the current time point is the key frame, obtain the key optimized image frame, and output the optimized posture of the key optimized image frame.

[0043] In a third aspect, an electronic device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to implement the steps of the image frame pose optimization method as described in any one of the first aspects above when executing the computer program stored in the memory.

[0044] In a fourth aspect, the present application further proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for optimizing the pose of an image frame according to any one of the first aspects.

[0045] In summary, the image frame pose optimization method proposed in this application simplifies the sliding window design problem for high-frequency output. Through lightweight design, the sliding window only undergoes secondary optimization when key frames are present, reducing the time and computational complexity of graph optimization and significantly improving the efficiency of pose optimization. Furthermore, the graph optimization backend can be used on mid-range and low-end computing chips, greatly expanding the scope of application of map positioning modules in autonomous driving. This improves the accuracy of vehicle positioning and map construction, enhances the safety and reliability of autonomous driving, and ultimately achieves efficient autonomous driving navigation and promotes the development and application of autonomous driving technology.

[0046] The image frame pose optimization method proposed in this application, and other advantages, objectives and features of this application will be reflected in part through the following description, and in part will also be understood by technical personnel in this field through research and practice of this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present description. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0048] Figure 1 A schematic diagram of the process of optimizing the pose of an image frame provided in an embodiment of the present application;

[0049] Figure 2 A flowchart for implementing pose optimization of an image frame provided in an embodiment of the present application;

[0050] Figure 3 A functional module diagram of an image frame pose optimization system provided in an embodiment of the present application;

[0051] Figure 4 A schematic diagram of the structure of an electronic device for optimizing the posture of an image frame provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to better understand the technical solutions provided by the embodiments of this specification, the technical solutions of the embodiments of this specification are described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of this specification and the specific features in the embodiments are detailed descriptions of the technical solutions of the embodiments of this specification, rather than limitations on the technical solutions of this specification. In the absence of conflict, the embodiments of this specification and the technical features in the embodiments can be combined with each other.

[0053] In this article, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also include elements inherent to such process, method, article or equipment. In the absence of further restrictions, the elements defined by the statement "comprising a ..." do not exclude the presence of other identical elements in the process, method, article or equipment comprising the elements. The term "two or more" includes two or more than two cases.

[0054] See also Figure 1 , which is a flow chart of a method for optimizing the pose of an image frame provided in an embodiment of the present application, which may specifically include:

[0055] S110. Acquire an image frame at a current time point; the image frame is an image frame collected by the vehicle sensor at different time points during the autonomous driving process.

[0056] S120 , optimizing the image frame at the current time point based on the posture optimizer to obtain an initial optimized image frame.

[0057] S130: Determine whether the image frame at the current time point is a key frame.

[0058] S140: When the image frame at the current time point is not a key frame, output the optimized pose of the initial optimized image frame.

[0059] S150. When the image frame at the current time point is a key frame, the initial optimized image frame is optimized based on the pose optimizer to obtain a key optimized image frame, and the optimized pose of the key optimized image frame is output.

[0060] For example, a more accurate optimized pose for the image frame at the current time point is obtained by fusing multiple pose information (predicted pose, single-frame matching pose, real-time dynamic positioning pose (RTK pose), and relative pose derived from pose estimation). For the first frame, after initializing the pose optimizer, the predicted pose is used as the node to be optimized. The RTK pose and single-frame matching pose are combined to construct an error edge. Optimization is performed to obtain the optimized pose and place it in the sliding window. Starting from the second frame, the pose optimizer is initialized, the optimized pose of the image frame at the previous time point is added as a fixed node, the predicted pose of the image frame at the current time point is added as the node to be optimized, and the relative pose, RTK pose, and single-frame matching pose are combined to construct an error edge for optimization. Key frames are also determined. If the conditions are met, they are added to the sliding window and optimized again. Finally, the optimized pose for the current frame is output.

[0061] In some examples, optimizing the image frame at the current time point based on the pose optimizer to obtain an initial optimized image frame includes:

[0062] Determine whether the image frame at the current time point is the first frame;

[0063] If the image frame at the current time point is the first frame, then the predicted pose of the image frame at the current time point is set as the first node to be optimized;

[0064] Constructing a first error edge based on the real-time dynamic positioning pose of the image frame at the current time point and the predicted pose of the image frame at the current time point;

[0065] Constructing a second error edge based on the single-frame matching pose of the image frame at the current time point and the predicted pose of the image frame at the current time point;

[0066] The value of the first node to be optimized is adjusted until the sum of the first error edge and the second error edge reaches a first preset threshold, and the optimization is completed to obtain an initial optimized image frame.

[0067] For example, it is determined whether the image frame at the current time point is the first frame in the entire image sequence. When it is determined that the image frame at the current time point is the first frame, the predicted position of the image frame at the current time point is added. As the first node to be optimized, the predicted pose is used as a node in the graph, and the pose of this node will be adjusted in the subsequent optimization process.

[0068] Add the RTK pose of the image frame at the current time point The first error edge is constructed using the RTK pose of the image frame at the current time point and the predicted pose. The error edge reflects the inconsistency between the two nodes, and the optimization goal is to minimize the error edge.

[0069] Add the single frame matching pose of the image frame at the current time point Similarly, the second error edge is constructed using the single-frame matching pose of the image frame at the current time point and the predicted pose.

[0070] The image frame at the current time point is optimized by the optimization algorithm, and the value of the first node to be optimized is adjusted so that the sum of the first error edge and the second error edge is minimized, and the initial optimized image frame and its optimized posture are obtained. , now optimize the pose This is the optimized pose of the first frame. The optimized pose of the first frame is then placed in an optimization sliding window, which manages and processes a series of continuous image frames.

[0071] In some examples, optimizing the image frame at the current time point based on the pose optimizer to obtain an initial optimized image frame further includes:

[0072] If the image frame at the current time point is not the first frame, the optimized pose of the image frame at the previous time point is set as a fixed node;

[0073] Set the predicted pose of the image frame at the current time point as the second node to be optimized;

[0074] Constructing a third error edge between the fixed node and the second node to be optimized based on the relative pose between the image frame at the current time point and the image frame at the preset time point;

[0075] Constructing a fourth error edge based on the real-time dynamic positioning pose of the image frame at the current time point and the predicted pose of the image frame at the current time point;

[0076] Constructing a fifth error edge based on the single-frame matching pose of the image frame at the current time point and the predicted pose of the image frame at the current time point;

[0077] The value of the second node to be optimized is adjusted until the sum of the third error edge, the fourth error edge, and the fifth error edge reaches a second preset threshold, and the optimization is completed to obtain an initial optimized image frame.

[0078] For example, if the image frame at the current time point is the second or subsequent non-key frame in the entire image sequence, the optimized pose of the previous frame is added and fixed as a fixed node. The optimized pose of the previous frame is added to the graph as a fixed node to provide a reference for optimizing the current frame.

[0079] Add the predicted pose of the image frame at the current time point As the second node to be optimized, similar to the first frame, the predicted pose of the image frame at the current time point is added to the graph as a node to be optimized.

[0080] The relative pose of the image frame at the current time point and the image frame at the preset time point is obtained through pose inference, and the relative pose is used to construct a third error edge between the fixed node (the optimized pose of the previous image frame) and the current second node to be optimized (the predicted pose of the current image frame).

[0081] Add the RTK pose of the image frame at the current time point , using the RTK pose of the image frame at the current time point and the current second node to be optimized to construct the fourth error edge.

[0082] Add the single frame matching pose of the image frame at the current time point ,Similar to the first frame, the fifth error edge is constructed using the single frame matching pose and the current second node to be optimized.

[0083] The image frame at the current time point is optimized by the optimization algorithm, and the value of the second node to be optimized is adjusted so that the sum of the third error edge, the fourth error edge and the fifth error edge is minimized, and the initial optimized image frame and its optimized pose are obtained. .

[0084] In some examples, determining whether an image frame at a current time point is a key frame includes:

[0085] Get the time difference and position distance between the image frame at the current time point and the previous key frame respectively;

[0086] If both the time difference and the position distance satisfy corresponding preset values, the image frame at the current time point is determined to be a key frame.

[0087] For example, if the time difference between the current image frame and the previous key frame is greater than 1 second and the position distance is greater than 1 meter, the image frame at the current time point is considered a key frame. The purpose of selecting key frames is to reduce the amount of calculation and improve efficiency while ensuring a certain level of accuracy.

[0088] In some examples, when the image frame at the current time point is a key frame, optimizing the initial optimized image frame based on the pose optimizer to obtain a key optimized image frame includes:

[0089] If the image frame at the current time point is a key frame, the initial optimized image frame is added to the optimized sliding window;

[0090] Constructing the associated error edge in the optimization sliding window;

[0091] The initial optimized image frame is optimized based on the associated error edge until the constraint of the associated error edge reaches a third preset threshold, thereby obtaining a key optimized image frame.

[0092] For example, if the image frame at the current time point is a keyframe, it is added to the optimization sliding window. By constructing associated error edges to provide additional constraints for keyframe optimization, the initial optimized image frame is optimized based on the associated error edges until the constraints on the associated error edges reach a third preset threshold. Through secondary optimization, the pose accuracy of the keyframe is further improved. Furthermore, secondary optimization using the sliding window is performed only when a keyframe exists. This significantly reduces the computational complexity of the sliding window and the iterative optimization time of the graph optimization, enabling the backend to be used on mid- and low-end computing chips.

[0093] In some examples, constructing associated error edges in an optimization sliding window includes:

[0094] Based on the single-frame matching pose of the image frame at the current time point and the real-time dynamic positioning pose of the image frame at the current time point, the correlation error edge between the predicted position and the actual position of the image frame at the current time point is constructed;

[0095] Based on the relative poses between different key frames in the optimized sliding window, the correlation error edges between key frames based on relative poses are constructed.

[0096] For example, the actual position of the image frame at the current time (the position determined by the RTK pose) is used as a fixed point, and the predicted position of the image frame at the current time (the position determined by the single-frame matching pose) is used as another point. A correlation error edge is constructed between these two points to represent the difference between the predicted and actual positions.

[0097] Furthermore, assuming there are two keyframes K1 and K2, the relative pose obtained by pose inference can determine the relative position and posture relationship between K1 and K2. In this case, we can use K1's actual pose as a fixed point and K2's predicted pose (the predicted pose considering the relative pose between K1 and K2 and K1's pose) as another point to construct an associated error edge, representing the difference between the predicted K2 pose and the actual pose.

[0098] In some examples, before optimizing the image frame at the current time point based on the pose optimizer, the method further includes:

[0099] Initialize the pose optimizer and set the input and output parameters; the input parameters include relative pose, predicted pose, single-frame matching pose and real-time dynamic positioning pose; the output parameters include optimized pose.

[0100] Exemplarily, before performing image frame optimization based on the pose optimizer, the pose optimizer must be initialized to ensure that the final pose is as close to the true value as possible. The input parameters include the relative pose of the pose inferred: the relative pose information between the image frame at the current time point and the image frame at the previous time point obtained by the pose inference method. The predicted pose: the predicted value of the pose of the image frame at the current time point, which can be obtained based on a motion model, etc. The single-frame matching pose of the image frame at the current time point: the pose of the image frame at the current time point obtained by methods such as single-frame image matching. The RTK pose of the image frame at the current time point: the high-precision pose of the image frame at the current time point obtained by real-time kinematic (RTK) technology. The output parameters include the optimized pose of the image frame at the current time point: the optimal pose estimate of the image frame at the current time point obtained after a series of optimization processes.

[0101] In summary, the image frame pose optimization method proposed in this application utilizes a sliding window for secondary optimization only when keyframes are present. This significantly reduces the computational effort required for the sliding window and improves the accuracy and efficiency of pose optimization. This further enhances the accuracy of vehicle positioning and map construction, thereby ensuring the safety and reliability of autonomous driving, enabling efficient autonomous driving navigation, and promoting the development and application of autonomous driving technology.

[0102] like Figure 2 As shown, the technical solution of this application is further described in detail below through specific embodiments.

[0103] Determine the input content: relative pose of the current frame, predicted pose of the current frame, single-frame matching pose of the current frame, and RTK pose of the current frame; output content: optimized pose of the current frame.

[0104] Determine whether it is the first frame. If it is the first frame, perform single-frame optimization first:

[0105] Initialize the optimizer; add the predicted pose as the node to be optimized; add the RTK pose of the current frame to build the error edge; add the single-frame matching pose of the current frame to build the error edge; perform the optimization to obtain the optimized pose of the current frame; and put it into the optimization sliding window.

[0106] Starting from the second frame, perform single frame optimization:

[0107] Initialize the optimizer; add the optimized pose of the previous frame and fix it as a fixed node; add the predicted pose of the current frame as the node to be optimized; add the relative pose of the pose inference to build the error edge between the two nodes; add the RTK pose of the current frame to build the error edge; add the single-frame matching pose of the current frame to build the error edge; perform optimization to obtain the optimized pose of the current frame.

[0108] Determine whether it is a key frame:

[0109] If the time difference with the key frame of the previous frame is greater than 1 second and the position distance is greater than 1 meter, it is considered a key frame and added to the sliding window; the single-frame matching pose of the current frame associated with the key frame and the RTK pose of the current frame are added as error edge constraints, and the relative pose calculated between the key frames is added as the error edge constraint between nodes to perform secondary optimization to obtain the optimized pose.

[0110] Complete the optimization and output the optimized pose of the current frame.

[0111] It should be noted that the above embodiments are only the best examples and are not intended to limit the implementation of the present application.

[0112] Based on the same application concept, the embodiments of the present application also provide an image frame posture optimization system corresponding to the image frame posture optimization method provided in the above embodiments. Since the principle of solving the problem by the image frame posture optimization system in the embodiments of the present application is similar to the image frame posture optimization method in the above embodiments of the present application, the implementation of the image frame posture optimization system can refer to the implementation of the image frame posture optimization method, and the repeated parts will not be repeated.

[0113] like Figure 3 As shown, Figure 3 This is a functional module diagram of an image frame pose optimization system provided in this application, the system includes:

[0114] Image acquisition module 21, used to obtain image frames at the current time point; image frames are image frames collected by vehicle sensors at different time points during the autonomous driving process;

[0115] An initial optimization module 22 is configured to optimize the image frame at the current time point based on a posture optimizer to obtain an initial optimized image frame;

[0116] A key frame determination module 23 is used to determine whether the image frame at the current time point is a key frame;

[0117] A first optimized pose output module 24 is configured to output an optimized pose of an initial optimized image frame when the image frame at the current time point is not a key frame;

[0118] The second optimized posture output module 25 is used to optimize the initial optimized image frame based on the posture optimizer when the image frame at the current time point is a key frame, obtain a key optimized image frame, and output the optimized posture of the key optimized image frame.

[0119] Exemplarily, the image frame at the current time point is collected by the image acquisition module 21 and sent to the initial optimization module. In the initial optimization module 22, for the first frame, after the optimizer is initialized, the predicted pose is used as the node to be optimized, and the error edge is constructed by combining the RTK pose and the single-frame matching pose. The optimization is performed to obtain the optimized pose and put it into the sliding window. Starting from the second frame, the optimizer is initialized, the optimized pose of the previous frame is added as a fixed node, the predicted pose of the current frame is added as the node to be optimized, and the error edge is constructed by combining the relative pose, RTK pose and the single-frame matching pose for optimization. The key frame is judged by the key frame judgment module 23. If the conditions are met, the sliding window is added and secondary optimization is performed. Finally, the optimized pose of the current frame is output by the first optimized pose output module 24 or the second optimized pose output module 25.

[0120] In summary, the image frame pose optimization system provided by this application utilizes a sliding window for secondary optimization only when key frames are present. This significantly reduces the computational complexity of the sliding window and improves the accuracy and efficiency of pose optimization. This further enhances the accuracy of vehicle positioning and map construction, thereby ensuring the safety and reliability of autonomous driving, enabling efficient autonomous driving navigation, and promoting the development and application of autonomous driving technology.

[0121] like Figure 4 As shown, based on the same application concept, an embodiment of the present application also provides an electronic device 300, including a memory 320, a processor 310, and a computer program 321 stored in the memory 320 and executable on the processor 310. When the processor 310 executes the computer program 321, the steps of any of the above-mentioned methods for optimizing the posture of the image frame are implemented.

[0122] Since the electronic device introduced in this embodiment is a device used to implement a method for optimizing the posture of an image frame in an embodiment of the present application, based on the method introduced in the embodiment of the present application, technical personnel in this field can understand the specific implementation of the electronic device of this embodiment and its various variations. Therefore, how the electronic device implements the method in the embodiment of the present application will not be introduced in detail here. As long as the equipment used by technical personnel in this field to implement the method in the embodiment of the present application falls within the scope of protection of this application.

[0123] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0124] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0125] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0126] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0127] An embodiment of the present application also provides a computer program product, which includes computer software instructions. When the computer software instructions are executed on a processing device, the processing device executes the process of the lightweight sliding window method in graph optimization in the corresponding embodiment.

[0128] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, they fully or partially produce the processes or functions according to the embodiments of the present application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be stored by a computer, or a data storage device such as a server or data center that integrates one or more available media. Available media can include magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0129] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0130] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0131] Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of these modules may be selected to achieve the purpose of this embodiment based on actual needs.

[0132] In addition, the functional modules in the various embodiments of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module. The above-mentioned integrated modules may be implemented in the form of hardware or software functional modules.

[0133] If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program code.

[0134] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for optimizing the pose of an image frame, characterized in that: include: Get the image frame at the current time point; The image frames are image frames collected by vehicle sensors at different time points during the autonomous driving process; Optimizing the image frame at the current time point based on the posture optimizer to obtain an initial optimized image frame, including: determining whether the image frame at the current time point is the first frame; if the image frame at the current time point is the first frame, setting the predicted posture of the image frame at the current time point as the first node to be optimized; constructing a first error edge based on the real-time dynamic positioning posture of the image frame at the current time point and the predicted posture of the image frame at the current time point; constructing a second error edge based on the single-frame matching posture of the image frame at the current time point and the predicted posture of the image frame at the current time point; adjusting the value of the first node to be optimized until the sum of the first error edge and the second error edge reaches a first preset threshold, and then completing the optimization to obtain the initial optimized image frame; Determining whether the image frame at the current time point is a key frame; When the image frame at the current time point is not the key frame, outputting the optimized pose of the initial optimized image frame; When the image frame at the current time point is the key frame, the initial optimized image frame is optimized based on the pose optimizer to obtain a key optimized image frame, and the optimized pose of the key optimized image frame is output; wherein, when the image frame at the current time point is the key frame, the initial optimized image frame is optimized based on the pose optimizer to obtain a key optimized image frame, including: if the image frame at the current time point is the key frame, the initial optimized image frame is added to the optimization sliding window; an associated error edge is constructed in the optimization sliding window; and the initial optimized image frame is optimized based on the associated error edge until the constraint of the associated error edge reaches a third preset threshold, thereby obtaining the key optimized image frame.

2. The method for optimizing the image frame posture according to claim 1, wherein: The step of optimizing the image frame at the current time point based on the pose optimizer to obtain an initial optimized image frame further includes: If the image frame at the current time point is not the first frame, setting the optimized pose of the image frame at the previous time point as a fixed node; Setting the predicted pose of the image frame at the current time point as the second node to be optimized; constructing a third error edge between the fixed node and the second node to be optimized based on the relative pose between the image frame at the current time point and the image frame at the preset time point; constructing a fourth error margin based on the real-time dynamic positioning pose of the image frame at the current time point and the predicted pose of the image frame at the current time point; constructing a fifth error edge based on the single-frame matching pose of the image frame at the current time point and the predicted pose of the image frame at the current time point; The value of the second node to be optimized is adjusted until the sum of the third error edge, the fourth error edge, and the fifth error edge reaches a second preset threshold, and the optimization is completed to obtain the initial optimized image frame.

3. The method for optimizing the image frame posture according to claim 1, wherein: The determining whether the image frame at the current time point is a key frame includes: Respectively obtaining the time difference and position distance between the image frame at the current time point and the previous key frame; If both the time difference and the position distance satisfy corresponding preset values, the image frame at the current time point is determined to be the key frame.

4. The method for optimizing the image frame posture according to claim 1, wherein: The constructing of the associated error edge in the optimized sliding window includes: Constructing a correlation error edge between a predicted position and an actual position of the image frame at the current time point based on the single-frame matching pose of the image frame at the current time point and the real-time dynamic positioning pose of the image frame at the current time point; Based on the relative poses between different key frames in the optimized sliding window, associated error edges between the key frames based on the relative poses are constructed.

5. The method for optimizing the image frame posture according to claim 1, wherein: Before optimizing the image frame at the current time point based on the posture optimizer, the method further includes: The pose optimizer is initialized and input parameters and output parameters are set; the input parameters include relative pose, predicted pose, single-frame matching pose and real-time dynamic positioning pose; the output parameters include optimized pose.

6. A system for optimizing the pose of an image frame, applied to the method for optimizing the pose of an image frame according to any one of claims 1 to 5, characterized in that: include: Image acquisition module, used to obtain the image frame at the current time point; The image frames are image frames collected by vehicle sensors at different time points during the autonomous driving process; An initial optimization module, configured to optimize the image frame at the current time point based on a posture optimizer to obtain an initial optimized image frame; A key frame determination module, configured to determine whether the image frame at the current time point is a key frame; A first optimized pose output module, configured to output the optimized pose of the initial optimized image frame when the image frame at the current time point is not the key frame; The second optimized posture output module is used to optimize the initial optimized image frame based on the posture optimizer when the image frame at the current time point is the key frame, obtain the key optimized image frame, and output the optimized posture of the key optimized image frame.

7. An electronic device comprising: A memory and a processor, characterized in that the processor is used to implement the steps of the image frame pose optimization method as described in any one of claims 1 to 5 when executing the computer program stored in the memory.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for optimizing the pose of an image frame according to any one of claims 1 to 5 are implemented.

Images