Image frame pose optimization method and related product

Abstract

The invention discloses a pose optimization method of an image frame and a related product, and relates to the technical field of automatic driving, and the method comprises the steps: obtaining an image frame of a current time point; the image frames are image frames collected by a vehicle sensor at different time points in the automatic driving process; optimizing the image frame at the current time point based on a pose optimizer to obtain an initial optimized image frame; judging whether the image frame at the current time point is a key frame or not; when the image frame at the current time point is not a key frame, outputting an optimized pose of the initial optimized image frame; and when the image frame at the current time point is a key frame, optimizing the initial optimized image frame based on the pose optimizer to obtain a key optimized image frame, and outputting the optimized pose of the key optimized image frame. According to the method, secondary optimization is carried out only when the key frame exists, so that the graph optimization time is shortened, the calculation amount is reduced, and the pose optimization efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a method for optimizing the posture of an image frame and related products. Background Art

[0002] With the continuous development of autonomous driving technology, map positioning modules play an increasingly important role in autonomous driving. In recent years, with the continuous improvement of chip computing power, the back-end technology of map positioning modules has gradually shifted from traditional filtering solutions to more advantageous graph optimization solutions.

[0003] Graph optimization constructs historical postures and related influencing factors into a graph, and optimizes the posture through node and edge constraints. Although graph optimization has the advantage of being able to optimize historical postures, it also has the disadvantages of complex calculations and high computing power consumption. When constructing a graph, the current graph optimization scheme has two options: one is to run incrementally, and add new nodes and constraints over time. This will lead to an increasing number of nodes and constraints, and the optimization time will increase. It is only suitable for offline mapping tasks that do not require high real-time performance. Another option is to consider real-time performance to build a sliding window. Although it reduces the amount of calculation to a certain extent, the sliding window length setting requirements are high, and marginalized calculations also affect the amount of calculation. For high-frequency output requirements, such as 100Hz, this solution still has a lot of pressure on chips with lower computing power because it is necessary to maintain the sliding window and perform a lot of calculations.

[0004] Therefore, how to reduce the time and computational complexity of graph optimization so as to quickly complete pose iterative optimization has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] A series of simplified concepts are introduced in the Summary of the Invention section, which will be further described in detail in the Detailed Description of the Invention section. The Summary of the Invention section of this application does not mean to attempt to define the key features and essential technical features of the claimed technical solution, nor does it mean to attempt to determine the scope of protection of the claimed technical solution.

[0006] In a first aspect, the present application proposes a method for optimizing the posture of an image frame, comprising:

[0007] Acquire an image frame at a current time point; the image frame is an image frame collected by a vehicle sensor at different time points during the autonomous driving process;

[0008] Optimizing the image frame at the current time point based on a posture optimizer to obtain an initial optimized image frame;

[0009] Determine whether the image frame at the current time point is a key frame;

[0010] When the image frame at the current time point is not the key frame, outputting the optimized posture of the initial optimized image frame;

[0011] When the image frame at the current time point is the key frame, the initial optimized image frame is optimized based on the pose optimizer to obtain a key optimized image frame, and the optimized pose of the key optimized image frame is output.

[0012] In a feasible implementation manner, the image frame at the current time point is optimized based on the posture optimizer to obtain an initial optimized image frame, including:

[0013] Determine whether the image frame at the current time point is the first frame;

[0014] If the image frame at the current time point is the first frame, setting the predicted position and posture of the image frame at the current time point as the first node to be optimized;

[0015] Constructing a first error edge based on the real-time dynamic positioning pose of the image frame at the current time point and the predicted pose of the image frame at the current time point;

[0016] Constructing a second error edge based on the single-frame matching pose of the image frame at the current time point and the predicted pose of the image frame at the current time point;

[0017] The value of the first node to be optimized is adjusted until the sum of the first error edge and the second error edge reaches a first preset threshold, and the optimization is completed to obtain the initial optimized image frame.

[0018] In a feasible implementation manner, the optimizing the image frame at the current time point based on the posture optimizer to obtain the initial optimized image frame further includes:

[0019] If the image frame at the current time point is not the first frame, setting the optimized pose of the image frame at the previous time point as a fixed node;

[0020] Setting the predicted pose of the image frame at the current time point as the second node to be optimized;

[0021] Constructing a third error edge between the fixed node and the second node to be optimized based on the relative posture between the image frame at the current time point and the image frame at the preset time point;

[0022] constructing a fourth error edge based on the real-time dynamic positioning pose of the image frame at the current time point and the predicted pose of the image frame at the current time point;

[0023] constructing a fifth error edge based on the single-frame matching pose of the image frame at the current time point and the predicted pose of the image frame at the current time point;

[0024] The value of the second node to be optimized is adjusted until the sum of the third error edge, the fourth error edge and the fifth error edge reaches a second preset threshold, and the optimization is completed to obtain the initial optimized image frame.

[0025] In a feasible implementation manner, the determining whether the image frame at the current time point is a key frame includes:

[0026] Respectively obtaining the time difference and position distance between the image frame at the current time point and the previous key frame;

[0027] If both the time difference and the position distance satisfy corresponding preset values, the image frame at the current time point is determined to be the key frame.

[0028] In a feasible implementation manner, when the image frame at the current time point is the key frame, optimizing the initial optimized image frame based on the pose optimizer to obtain the key optimized image frame includes:

[0029] If the image frame at the current time point is the key frame, adding the initial optimized image frame to the optimized sliding window;

[0030] constructing an associated error edge in the optimized sliding window;

[0031] The initial optimized image frame is optimized based on the associated error edge until the constraint of the associated error edge reaches a third preset threshold, thereby obtaining the key optimized image frame.

[0032] In a feasible implementation manner, constructing the associated error edge in the optimized sliding window includes:

[0033] Based on the single-frame matching pose of the image frame at the current time point and the real-time dynamic positioning pose of the image frame at the current time point, constructing an associated error edge between the predicted position and the actual position of the image frame at the current time point;

[0034] Based on the relative postures between different key frames in the optimized sliding window, associated error edges between the key frames based on the relative postures are constructed.

[0035] In a feasible implementation manner, before optimizing the image frame at the current time point based on the posture optimizer, the method further includes:

[0036] The pose optimizer is initialized, and input parameters and output parameters are set; the input parameters include relative pose, predicted pose, single-frame matching pose and real-time dynamic positioning pose; the output parameters include optimized pose.

[0037] In a second aspect, the present application proposes a posture optimization system for an image frame, which is applied to the posture optimization method for an image frame described in any one of the above embodiments, comprising:

[0038] An image acquisition module, used to acquire an image frame at a current time point; the image frame is an image frame acquired by the vehicle sensor at different time points during the autonomous driving process;

[0039] An initial optimization module, used to optimize the image frame at the current time point based on a posture optimizer to obtain an initial optimized image frame;

[0040] A key frame determination module, used to determine whether the image frame at the current time point is a key frame;

[0041] A first optimized posture output module, configured to output the optimized posture of the initial optimized image frame when the image frame at the current time point is not the key frame;

[0042] The second optimized posture output module is used to optimize the initial optimized image frame based on the posture optimizer when the image frame at the current time point is the key frame, obtain the key optimized image frame, and output the optimized posture of the key optimized image frame.

[0043] In a third aspect, an electronic device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is used to implement the steps of the method for optimizing the pose of an image frame as described in any one of the first aspects above when executing the computer program stored in the memory.

[0044] In a fourth aspect, the present application further proposes a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for optimizing the pose of an image frame of any one of the first aspects is implemented.

[0045] In summary, the pose optimization method of the image frame proposed in this application simplifies the sliding window design problem under high-frequency output. Through lightweight design, the sliding window is only optimized twice when there are key frames, which reduces the time and calculation amount of graph optimization, greatly improves the efficiency of pose optimization, and further enables the back-end of the graph optimization to be used on mid- and low-end example chips, greatly expanding the scope of use of map positioning modules in autonomous driving. This improves the accuracy of vehicle positioning and map construction accuracy, improves the safety and reliability of autonomous driving, and ultimately achieves efficient autonomous driving navigation and promotes the development and application of autonomous driving technology.

[0046] The image frame pose optimization method proposed in this application and other advantages, objectives and features of this application will be partially reflected through the following description, and will also be partially understood by technical personnel in this field through research and practice of this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present specification. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0048] Figure 1 A schematic diagram of the process of optimizing the posture of an image frame provided in an embodiment of the present application;

[0049] Figure 2 A flowchart for implementing posture optimization of an image frame provided in an embodiment of the present application;

[0050] Figure 3 A functional module diagram of a posture optimization system for an image frame provided in an embodiment of the present application;

[0051] Figure 4 A schematic diagram of the structure of an electronic device for optimizing the posture of an image frame provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to better understand the technical solutions provided by the embodiments of this specification, the technical solutions of the embodiments of this specification are described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of this specification and the specific features in the embodiments are detailed descriptions of the technical solutions of the embodiments of this specification, rather than limitations on the technical solutions of this specification. In the absence of conflict, the embodiments of this specification and the technical features in the embodiments can be combined with each other.

[0053] In this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the statement "comprise one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements. The term "more than two" includes two or more than two situations.

[0054] See also Figure 1 , is a flow chart of a method for optimizing the posture of an image frame provided in an embodiment of the present application, which may specifically include:

[0055] S110, obtaining an image frame at the current time point; the image frame is an image frame collected by the vehicle sensor at different time points during the automatic driving process.

[0056] S120, optimizing the image frame at the current time point based on the posture optimizer to obtain an initial optimized image frame.

[0057] S130: Determine whether the image frame at the current time point is a key frame.

[0058] S140: When the image frame at the current time point is not a key frame, output the optimized posture of the initial optimized image frame.

[0059] S150. When the image frame at the current time point is a key frame, the initial optimized image frame is optimized based on the posture optimizer to obtain a key optimized image frame, and the optimized posture of the key optimized image frame is output.

[0060] Exemplarily, a more accurate optimized pose of the image frame at the current time point is obtained by fusing multiple pose information (predicted pose, single-frame matching pose, real-time dynamic positioning pose (i.e., RTK pose) and relative pose calculated by pose). For the first frame, after initializing the pose optimizer, the predicted pose is used as the node to be optimized, and the error edge is constructed by combining the RTK pose and the single-frame matching pose. The optimization is performed to obtain the optimized pose and put it into the sliding window. Starting from the second frame, the pose optimizer is initialized, the optimized pose of the image frame at the previous time point is added as a fixed node, the predicted pose of the image frame at the current time point is added as a node to be optimized, and the error edge is constructed by combining the relative pose, RTK pose and the single-frame matching pose for optimization. The key frame will also be judged. If the conditions are met, the sliding window will be added and secondary optimization will be performed, and finally the optimized pose of the current frame will be output.

[0061] In some examples, optimizing the image frame at the current time point based on the pose optimizer to obtain an initial optimized image frame includes:

[0062] Determine whether the image frame at the current time point is the first frame;

[0063] If the image frame at the current time point is the first frame, the predicted position of the image frame at the current time point is set as the first node to be optimized;

[0064] Constructing a first error edge based on the real-time dynamic positioning pose of the image frame at the current time point and the predicted pose of the image frame at the current time point;

[0065] Constructing a second error edge based on the single-frame matching pose of the image frame at the current time point and the predicted pose of the image frame at the current time point;

[0066] The value of the first node to be optimized is adjusted until the sum of the first error edge and the second error edge reaches a first preset threshold, and the optimization is completed to obtain an initial optimized image frame.

[0067] Exemplarily, it is determined whether the image frame at the current time point is the first frame in the entire image sequence. When it is determined that the image frame at the current time point is the first frame, the predicted position T of the image frame at the current time point is added. Pred As the first node to be optimized. This predicted pose is used as a node in the graph, and the pose of this node will be adjusted in the subsequent optimization process.

[0068] Add the RTK pose T of the image frame at the current time point RTK , the first error edge is constructed using the RTK pose of the image frame at the current time point and the predicted pose. The error edge reflects the inconsistency between the two nodes, and the optimization goal is to minimize the error edge.

[0069] Add the single frame matching pose T of the image frame at the current time point SF Similarly, the second error edge is constructed using the single-frame matching pose of the image frame at the current time point and the predicted pose.

[0070] The image frame at the current time point is optimized by the optimization algorithm, and the value of the first node to be optimized is adjusted so that the sum of the first error edge and the second error edge is minimized, and the initial optimized image frame and its optimized posture T are obtained. OPT , at this time, the optimized posture T OPT That is the optimized first frame pose. Then the optimized first frame pose is placed in an optimization sliding window, and a series of continuous image frames are managed and processed through the sliding window.

[0071] In some examples, optimizing the image frame at the current time point based on the pose optimizer to obtain an initial optimized image frame also includes:

[0072] If the image frame at the current time point is not the first frame, the optimized pose of the image frame at the previous time point is set as a fixed node;

[0073] Set the predicted pose of the image frame at the current time point as the second node to be optimized;

[0074] Constructing a third error edge between the fixed node and the second node to be optimized based on the relative pose between the image frame at the current time point and the image frame at the preset time point;

[0075] Constructing a fourth error edge based on the real-time dynamic positioning pose of the image frame at the current time point and the predicted pose of the image frame at the current time point;

[0076] Constructing a fifth error edge based on the single frame matching pose of the image frame at the current time point and the predicted pose of the image frame at the current time point;

[0077] The value of the second node to be optimized is adjusted until the sum of the third error edge, the fourth error edge and the fifth error edge reaches a second preset threshold, and the optimization is completed to obtain an initial optimized image frame.

[0078] For example, if the image frame at the current time point is the second frame and subsequent non-key frames of the entire image sequence, the optimized pose of the previous frame is added and fixed as a fixed node. The pose obtained after optimization of the previous frame is added to the graph as a fixed node to provide a reference for the optimization of the current frame.

[0079] Add the predicted pose T of the image frame at the current time point Pred1 As the second node to be optimized, similar to the first frame, the predicted pose of the image frame at the current time point is added to the graph as a node to be optimized.

[0080] The relative pose of the image frame at the current time point and the image frame at the preset time point is obtained through pose inference, and the relative pose is used to construct a third error edge between the fixed node (the optimized pose of the previous image frame) and the current second node to be optimized (the predicted pose of the current image frame).

[0081] Add the RTK pose T of the image frame at the current time point RTK1 , the fourth error edge is constructed using the RTK pose of the image frame at the current time point and the current second node to be optimized.

[0082] Add the single frame matching pose T of the image frame at the current time point SF1 , Similar to the first frame, the fifth error edge is constructed using the single frame matching pose and the current second node to be optimized.

[0083] The image frame at the current time point is optimized by the optimization algorithm, and the value of the second node to be optimized is adjusted so that the sum of the third error edge, the fourth error edge and the fifth error edge is minimized, and the initial optimized image frame and its optimized posture T are obtained. OPT1 .

[0084] In some examples, determining whether an image frame at a current time point is a key frame includes:

[0085] Obtain the time difference and position distance between the image frame at the current time point and the previous key frame respectively;

[0086] If both the time difference and the position distance satisfy corresponding preset values, the image frame at the current time point is determined to be a key frame.

[0087] For example, if the time difference between the image frame at the current time point and the previous key frame is greater than 1 second, and the position distance is greater than 1 meter, the image frame at the current time point is considered to be a key frame. The selection of key frames is to reduce the amount of calculation and improve efficiency while ensuring a certain accuracy.

[0088] In some examples, when the image frame at the current time point is a key frame, the initial optimized image frame is optimized based on the pose optimizer to obtain a key optimized image frame, including:

[0089] If the image frame at the current time point is a key frame, the initial optimized image frame is added to the optimized sliding window;

[0090] Constructing the associated error edge in the optimization sliding window;

[0091] The initial optimized image frame is optimized based on the associated error edge until the constraint of the associated error edge reaches a third preset threshold, thereby obtaining a key optimized image frame.

[0092] Exemplarily, if the image frame at the current time point is a key frame, it is added to the optimization sliding window. More constraints are provided for the optimization of the key frame by constructing the associated error edge, and the initial optimized image frame is optimized based on the associated error edge until the constraint of the associated error edge reaches the third preset threshold. Through secondary optimization, the pose accuracy of the key frame is further improved. Furthermore, the sliding window is used for secondary optimization only when there is a key frame, which greatly reduces the amount of calculation of the sliding window and the iterative optimization time of the graph optimization, so that the backend can be used on mid- and low-end computing chips.

[0093] In some examples, constructing an associated error edge in an optimization sliding window includes:

[0094] Based on the single-frame matching pose of the image frame at the current time point and the real-time dynamic positioning pose of the image frame at the current time point, a correlation error edge between the predicted position and the actual position of the image frame at the current time point is constructed;

[0095] Based on the relative poses between different key frames in the optimized sliding window, the associated error edges between the key frames based on the relative poses are constructed.

[0096] For example, the actual position of the image frame at the current time point (the position determined by the RTK pose) is used as a fixed point, and the predicted position of the image frame at the current time point (the position determined by the single frame matching pose) is used as another point. A correlation error edge is constructed through these two points to represent the difference between the predicted position and the actual position.

[0097] Furthermore, assuming that there are two keyframes K1 and K2, the relative pose obtained by pose estimation can determine the relative position and posture relationship between K1 and K2. At this time, the actual pose of K1 can be used as a fixed point, and the predicted pose of K2 (the predicted pose obtained by considering the relative pose of K1 to K2 and the pose of K1) can be used as another point to construct an associated error edge, which represents the difference between the predicted pose of K2 and the actual pose.

[0098] In some examples, before optimizing the image frame at the current time point based on the pose optimizer, the method further includes:

[0099] Initialize the pose optimizer and set the input and output parameters; the input parameters include relative pose, predicted pose, single frame matching pose and real-time dynamic positioning pose; the output parameters include optimized pose.

[0100] Exemplarily, before performing image frame optimization based on the pose optimizer, the pose optimizer needs to be initialized to ensure that the final pose is as close to the true value as possible. Among them, the input parameters include the relative pose calculated by pose: the relative pose information between the image frame at the current time point and the image frame at the previous time point obtained by the pose calculation method. Predicted pose: the predicted value of the pose of the image frame at the current time point can be obtained based on a motion model, etc. The single-frame matching pose of the image frame at the current time point: the pose of the image frame at the current time point obtained by methods such as single-frame image matching. The RTK pose of the image frame at the current time point: the high-precision pose of the image frame at the current time point obtained by real-time kinematic positioning (RTK) technology. The output parameters include the optimized pose of the image frame at the current time point: the optimal pose estimate of the image frame at the current time point obtained after a series of optimization processes.

[0101] In summary, in the pose optimization method of the image frame proposed in this application, the sliding window is used for secondary optimization only when there is a key frame, which greatly reduces the amount of sliding window calculation and improves the accuracy and efficiency of pose optimization. It also further improves the accuracy of vehicle positioning and the accuracy of map construction, thereby ensuring the safety and reliability of autonomous driving, achieving efficient autonomous driving navigation, and promoting the development and application of autonomous driving technology.

[0102] like Figure 2 As shown, the technical solution of the present application is further described in detail below through specific embodiments.

[0103] Determine the input content: relative pose of the current frame, predicted pose of the current frame, single frame matching pose of the current frame, RTK pose of the current frame; output content: optimized pose of the current frame.

[0104] Determine whether it is the first frame. If it is the first frame, perform single frame optimization first:

[0105] Initialize the optimizer; add the predicted pose as the node to be optimized; add the RTK pose of the current frame to build the error edge; add the single frame matching pose of the current frame to build the error edge; perform the optimization to get the optimized pose of the current frame; put it into the optimization sliding window.

[0106] Starting from the second frame, single frame optimization is performed:

[0107] Initialize the optimizer; add the optimized pose of the previous frame and fix it as a fixed node; add the predicted pose of the current frame as the node to be optimized; add the relative pose of the pose inference to construct the error edge between the two nodes; add the RTK pose of the current frame to construct the error edge; add the single-frame matching pose of the current frame to construct the error edge; perform optimization to obtain the optimized pose of the current frame.

[0108] Determine whether it is a key frame:

[0109] If the time difference with the key frame of the previous frame is greater than 1 second and the position distance is greater than 1 meter, it is considered to be a key frame and added to the sliding window; the single frame matching pose of the current frame associated with the key frame and the RTK pose of the current frame are added as error edge constraints, and the relative pose calculated between the key frames is added as the error edge constraint between nodes to perform secondary optimization and obtain the optimized pose.

[0110] Complete the optimization and output the optimized pose of the current frame.

[0111] It should be noted that the above embodiments are only the best examples and are not limitations on the implementation methods of the present application.

[0112] Based on the same application concept, the embodiment of the present application also provides an image frame posture optimization system corresponding to the image frame posture optimization method provided in the above embodiment. Since the principle of solving the problem by the image frame posture optimization system in the embodiment of the present application is similar to the image frame posture optimization method in the above embodiment of the present application, the implementation of the image frame posture optimization system can refer to the implementation of the image frame posture optimization method, and the repeated parts will not be repeated.

[0113] like Figure 3 As shown, Figure 3 A functional module diagram of an image frame posture optimization system provided in this application, the system includes:

[0114] An image acquisition module 21 is used to acquire an image frame at a current time point; the image frame is an image frame acquired by the vehicle sensor at different time points during the autonomous driving process;

[0115] An initial optimization module 22, used to optimize the image frame at the current time point based on the posture optimizer to obtain an initial optimized image frame;

[0116] A key frame determination module 23 is used to determine whether the image frame at the current time point is a key frame;

[0117] A first optimized posture output module 24, configured to output an optimized posture of an initial optimized image frame when the image frame at the current time point is not a key frame;

[0118] The second optimized posture output module 25 is used to optimize the initial optimized image frame based on the posture optimizer when the image frame at the current time point is a key frame, obtain the key optimized image frame, and output the optimized posture of the key optimized image frame.

[0119] Exemplarily, the image frame at the current time point is collected by the image acquisition module 21 and sent to the initial optimization module. In the initial optimization module 22, for the first frame, after the optimizer is initialized, the predicted pose is used as the node to be optimized, and the error edge is constructed by combining the RTK pose and the single-frame matching pose, and the optimization is performed to obtain the optimized pose and put it into the sliding window. Starting from the second frame, the optimizer is initialized, the optimized pose of the previous frame is added as a fixed node, the predicted pose of the current frame is added as a node to be optimized, and the error edge is constructed by combining the relative pose, RTK pose and the single-frame matching pose for optimization. The key frame is judged by the key frame judgment module 23. If the conditions are met, the sliding window is added and secondary optimization is performed. Finally, the optimized pose of the current frame is output through the first optimized pose output module 24 or the second optimized pose output module 25.

[0120] In summary, the image frame pose optimization system provided by this application uses sliding windows for secondary optimization only when key frames exist, which greatly reduces the amount of sliding window calculations and improves the accuracy and efficiency of pose optimization. It also further improves the accuracy of vehicle positioning and the accuracy of map construction, thereby ensuring the safety and reliability of autonomous driving, achieving efficient autonomous driving navigation, and promoting the development and application of autonomous driving technology.

[0121] like Figure 4 As shown, based on the same application concept, an embodiment of the present application also provides an electronic device 300, including a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor, and when the processor 320 executes the computer program 311, the steps of any method of posture optimization of the above-mentioned image frame are implemented.

[0122] Since the electronic device introduced in this embodiment is a device used to implement a method for optimizing the posture of an image frame in the embodiment of the present application, based on the method introduced in the embodiment of the present application, technical personnel in this field can understand the specific implementation mode of the electronic device of this embodiment and its various variations. Therefore, how the electronic device implements the method in the embodiment of the present application is not introduced in detail here. As long as the equipment used by technical personnel in this field to implement the method in the embodiment of the present application falls within the scope of protection of this application.

[0123] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0124] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0125] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0126] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0127] An embodiment of the present application also provides a computer program product, which includes computer software instructions. When the computer software instructions are executed on a processing device, the processing device executes the process of the lightweight sliding window method in the graph optimization in the corresponding embodiment.

[0128] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)), etc.

[0129] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0130] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0131] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0132] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0133] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0134] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for optimizing the pose of an image frame, characterized in that: include: Get the image frame at the current time point; The image frames are image frames collected by vehicle sensors at different time points during the autonomous driving process; Optimizing the image frame at the current time point based on a posture optimizer to obtain an initial optimized image frame; Determine whether the image frame at the current time point is a key frame; When the image frame at the current time point is not the key frame, outputting the optimized posture of the initial optimized image frame; When the image frame at the current time point is the key frame, the initial optimized image frame is optimized based on the pose optimizer to obtain a key optimized image frame, and the optimized pose of the key optimized image frame is output.

2. The method for optimizing the pose of an image frame according to claim 1, characterized in that: The step of optimizing the image frame at the current time point based on the posture optimizer to obtain an initial optimized image frame includes: Determine whether the image frame at the current time point is the first frame; If the image frame at the current time point is the first frame, setting the predicted position and posture of the image frame at the current time point as the first node to be optimized; Constructing a first error edge based on the real-time dynamic positioning pose of the image frame at the current time point and the predicted pose of the image frame at the current time point; Constructing a second error edge based on the single-frame matching pose of the image frame at the current time point and the predicted pose of the image frame at the current time point; The value of the first node to be optimized is adjusted until the sum of the first error edge and the second error edge reaches a first preset threshold, and the optimization is completed to obtain the initial optimized image frame.

3. The method for optimizing the pose of an image frame according to claim 2, characterized in that: The step of optimizing the image frame at the current time point based on the posture optimizer to obtain an initial optimized image frame further includes: If the image frame at the current time point is not the first frame, setting the optimized pose of the image frame at the previous time point as a fixed node; Setting the predicted pose of the image frame at the current time point as the second node to be optimized; Constructing a third error edge between the fixed node and the second node to be optimized based on the relative posture between the image frame at the current time point and the image frame at the preset time point; constructing a fourth error edge based on the real-time dynamic positioning pose of the image frame at the current time point and the predicted pose of the image frame at the current time point; constructing a fifth error edge based on the single-frame matching pose of the image frame at the current time point and the predicted pose of the image frame at the current time point; The value of the second node to be optimized is adjusted until the sum of the third error edge, the fourth error edge and the fifth error edge reaches a second preset threshold, and the optimization is completed to obtain the initial optimized image frame.

4. The method for optimizing the position and posture of an image frame according to claim 1, characterized in that: The determining whether the image frame at the current time point is a key frame includes: Respectively obtaining the time difference and position distance between the image frame at the current time point and the previous key frame; If both the time difference and the position distance satisfy corresponding preset values, the image frame at the current time point is determined to be the key frame.

5. The method for optimizing the position and posture of an image frame according to claim 1, characterized in that: When the image frame at the current time point is the key frame, optimizing the initial optimized image frame based on the pose optimizer to obtain a key optimized image frame includes: If the image frame at the current time point is the key frame, adding the initial optimized image frame to the optimized sliding window; constructing an associated error edge in the optimized sliding window; The initial optimized image frame is optimized based on the associated error edge until the constraint of the associated error edge reaches a third preset threshold, thereby obtaining the key optimized image frame.

6. The method for optimizing the position and posture of an image frame according to claim 5, characterized in that: The constructing the associated error edge in the optimized sliding window includes: Based on the single-frame matching pose of the image frame at the current time point and the real-time dynamic positioning pose of the image frame at the current time point, constructing an associated error edge between the predicted position and the actual position of the image frame at the current time point; Based on the relative postures between different key frames in the optimized sliding window, associated error edges between the key frames based on the relative postures are constructed.

7. The method for optimizing the position and posture of an image frame according to claim 1, characterized in that: Before optimizing the image frame at the current time point based on the posture optimizer, the method further includes: The pose optimizer is initialized, and input parameters and output parameters are set; the input parameters include relative pose, predicted pose, single-frame matching pose and real-time dynamic positioning pose; the output parameters include optimized pose.

8. A system for optimizing the posture of an image frame, applied to the method for optimizing the posture of an image frame according to any one of claims 1 to 7, characterized in that: include: An image acquisition module, used to acquire an image frame at a current time point; the image frame is an image frame acquired by the vehicle sensor at different time points during the autonomous driving process; An initial optimization module, used to optimize the image frame at the current time point based on a posture optimizer to obtain an initial optimized image frame; A key frame determination module, used to determine whether the image frame at the current time point is a key frame; A first optimized posture output module, configured to output the optimized posture of the initial optimized image frame when the image frame at the current time point is not the key frame; The second optimized posture output module is used to optimize the initial optimized image frame based on the posture optimizer when the image frame at the current time point is the key frame, obtain the key optimized image frame, and output the optimized posture of the key optimized image frame.

9. An electronic device, comprising: A memory and a processor, characterized in that the processor is used to implement the steps of the image frame posture optimization method as described in any one of claims 1 to 7 when executing the computer program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for optimizing the pose of an image frame as claimed in any one of claims 1 to 7 are implemented.

Images