Multi-camera real-time mapping method and device, electronic equipment and storage medium
Through the multi-camera real-time map construction method, combined with pre-integration processing and feature extraction, the problem of low real-time map construction accuracy in autonomous driving systems is solved, reducing the system construction cost and improving measurement accuracy and map construction accuracy.
Patent Information
- Application Number
- CN202411344728.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-25
- Publication Date
- 2025-01-14
AI Technical Summary
In the autonomous driving system, the real-time map construction accuracy is low, the sensor cost is high, and the light is sensitive, resulting in large data processing volume and unstable accuracy.
Using multi-camera real-time mapping method, pre-integration processing, feature extraction, initialization and fusion by obtaining sensor data and multi-camera data, combining pose matching and inter-frame matching, reduce sensor usage, coordinate multiple cameras, and avoid external environmental impact. , improve measurement accuracy and graph construction accuracy.
On the premise of reducing the use of sensors, reduce the construction cost of autonomous driving systems, improve real-time mapping accuracy and speed, avoid data divergence and blurring problems, and achieve higher measurement accuracy and mapping accuracy.
Smart Images
Figure CN119313758A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of real-time map construction, and in particular to a multi-camera real-time mapping method, device, electronic device and storage medium. Background Art
[0002] At present, in the autonomous driving system, the mapping module is one of the most basic and important modules, which can provide the system with prior map information and enable the system to have "beyond visual range" perception capabilities. The implementation scheme of the mapping module can be divided into vision-based mapping scheme, lidar-based mapping scheme and multi-sensor fusion-based mapping scheme. Among them, the technical scheme of lidar and multi-sensor fusion can achieve higher-precision mapping results because it can directly observe depth information through sensors.
[0003] However, due to the large amount of data that needs to be processed in the real-time mapping process of autonomous driving, a large amount of computing and storage resources are usually consumed, and due to the high cost of sensors, their application in autonomous driving systems is limited. Therefore, cameras are widely used in autonomous driving systems due to their low cost. However, in the existing technology, since camera sensors cannot directly measure depth information and are sensitive to external factors such as lighting, the measurement accuracy cannot be guaranteed, resulting in low accuracy in real-time mapping of autonomous driving. Summary of the invention
[0004] The purpose of this application is to provide a multi-camera real-time mapping method, device, electronic device and storage medium, which can fully cover the real-time mapping in the process of autonomous driving, reduce the construction cost of the autonomous driving system while reducing the use of sensors, and coordinate multiple cameras to avoid the influence of the external environment on camera acquisition, avoid divergence and ambiguity in the collected data, and improve the measurement accuracy and mapping accuracy.
[0005] In a first aspect, an embodiment of the present application provides a multi-camera real-time mapping method, the method comprising:
[0006] Acquire sensor data and multi-camera data;
[0007] Performing pre-integration processing on the sensor data to obtain pre-integration data;
[0008] Performing feature extraction on the multi-camera data to obtain feature point data;
[0009] Initializing the pre-integration data and the feature point data to obtain initialization data;
[0010] fusing the pre-integrated data and the feature point data to obtain first fused data;
[0011] A mapping result is obtained according to the initialization data and the first fused data.
[0012] In the above implementation process, through multi-dimensional processing of sensor data and multi-camera data, real-time mapping in the process of autonomous driving can be fully covered. On the premise of reducing the use of sensors, the construction cost of the autonomous driving system is reduced. At the same time, multiple cameras can be coordinated to avoid the influence of the external environment on camera acquisition, avoid divergence and ambiguity in the collected data, and improve the measurement accuracy and mapping accuracy.
[0013] Furthermore, the step of initializing the pre-integrated data and the feature point data to obtain the initialized data comprises:
[0014] Performing binocular matching on first feature point data and second feature point data in the feature point data to obtain depth information;
[0015] Matching adjacent frame data of the first feature point data and the second feature point data according to the depth information to obtain a pose result;
[0016] Performing joint optimization processing on the pre-integrated data and the position and posture result to obtain first optimized data;
[0017] Obtaining second optimized data according to third feature point data and fourth feature point data in the feature point data;
[0018] The initialization data is obtained according to the first optimization data and the second optimization data.
[0019] In the above implementation process, the pre-integration data and the feature point data are initialized in two stages, and are jointly optimized from the perspective of posture matching and the perspective of inter-frame matching, respectively, to better combine and apply the integral data and feature point data, improve the posture optimization results, and improve the accuracy of the initialized data.
[0020] Further, the step of obtaining the second optimized data according to the third feature point data and the fourth feature point data in the feature point data includes:
[0021] constructing a local map according to the first optimized data;
[0022] Performing inter-frame matching on the third feature point data and the fourth feature point data to obtain a matching result;
[0023] Performing map matching on the third feature point data and the fourth feature point data according to the local map to obtain a map result;
[0024] The matching result and the map result are jointly optimized to obtain the second optimized data.
[0025] In the above implementation process, inter-frame matching and map matching are performed based on the third feature point data and the fourth feature point data, without relying on sensor equipment. At the same time, the posture information of the road surface can be fully obtained, providing basic assistance for subsequent map construction.
[0026] Furthermore, the step of obtaining a mapping result according to the initialization data and the first fusion data includes:
[0027] Updating the initialization data and the first fused data into the local map to obtain an updated local map;
[0028] Backend optimization is performed according to the pre-integrated data and the updated local map to obtain the mapping result.
[0029] In the above implementation process, the initialization data and the first fused data are updated to the local map to achieve real-time update optimization of the local map, and then the back-end optimization is performed, which can optimize the map in multiple aspects and dimensions and improve the accuracy and speed of real-time mapping.
[0030] Furthermore, the step of performing backend optimization according to the pre-integrated data and the updated local map to obtain the mapping result includes:
[0031] constructing a pre-integration constraint according to the pre-integration data;
[0032] Constructing map constraints based on the updated local map;
[0033] Construct common view constraints;
[0034] The pre-integration constraint, the map constraint and the common view constraint are optimized by back-end to obtain the mapping result.
[0035] In the above implementation process, the back-end optimization is carried out together according to the pre-integration constraints, map constraints and common view constraints to achieve refined correction of various data, so that the impact of data errors on mapping is smaller, and the accuracy of mapping results is improved.
[0036] Furthermore, the step of constructing the common-view constraint includes:
[0037] Fusing the first feature point data, the second feature point data, the third feature point data, and the fourth feature point data to obtain second fused data;
[0038] The common view constraint is constructed according to the spatial relative pose of the second fused data.
[0039] In the above implementation process, the first feature point data, the second feature point data, the third feature point data, and the fourth feature point data are fused, and a common view constraint is constructed according to their relative spatial postures. The viewpoints of multiple cameras at different angles can be fused, and the visual deviations caused by multiple cameras can be corrected, thereby facilitating real-time mapping.
[0040] Furthermore, the step of performing backend optimization according to the pre-integrated data and the updated local map to obtain the mapping result further includes:
[0041] Perform loop closure detection on historical data based on the pre-built bag-of-words model to obtain loop closure constraints;
[0042] The pose drift of the mapping result is corrected according to the loop closure constraint.
[0043] In the above implementation process, the loop closure constraint is obtained by loop closure detection of historical data according to the bag-of-words model, and the mapping result is corrected according to the loop closure constraint, so that the mapping result can avoid the errors existing in the historical data and improve the accuracy.
[0044] In a second aspect, an embodiment of the present application further provides a multi-camera real-time mapping device, the device comprising:
[0045] Acquisition module, used to acquire sensor data and multi-camera data;
[0046] A pre-integration processing module, used for performing pre-integration processing on the sensor data to obtain pre-integration data;
[0047] A feature extraction module, used to extract features from the multi-camera data to obtain feature point data;
[0048] An initialization module, used to initialize the pre-integration data and the feature point data to obtain initialization data;
[0049] A fusion module, used for fusing the pre-integrated data and the feature point data to obtain first fused data;
[0050] A mapping module is used to obtain a mapping result according to the initialization data and the first fused data.
[0051] In the above implementation process, through multi-dimensional processing of sensor data and multi-camera data, real-time mapping in the process of autonomous driving can be fully covered. On the premise of reducing the use of sensors, the construction cost of the autonomous driving system is reduced. At the same time, multiple cameras can be coordinated to avoid the influence of the external environment on camera acquisition, avoid divergence and ambiguity in the collected data, and improve the measurement accuracy and mapping accuracy.
[0052] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in any one of the first aspects when executing the computer program.
[0053] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which instructions are stored. When the instructions are executed on a computer, the computer executes the method as described in any one of the first aspects.
[0054] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when running on a computer, enables the computer to execute the method as described in any one of the first aspects.
[0055] Other features and advantages of the present disclosure will be set forth in the following description, or some features and advantages may be inferred or unambiguously determined from the description, or may be learned by implementing the above-mentioned technology of the present disclosure.
[0056] And it can be implemented according to the contents of the specification. The following is a detailed description of the preferred embodiments of the present application with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the range values. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.
[0058] Figure 1 A schematic diagram of the process of a multi-camera real-time mapping method provided in an embodiment of the present application;
[0059] Figure 2 A schematic diagram of the structure of a multi-camera real-time mapping device provided in an embodiment of the present application;
[0060] Figure 3 A schematic diagram of the structural composition of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0061] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.
[0062] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0063] The specific implementation methods of the present application are further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present application, but are not used to limit the scope of the present application.
[0064] Embodiment 1
[0065] Figure 1 is a flow chart of a multi-camera real-time mapping method provided in an embodiment of the present application, such as Figure 1 As shown, the method includes:
[0066] S1, obtain sensor data and multi-camera data;
[0067] S2, performing pre-integration processing on the sensor data to obtain pre-integration data;
[0068] S3, extracting features from multi-camera data to obtain feature point data;
[0069] S4, initializing the pre-integration data and the feature point data to obtain initialization data;
[0070] S5, fusing the pre-integrated data and the feature point data to obtain first fused data;
[0071] S6, obtaining a mapping result according to the initialization data and the first fused data.
[0072] In the above implementation process, through multi-dimensional processing of sensor data and multi-camera data, real-time mapping in the process of autonomous driving can be fully covered. On the premise of reducing the use of sensors, the construction cost of the autonomous driving system is reduced. At the same time, multiple cameras can be coordinated to avoid the influence of the external environment on camera acquisition, avoid divergence and ambiguity in the collected data, and improve the measurement accuracy and mapping accuracy.
[0073] In S1, the sensor and camera configuration scheme of the autonomous driving system targeted by the embodiment of the present application is as follows: the system consists of four cameras (two front views + two side views), an inertial measurement unit (IMU) and a set of GPS equipment. The function of the camera is to collect image signals, and the IMU can collect inertial signals of the vehicle, including three-axis acceleration and three-axis angular velocity. The GPS equipment can provide accurate measurement of absolute position information.
[0074] Before use, the sensor needs to be calibrated in advance, including the external parameters between each sensor (relative position and relative posture) and the internal parameters of the sensor (camera internal parameters, IMU internal parameters). In addition, the sensors in the autonomous driving system of this application are all timed by the same clock source, and the synchronization trigger mechanism is satisfied between each camera, that is, each camera is guaranteed to collect image signals at the same time. Optionally, the GPS device of this application is an optional sensor, and the number of cameras can be appropriately increased or decreased, but the cameras need to have a common view area.
[0075] The multi-camera of the present application includes a front-looking left camera (CamFL, corresponding to the first camera in the present application), a front-looking right camera (CamFR, corresponding to the second camera in the present application), a left camera (CamL, corresponding to the third camera in the present application), and a right camera (CamR, corresponding to the fourth camera in the present application).
[0076] In S2, the pre-integration processing of sensor data in this application is mainly to pre-integrate the IMU collected data, process the acceleration signals and angular velocity signals collected by several frames of IMU sensors, and describe the inertial data in the interval in the form of increments, which can reduce repeated integration calculations during the optimization process.
[0077] Since the posture at the next moment is strongly correlated with the result at the previous moment during the integration process, in order to avoid repeated integral calculations during the optimization process, the embodiment of the present application uses a pre-integration method.
[0078] In S3, before feature extraction, the embodiment of the present application preprocesses the visual image signal (multi-camera data) collected by the camera, and preprocesses the image pixel value through a histogram equalization algorithm to avoid changes in external factors such as lighting that cause image quality degradation and thus affect the performance of subsequent feature extraction.
[0079] Feature extraction is achieved by extracting feature points from image texture, grayscale and other data, which can ensure the quality of mapping and achieve downsampling of image data.
[0080] Furthermore, S4 includes:
[0081] Performing binocular matching on first feature point data and second feature point data in the feature point data to obtain depth information;
[0082] Matching adjacent frame data of the first feature point data and the second feature point data according to the depth information to obtain a pose result;
[0083] Performing joint optimization processing on the pre-integrated data and the position and posture results to obtain first optimized data;
[0084] Obtaining second optimized data according to third feature point data and fourth feature point data in the feature point data;
[0085] Initialization data is obtained according to the first optimization data and the second optimization data.
[0086] In the above implementation process, the pre-integration data and feature point data are initialized in two stages, and are jointly optimized from the perspective of pose matching and inter-frame matching, respectively, to better combine and apply the integral data and feature point data to improve the pose optimization results and the accuracy of the initialized data.
[0087] The purpose of initialization in this application is to align the IMU measurement data (sensor data) with the feature point data, to initialize the local map data and to estimate the zero bias of the sensor data.
[0088] In the embodiment of the present application, the first feature point data is the feature point data after feature extraction is performed on the data collected by the first camera, the second feature point data is the feature point data after feature extraction is performed on the data collected by the second camera, the third feature point data is the feature point data after feature extraction is performed on the data collected by the third camera, and the fourth feature point data is the feature point data after feature extraction is performed on the data collected by the fourth camera.
[0089] Initialization is implemented in two stages. The first stage is to perform binocular matching and pose optimization processing on the feature point data based on image features (in order to ensure execution efficiency, the image features are implemented in advance based on the ORB algorithm in the embodiment of the present application). The binocular matching obtains the depth information of the matched feature points based on the baseline of the forward-looking binocular camera (the forward-looking left camera and the forward-looking right camera), and obtains the pose result of each frame of the image by matching the adjacent frame data (the embodiment of the present application adopts the ORB-SLAM2 algorithm).
[0090] Joint optimization is achieved by fusing the pre-integrated data with the pose results, which can be simplified to a nonlinear optimization problem as shown below:
[0091]
[0092] in, is the set of variables to be optimized, p i is the position of the image, R i For the image's posture, is the map point, NI is the number of image frames, NM is the number of map points, f p 、f R 、f M is the constraint function, b ω is the zero bias data of the gyroscope, b a is the zero bias data of acceleration.
[0093] Further, the step of obtaining the second optimized data according to the third feature point data and the fourth feature point data in the feature point data includes:
[0094] constructing a local map according to the first optimization data;
[0095] Performing inter-frame matching on the third feature point data and the fourth feature point data to obtain a matching result;
[0096] Performing map matching on the third feature point data and the fourth feature point data according to the local map to obtain a map result;
[0097] The matching result and the map result are jointly optimized to obtain second optimized data.
[0098] In the above implementation process, inter-frame matching and map matching are performed based on the third feature point data and the fourth feature point data, without relying on sensor equipment. At the same time, the posture information of the road surface can be fully obtained, providing basic assistance for subsequent map construction.
[0099] The second stage of initialization is to process the data from the left camera and the right camera. By jointly optimizing the inter-frame matching results, map matching results and pre-integrated data, the optimized results are obtained and updated to the local map.
[0100] It is worth noting that the present application also considers that the binocular can achieve high-precision posture optimization results in the first stage. Therefore, in the second stage, the posture of the sensor is directly assigned by the results of the first stage, and the second stage only optimizes the position of the map points formed by the third feature point data and the fourth feature point data.
[0101] In S5, by fusing the feature point data and the pre-integrated data, the relative position between adjacent frames is estimated and the local map data is updated. The fusion target only includes the position and posture of the image, and does not include the information of the map point. The embodiment of the present application is implemented by extracting representative data as key frames, wherein the selection of key frames is implemented based on distance changes, angle changes, and tracking point changes.
[0102] Furthermore, S6 includes:
[0103] Updating the initialization data and the first fused data into the local map to obtain an updated local map;
[0104] The backend is optimized based on the pre-integrated data and the updated local map to obtain the mapping result.
[0105] In the above implementation process, the initialization data and the first fused data are updated to the local map to achieve real-time update optimization of the local map, and then the back-end optimization is performed, which can optimize the map in multiple aspects and dimensions and improve the accuracy and speed of real-time mapping.
[0106] Furthermore, the step of performing backend optimization based on the pre-integrated data and the updated local map to obtain a mapping result includes:
[0107] Construct pre-integration constraints based on pre-integration data;
[0108] Constructing map constraints based on the updated local map;
[0109] Construct common view constraints;
[0110] The pre-integration constraints, map constraints, and common-view constraints are optimized on the back end to obtain the mapping results.
[0111] In the above implementation process, the back-end optimization is carried out together according to the pre-integration constraints, map constraints and common view constraints to achieve refined correction of various data, so that the impact of data errors on mapping is smaller, and the accuracy of mapping results is improved.
[0112] When the system detects that a new key frame is generated, the key frame needs to be optimized on the backend.
[0113] Backend optimization of pre-integration constraints, map constraints, and common-view constraints is achieved through the following formula:
[0114]
[0115] in, is the image feature point, For location data, is the image feature point, M is the local map, α t-1,t , β t-1,t ,δ t-1,t is the pre-integrated data, f Match is the sum of the pre-integration constraint and the map constraint, f Convis is the common view constraint, b ω is the zero bias data of the gyroscope, b a is the zero bias data of acceleration.
[0116] Furthermore, the steps of constructing the common view constraint include:
[0117] Fusing the first feature point data, the second feature point data, the third feature point data, and the fourth feature point data to obtain second fused data;
[0118] A common view constraint is constructed according to the spatial relative pose of the second fused data.
[0119] In the above implementation process, the first feature point data, the second feature point data, the third feature point data, and the fourth feature point data are fused, and a common view constraint is constructed according to their relative spatial postures. The viewpoints of multiple cameras at different angles can be fused, and the visual deviations caused by multiple cameras can be corrected, thereby facilitating real-time mapping.
[0120] The pose and map point data are optimized by fusing the observation data of the side view camera and the front view camera and constructing error constraints based on the spatial relative pose between the cameras.
[0121] In order to improve the execution efficiency of the algorithm, in this application, only CamFL+CamL and CamFR+CamR are modeled, that is, the co-viewing relationship on the left and right sides is searched respectively.
[0122] Furthermore, the step of performing backend optimization based on the pre-integrated data and the updated local map to obtain a mapping result also includes:
[0123] Perform loop closure detection on historical data based on the pre-built bag-of-words model to obtain loop closure constraints;
[0124] The pose drift of the mapping result is corrected according to the loop closure constraint.
[0125] In the above implementation process, the loop closure constraint is obtained by loop closure detection of historical data according to the bag-of-words model, and the mapping result is corrected according to the loop closure constraint, so that the mapping result can avoid the errors existing in the historical data and improve the accuracy.
[0126] The loop detection module retrieves the historical data before the observation of the current image through the bag-of-words model, and corrects the drift of the mapping posture by constructing loop constraints.
[0127] Embodiment 2
[0128] In order to execute the method corresponding to the above embodiment 1 to achieve the corresponding functions and technical effects, a multi-camera real-time mapping device is provided below, such as Figure 2 As shown, the device comprises:
[0129] Acquisition module 1, used to acquire sensor data and multi-camera data;
[0130] The pre-integration processing module 2 is used to perform pre-integration processing on the sensor data to obtain pre-integration data;
[0131] Feature extraction module 3, used to extract features from multi-camera data to obtain feature point data;
[0132] Initialization module 4, used to initialize the pre-integration data and feature point data to obtain initialization data;
[0133] A fusion module 5 is used to fuse the pre-integrated data and the feature point data to obtain first fused data;
[0134] The mapping module 6 is used to obtain a mapping result according to the initialization data and the first fusion data.
[0135] In the above implementation process, through multi-dimensional processing of sensor data and multi-camera data, real-time mapping in the process of autonomous driving can be fully covered. On the premise of reducing the use of sensors, the construction cost of the autonomous driving system is reduced. At the same time, multiple cameras can be coordinated to avoid the influence of the external environment on camera acquisition, avoid divergence and ambiguity in the collected data, and improve the measurement accuracy and mapping accuracy.
[0136] Furthermore, the initialization module 4 is also used for:
[0137] Performing binocular matching on first feature point data and second feature point data in the feature point data to obtain depth information;
[0138] Matching adjacent frame data of the first feature point data and the second feature point data according to the depth information to obtain a pose result;
[0139] Performing joint optimization processing on the pre-integrated data and the position and posture results to obtain first optimized data;
[0140] Obtaining second optimized data according to third feature point data and fourth feature point data in the feature point data;
[0141] Initialization data is obtained according to the first optimization data and the second optimization data.
[0142] In the above implementation process, the pre-integration data and feature point data are initialized in two stages, and are jointly optimized from the perspective of pose matching and inter-frame matching, respectively, to better combine and apply the integral data and feature point data to improve the pose optimization results and the accuracy of the initialized data.
[0143] Furthermore, the initialization module 4 is also used for:
[0144] constructing a local map according to the first optimization data;
[0145] Performing inter-frame matching on the third feature point data and the fourth feature point data to obtain a matching result;
[0146] Performing map matching on the third feature point data and the fourth feature point data according to the local map to obtain a map result;
[0147] The matching result and the map result are jointly optimized to obtain second optimized data.
[0148] In the above implementation process, inter-frame matching and map matching are performed based on the third feature point data and the fourth feature point data, without relying on sensor equipment. At the same time, the posture information of the road surface can be fully obtained, providing basic assistance for subsequent map construction.
[0149] Furthermore, the mapping module 6 is also used for:
[0150] Updating the initialization data and the first fused data into the local map to obtain an updated local map;
[0151] The backend is optimized based on the pre-integrated data and the updated local map to obtain the mapping result.
[0152] In the above implementation process, the initialization data and the first fused data are updated to the local map to achieve real-time update optimization of the local map, and then the back-end optimization is performed, which can optimize the map in multiple aspects and dimensions and improve the accuracy and speed of real-time mapping.
[0153] Furthermore, the mapping module 6 is also used for:
[0154] Construct pre-integration constraints based on pre-integration data;
[0155] Constructing map constraints based on the updated local map;
[0156] Construct common view constraints;
[0157] The pre-integration constraints, map constraints, and common-view constraints are optimized on the back end to obtain the mapping results.
[0158] In the above implementation process, the back-end optimization is carried out together according to the pre-integration constraints, map constraints and common view constraints to achieve refined correction of various data, so that the impact of data errors on mapping is smaller, and the accuracy of mapping results is improved.
[0159] Furthermore, the mapping module 6 is also used for:
[0160] Fusing the first feature point data, the second feature point data, the third feature point data, and the fourth feature point data to obtain second fused data;
[0161] A common view constraint is constructed according to the spatial relative pose of the second fused data.
[0162] In the above implementation process, the first feature point data, the second feature point data, the third feature point data, and the fourth feature point data are fused, and a common view constraint is constructed according to their relative spatial postures. The viewpoints of multiple cameras at different angles can be fused, and the visual deviations caused by multiple cameras can be corrected, thereby facilitating real-time mapping.
[0163] Furthermore, the device also includes a loop correction module, which is used to:
[0164] Perform loop closure detection on historical data based on the pre-built bag-of-words model to obtain loop closure constraints;
[0165] The pose drift of the mapping result is corrected according to the loop closure constraint.
[0166] In the above implementation process, the loop closure constraint is obtained by loop closure detection of historical data according to the bag-of-words model, and the mapping result is corrected according to the loop closure constraint, so that the mapping result can avoid the errors existing in the historical data and improve the accuracy.
[0167] The above-mentioned multi-camera real-time mapping device can implement the method of the above-mentioned embodiment 1. The options in the above-mentioned embodiment 1 are also applicable to this embodiment and will not be described in detail here.
[0168] The rest of the contents of the embodiments of the present application can refer to the contents of the above-mentioned embodiment 1, and will not be repeated in this embodiment.
[0169] Embodiment 3
[0170] An embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the multi-camera real-time mapping method of embodiment 1.
[0171] Optionally, the above-mentioned electronic device may be a server.
[0172] See also Figure 3 , Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include a processor 31, a communication interface 32, a memory 33 and at least one communication bus 34. The communication bus 34 is used to realize direct connection and communication between these components. The communication interface 32 of the device in the embodiment of the present application is used to communicate signaling or data with other node devices. The processor 31 may be an integrated circuit chip with signal processing capabilities.
[0173] The processor 31 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor 31 can also be any conventional processor, etc.
[0174] The memory 33 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The memory 33 stores computer-readable instructions. When the computer-readable instructions are executed by the processor 31, the device can execute the above-mentioned Figure 1 The method embodiment involves various steps.
[0175] Optionally, the electronic device may further include a storage controller and an input / output unit. The memory 33, the storage controller, the processor 31, the peripheral interface, and the input / output unit are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components may be electrically connected to each other via a communication bus 34. The processor 31 is used to execute executable modules stored in the memory 33, such as software function modules or computer programs included in the device.
[0176] The input and output unit is used to provide users with the task creation and to create an optional time period or preset execution time for the task to enable interaction between the user and the server. The input and output unit can be, but is not limited to, a mouse and a keyboard.
[0177] Understandably, Figure 3 The structure shown is for illustration only. The electronic device may also include Figure 3 More or fewer components as shown, or with Figure 3 Different configurations shown. Figure 3 Each component shown in the figure can be implemented by hardware, software or a combination thereof.
[0178] In addition, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the multi-camera real-time mapping method of embodiment 1.
[0179] The embodiment of the present application also provides a computer program product, which, when running on a computer, enables the computer to execute the method described in the method embodiment.
[0180] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart, can be implemented with a dedicated hardware-based device that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0181] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
[0182] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.
[0183] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application. It should be noted that similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.
[0184] The above is only a specific implementation of the present application, but the protection range of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical range disclosed in the present application, which should be included in the protection range of the present application. Therefore, the protection range of the present application should be based on the protection range of the claims.
[0185] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
Claims
1. A multi-camera real-time mapping method, characterized in that: The method comprises: Acquire sensor data and multi-camera data; Performing pre-integration processing on the sensor data to obtain pre-integration data; Performing feature extraction on the multi-camera data to obtain feature point data; Initializing the pre-integration data and the feature point data to obtain initialization data; fusing the pre-integrated data and the feature point data to obtain first fused data; A mapping result is obtained according to the initialization data and the first fused data.
2. The multi-camera real-time mapping method according to claim 1, characterized in that: The step of initializing the pre-integrated data and the feature point data to obtain the initialized data comprises: Performing binocular matching on first feature point data and second feature point data in the feature point data to obtain depth information; Matching adjacent frame data of the first feature point data and the second feature point data according to the depth information to obtain a pose result; Performing joint optimization processing on the pre-integrated data and the position and posture result to obtain first optimized data; Obtaining second optimized data according to third feature point data and fourth feature point data in the feature point data; The initialization data is obtained according to the first optimization data and the second optimization data.
3. The multi-camera real-time mapping method according to claim 2, characterized in that: The step of obtaining the second optimized data according to the third feature point data and the fourth feature point data in the feature point data comprises: constructing a local map according to the first optimized data; Performing inter-frame matching on the third feature point data and the fourth feature point data to obtain a matching result; Performing map matching on the third feature point data and the fourth feature point data according to the local map to obtain a map result; The matching result and the map result are jointly optimized to obtain the second optimized data.
4. The multi-camera real-time mapping method according to claim 1, characterized in that: The step of obtaining a mapping result according to the initialization data and the first fusion data comprises: Updating the initialization data and the first fused data into the local map to obtain an updated local map; Backend optimization is performed according to the pre-integrated data and the updated local map to obtain the mapping result.
5. The multi-camera real-time mapping method according to claim 1, characterized in that: The step of performing backend optimization according to the pre-integrated data and the updated local map to obtain the mapping result comprises: constructing a pre-integration constraint according to the pre-integration data; Constructing map constraints based on the updated local map; Construct common view constraints; The pre-integration constraint, the map constraint and the common view constraint are optimized by back-end to obtain the mapping result.
6. The multi-camera real-time mapping method according to claim 5, characterized in that: The step of constructing the common-view constraint includes: Fusing the first feature point data, the second feature point data, the third feature point data, and the fourth feature point data to obtain second fused data; The common view constraint is constructed according to the spatial relative pose of the second fused data.
7. The multi-camera real-time mapping method according to claim 1, characterized in that: The step of performing backend optimization according to the pre-integrated data and the updated local map to obtain the mapping result further includes: Perform loop closure detection on historical data based on the pre-built bag-of-words model to obtain loop closure constraints; The pose drift of the mapping result is corrected according to the loop closure constraint.
8. A multi-camera real-time mapping device, characterized in that: The device comprises: Acquisition module, used to acquire sensor data and multi-camera data; A pre-integration processing module, used for performing pre-integration processing on the sensor data to obtain pre-integration data; A feature extraction module, used to extract features from the multi-camera data to obtain feature point data; An initialization module, used to initialize the pre-integration data and the feature point data to obtain initialization data; A fusion module, used for fusing the pre-integrated data and the feature point data to obtain first fused data; A mapping module is used to obtain a mapping result according to the initialization data and the first fused data.
9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the multi-camera real-time mapping method according to any one of claims 1 to 7.
10. A storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the multi-camera real-time mapping method as described in any one of claims 1 to 7.