Visual mapping method and device, electronic equipment and computer readable storage medium

Abstract

The application provides a visual mapping method and device, electronic equipment, computer readable storage medium and computer program product. The method comprises: obtaining a plurality of image frames obtained by a camera shooting at a plurality of times, and camera poses of the camera at the plurality of times estimated by an inertial measurement unit (IMU); performing division processing on a map constructed according to the plurality of image frames and the camera poses of the camera at the plurality of times to obtain a plurality of sub-maps; determining a spatial size of a target in each sub-map, and determining a scale factor of each sub-map according to a spatial size ratio of the same target between adjacent sub-maps; calculating a reprojection error of the camera at the plurality of times according to the scale factors of the plurality of sub-maps; and performing solving processing on a reprojection error model to obtain optimized camera poses of the camera at the plurality of times and optimized spatial map points in the map. Through the application, robust and robust visual-IMU mapping can be realized under the premise of no specific constraint on the moving track.

Description

Technical Field

[0001] The present application relates to computer technology, and in particular to a visual mapping method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] A key approach in intelligent driving technology is to use various sensors to collect scene data and then map the scene using simultaneous localization and mapping techniques—generating a digital description of the scene. This mapping result can then be used for real-time positioning, providing scene positioning and environmental perception support for subsequent processes such as intelligent vehicle path planning and driving control within the scene.

[0003] Smart parking is often used in indoor parking lots. In such relatively isolated and closed environments, sub-meter positioning methods such as satellite navigation and wireless navigation are often not completely reliable under unstable network communication conditions. Therefore, various vehicle-side sensors and corresponding mapping and positioning algorithms are usually used to implement smart parking. The visual-inertial measurement unit (IMU) mapping and positioning algorithm has the advantages of low hardware cost and wide application scenarios, making it a frequently used solution.

[0004] In the vision-IMU mapping process, due to the inherent lack of accurate 3D spatial information in vision, there are large errors in the estimation of distance. Although the IMU can obtain 3D spatial information through trajectory measurement, due to inherent device defects such as zero bias and temperature drift, errors will accumulate during long-distance mapping, resulting in less than ideal scale of the final map and recovery of 3D map points, or even direct mapping failure. In the solutions provided by related technologies, loop closure detection is usually used to solve this problem. That is, the scale of the repeated frames that appear at the beginning and end and the consistency of the 3D map points are used to impose stronger constraints on the mapping process, thereby more accurately restoring the scale information of the entire map. However, loop closure detection requires that the moving trajectory must be connected from beginning to end to form a loop, which cannot be met in many parking tasks. Summary of the Invention

[0005] The present application provides a visual mapping method, device, electronic device, computer-readable storage medium, and computer program product, which can achieve robust visual-IMU mapping without specific constraints on movement trajectories.

[0006] The technical solution of this application is achieved as follows:

[0007] This application provides a visual mapping method, including:

[0008] obtain a plurality of image frames captured by a camera at a plurality of time instants, and camera poses of the camera at the plurality of time instants estimated by an inertial measurement unit (IMU);

[0009] perform partitioning on a map constructed according to the plurality of image frames and the camera poses of the camera at the plurality of time instants to obtain a plurality of sub-maps;

[0010] determine spatial sizes of targets in each sub-map, and determine a scale factor of each sub-map according to a spatial size ratio of a same target between adjacent sub-maps;

[0011] calculate re-projection errors of the camera at the plurality of time instants according to the scale factors of the plurality of sub-maps;

[0012] perform solving on a re-projection error model to obtain optimized camera poses of the camera at the plurality of time instants and optimized spatial map points in the map, wherein the re-projection error model is used to minimize the re-projection errors at the plurality of time instants.

[0013] The application provides a visual mapping device, comprising:

[0014] an obtaining module, configured to obtain a plurality of image frames captured by a camera at a plurality of time instants, and camera poses of the camera at the plurality of time instants estimated by an inertial measurement unit (IMU);

[0015] a partitioning module, configured to perform partitioning on a map constructed according to the plurality of image frames and the camera poses of the camera at the plurality of time instants to obtain a plurality of sub-maps;

[0016] a scale factor determining module, configured to determine spatial sizes of targets in each sub-map, and determine a scale factor of each sub-map according to a spatial size ratio of a same target between adjacent sub-maps;

[0017] an error calculating module, configured to calculate re-projection errors of the camera at the plurality of time instants according to the scale factors of the plurality of sub-maps;

[0018] an optimization module, configured to perform optimization on the re-projection errors at the plurality of time instants to obtain optimized spatial map points in the map.

[0019] The application provides an electronic device, comprising:

[0020] a memory, configured to store executable instructions;

[0021] a processor, configured to execute the executable instructions stored in the memory, and implement the visual mapping method provided by the application.

[0022] The application provides a computer readable storage medium, which stores executable instructions, so that when a processor executes the instructions, a visual mapping method is realized.

[0023] The application provides a computer program product, which comprises executable instructions, so that when a processor executes the instructions, a visual mapping method is realized.

[0024] The application has the following beneficial effects:

[0025] The application obtains a plurality of image frames obtained by a camera at a plurality of moments and camera poses of the camera at the plurality of moments estimated by an inertial measurement unit (IMU); divides a map constructed according to the plurality of image frames and the camera poses of the camera at the plurality of moments to obtain a plurality of sub-maps; determines a spatial size of a target in each sub-map, and determines a scale factor of each sub-map according to a spatial size ratio of the same target between adjacent sub-maps; calculates reprojection errors of the camera at the plurality of moments according to the scale factors of the plurality of sub-maps; and solves a reprojection error model to obtain optimized camera poses of the camera at the plurality of moments and optimized spatial map points in the map; wherein the reprojection error model is used to minimize the reprojection errors at the plurality of moments. The application considers that scale factors of adjacent moments usually have little difference, so that the whole map is divided into a plurality of sub-maps and the scale factor of each sub-map is calculated, so that the optimization problem of global mapping is changed into optimization problems of a plurality of sub-maps, so that the optimized camera poses and the optimized spatial map points can be calculated stably and robustly without specific constraints on a moving track. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort.

[0027] Figure 1 is an architecture schematic diagram of a visual mapping system provided by the embodiments of the application;

[0028] Figure 2 is a structural schematic diagram of a vehicle-mounted device provided by the embodiments of the application;

[0029] Figure 3A is a first flow schematic diagram of a visual mapping method provided by the embodiments of the application;

[0030] Figure 3B is a second flow schematic diagram of a visual mapping method provided by the embodiments of the application;

[0031] Figure 4 This is a third flow chart of the visual mapping method provided in an embodiment of the present application;

[0032] Figure 5 This is a schematic diagram of projection from three-dimensional space to an image provided by an embodiment of the present application;

[0033] Figure 6 This is a schematic diagram of dividing a segmented trajectory (dividing a sub-map) provided in an embodiment of the present application;

[0034] Figure 7 This is a schematic diagram of calculating the 3D frame size provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0036] In the following description, references to "some embodiments" describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. In the following description, the term "plurality" refers to at least two.

[0037] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0039] During the vision-IMU mapping process, due to the inherent lack of accurate 3D spatial information in vision, distance estimation suffers from significant errors. While IMUs can obtain 3D spatial information through trajectory measurement, inherent device defects such as bias and temperature drift can lead to cumulative errors during long-distance mapping, resulting in suboptimal final map scale and 3D map point recovery, and even leading to direct mapping failure. To address this issue, related technologies primarily offer the following two solutions:

[0040] 1) Introduce observation constraints of global absolute position by Global Navigation Satellite System (GNSS) signals. Since the observation of GNSS has global nature, it can give certain constraints on the estimation of scale and 3D map points in a large range. However, this scheme needs the support of GNSS signals, which is often not well implemented in relatively closed and isolated scenes such as indoor parking lots, and is only feasible in places close to the external environment such as parking lot entrances.

[0041] 2) Introduce constraints by means of loop closure detection. That is, the consistency of scale and 3D map points of repeated frames appearing at the beginning and end is used to impose constraints on the mapping process, so as to more accurately recover the scale information of the overall map. However, this scheme requires that the mobile trajectory must be connected at the beginning and end to form a loop, which cannot be met in many cases.

[0042] Embodiments of the present application provide a visual mapping method and device, electronic equipment, computer readable storage medium and computer program product, which can realize robust visual-IMU mapping without relying on external signals and mobile trajectories. The following describes an exemplary application of the electronic equipment provided by the embodiments of the present application. The electronic equipment provided by the embodiments of the present application can be implemented as various types of terminal equipment, such as a vehicle-mounted device, and can also be implemented as a server.

[0043] Referring to Figure 1 , Figure 1 is an architecture schematic diagram of a visual mapping system 100 provided by the embodiments of the present application, taking a terminal device as a vehicle-mounted device for example. The vehicle-mounted device 400 is connected to a server 200 through a network 300, wherein the network 300 can be a wide area network or a local area network, or a combination of the two.

[0044] In some embodiments, taking the electronic device as an example, the visual mapping method provided by the embodiments of the present application can be implemented by a terminal device. For example, the camera, the IMU, and the vehicle-mounted device 400 are deployed in the same vehicle, or the camera and the IMU can be regarded as part of the vehicle-mounted device 400. The vehicle-mounted device 400 obtains a plurality of image frames obtained by the camera at a plurality of times, and camera poses of the camera at the plurality of times estimated by the inertial measurement unit IMU; performs division processing on a map constructed according to the plurality of image frames and the camera poses of the camera at the plurality of times, to obtain a plurality of sub-maps; determines a spatial size of a target in each sub-map, and determines a scale factor of each sub-map according to a spatial size ratio of the same target between adjacent sub-maps; calculates reprojection errors of the camera at the plurality of times according to the scale factors of the plurality of sub-maps; and performs solving processing on a reprojection error model to obtain optimized camera poses of the camera at the plurality of times and optimized spatial map points in the map; wherein the reprojection error model is used to minimize the reprojection errors at the plurality of times. The vehicle-mounted device 400 can use the obtained optimized camera poses of the camera at the plurality of times and the optimized spatial map points in the map for intelligent driving, for example, to further implement path planning, driving control and other functions. The above-mentioned method is implemented by the vehicle-mounted device 400, which can be suitable for various situations, such as no network or poor network conditions, thereby improving the applicability.

[0045] In some embodiments, taking the electronic device as an example, the visual mapping method provided by the embodiments of the present application can be implemented by a terminal device. For example, the camera, the IMU, and the vehicle-mounted device 400 are deployed in the same vehicle, or the camera and the IMU can be regarded as part of the vehicle-mounted device 400. The vehicle-mounted device 400 obtains a plurality of image frames obtained by the camera at a plurality of times, and camera poses of the camera at the plurality of times estimated by the inertial measurement unit IMU; performs division processing on a map constructed according to the plurality of image frames and the camera poses of the camera at the plurality of times, to obtain a plurality of sub-maps; determines a spatial size of a target in each sub-map, and determines a scale factor of each sub-map according to a spatial size ratio of the same target between adjacent sub-maps; calculates reprojection errors of the camera at the plurality of times according to the scale factors of the plurality of sub-maps; and performs solving processing on a reprojection error model to obtain optimized camera poses of the camera at the plurality of times and optimized spatial map points in the map; wherein the reprojection error model is used to minimize the reprojection errors at the plurality of times. The vehicle-mounted device 400 can use the obtained optimized camera poses of the camera at the plurality of times and the optimized spatial map points in the map for intelligent driving, for example, to further implement path planning, driving control and other functions. The above-mentioned method is implemented by the vehicle-mounted device 400, which can be suitable for various situations, such as no network or poor network conditions, thereby improving the applicability.

[0046] In some embodiments, the vehicle-mounted device 400 or the server 200 can implement the visual mapping method provided in the embodiments of the present application by running a computer program. For example, the computer program can be a native program or a software module in an operating system; can be a native application program (APP), i.e., a program that needs to be installed in an operating system to run; can also be a small program, i.e., a program that only needs to be downloaded into a browser environment to run; and can also be a small program that can be embedded into any APP, which can be controlled by a user to run or shut down. In summary, the above computer program can be any form of application program, module or plug-in.

[0047] For example, the electronic device is a vehicle-mounted device provided in the embodiments of the present application. It can be understood that for the case where the electronic device is a server, Figure 2 The structures shown in the above embodiments can be omitted. For example, the user interface, the presentation module and the input processing module can be omitted. See Figure 2 , Figure 2 FIG. 4 is a structural schematic diagram of a vehicle-mounted device 400 provided in the embodiments of the present application, Figure 2 The vehicle-mounted device 400 shown in FIG. 4 includes at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the vehicle-mounted device 400 are coupled together through a bus system 440. It can be understood that the bus system 440 is used to realize the connection and communication between the components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the purpose of clear illustration, all kinds of buses are marked as the bus system 440 in Figure 2 FIG. 4.

[0048] The processor 410 can be an integrated circuit chip with signal processing capability, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0049] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432 that facilitate user input, such as microphones, touch screen displays, cameras, other input buttons and controls.

[0050] The memory 450 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, and the like. The memory 450 optionally includes one or more storage devices remotely located from the processor(s) 410.

[0051] The memory 450 includes volatile memory or nonvolatile memory, or both. Nonvolatile memory can be read only memory (ROM), and volatile memory can be random access memory (RAM). The memory 450 described herein is intended to include any suitable type of memory.

[0052] In some embodiments, the memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or superset thereof, which are exemplarily illustrated below.

[0053] The operating system 451 includes systems programs for handling various basic system services and for performing hardware dependent tasks, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services and processing hardware-based tasks;

[0054] The network communication module 452 is configured to communicate with other electronic devices via one or more (wired or wireless) network interfaces 420, examples of which include Bluetooth, wireless fidelity (WiFi), universal serial bus (USB), and the like;

[0055] The presentation module 453 is configured to enable presentation of information via one or more output devices 431 (e.g., a display screen, a speaker, and the like) associated with the user interface 430 (e.g., a user interface for operating a peripheral device and displaying content and information);

[0056] The input processing module 454 is configured to detect and interpret one or more user inputs or interactions from one or more input devices 432.

[0057] In some embodiments, the visual mapping device provided by the embodiments of the present application can be implemented in a software manner, Figure 2A visual mapping device 455 stored in the memory 450 is shown, which can be software in the form of programs and plug-ins, etc., including the following software modules: an acquisition module 4551, a division module 4552, a scale factor determination module 4553, an error calculation module 4554, and an optimization module 4555, which are logical and thus can be combined or further split according to the implemented functions. The functions of the individual modules will be described below.

[0058] An exemplary application and implementation of the electronic device provided by the embodiments of the present application will be described in conjunction with the visual mapping method provided by the embodiments of the present application.

[0059] Referring to Figure 3A , Figure 3A is a flowchart of the visual mapping method provided by the embodiments of the present application, which will be described in conjunction with Figure 3A the steps shown.

[0060] In step 101, a plurality of image frames obtained by the camera at a plurality of times and camera poses estimated by an inertial measurement unit (IMU) at the plurality of times are acquired.

[0061] Here, the camera and the IMU are deployed on the same physical device, and the embodiments of the present application do not limit the type of the physical device, which can be a vehicle or a robot, etc. During the movement of the physical device, a plurality of image frames obtained by the camera at a plurality of times and camera poses estimated by the IMU at the plurality of times are acquired, so as to construct a map of the scene around the physical device by a simultaneous localization and mapping (SLAM) algorithm. It is worth noting that for each time, an image frame and a camera pose can be obtained, and thus a corresponding relationship between the image frames and the camera poses can be established.

[0062] In some embodiments, after the plurality of image frames obtained by the camera at the plurality of times and the camera poses estimated by the IMU at the plurality of times are acquired, the visual mapping method further includes: performing screening processing on the plurality of image frames to obtain a plurality of key image frames; and wherein the plurality of key image frames and the camera poses at the corresponding times of the plurality of key image frames are used to construct the map.

[0063] After the plurality of image frames obtained by the camera at the plurality of times are acquired, the plurality of image frames can be screened to obtain a plurality of key image frames, and the map can be constructed according to the plurality of key image frames and the camera poses at the corresponding times of the plurality of key image frames, wherein the number of key image frames is less than the number of image frames.

[0064] The screening processing can refer to the following factors:

[0065] 1) Time: For example, an image frame is selected as a key image frame every fixed time (such as 1 second).

[0066] 2) Distance: For example, an image frame is selected as a key image frame at a fixed distance, where the distance can be measured by an IMU.

[0067] 3) Frame number: For example, an image frame is selected as a key image frame every fixed number of frames (such as 20 frames).

[0068] 4) The co-view relationship with the previous key image frame. For example, the number of feature points observed in common with a certain image frame and the previous key image frame is calculated. When the ratio between the number of common feature points and the number of feature points in the previous key image frame is less than a ratio threshold (such as 0.75), the image frame is determined to be a key image frame, thereby ensuring the continuity of the subsequently constructed map.

[0069] The above factors may be selected individually or in combination, without limitation.

[0070] This approach can reduce the information redundancy of image frames, greatly reduce the amount of data in subsequent processing, and improve computational efficiency. It can also prevent useless or erroneous information from entering the subsequent processing process, thereby improving mapping accuracy.

[0071] In step 102, a map constructed based on a plurality of image frames and camera positions at a plurality of moments is divided to obtain a plurality of sub-maps.

[0072] Here, a map is constructed based on multiple image frames and camera poses at multiple moments. The entire map is divided into multiple submaps, and constraints are introduced based on the consistency of the scale factors of each submap. It is worth noting that each submap can be described by a series of image frames and the camera pose corresponding to each image frame.

[0073] In some embodiments, the above-mentioned division and processing of the map constructed based on multiple image frames and the camera poses at multiple moments to obtain multiple sub-maps can be achieved in the following manner: the movement trajectory measured by the IMU is divided according to the distance to obtain multiple segmented trajectories; and the map is divided into sub-maps corresponding to the multiple segmented trajectories.

[0074] Here, the moving trajectory measured by the IMU is divided according to distance to obtain multiple segmented trajectories, for example, a segmented trajectory is divided every fixed distance. It is worth noting that the entire moving trajectory measured by the IMU corresponds to the entire map, and based on this, the division processing of the entire moving trajectory is equivalent to the division processing of the entire map, and finally multiple segmented trajectories correspond to multiple sub-maps.

[0075] In step 103, the spatial size of the target in each sub-map is determined, and the scale factor of each sub-map is determined according to the spatial size ratio of the same target between adjacent sub-maps.

[0076] Here, the target recognition processing is performed on each sub-map separately, and the spatial size of the target in each sub-map is determined. Then, the same target in adjacent sub-maps is determined, and the spatial size ratio of the same target between adjacent sub-maps is calculated, which reflects the scale factor difference between adjacent sub-maps, based on which the scale factor of each sub-map can be recovered.

[0077] In some embodiments, before determining the scale factor of each sub-map according to the spatial size ratio of the same target between adjacent sub-maps, the visual mapping method further comprises: performing semantic matching processing on the target in the first sub-map and the target in the second sub-map to obtain the same target in the first sub-map and the second sub-map; wherein the second sub-map is the next sub-map of the first sub-map; the above-mentioned determination of the scale factor of each sub-map according to the spatial size ratio of the same target between adjacent sub-maps can be achieved in the following way: determining the scale factor of the first sub-map according to the spatial size ratio of the same target between the first sub-map and the second sub-map.

[0078] For ease of illustration, the process of recovering the scale factor is described taking the first sub-map and the second sub-map as an example. The second sub-map is adjacent to the first sub-map, and the second sub-map is the next sub-map of the first sub-map (equivalent to the segmented trajectory corresponding to the second sub-map is the next segmented trajectory of the segmented trajectory corresponding to the first sub-map).

[0079] First, the target in the first sub-map is semantically matched with the target in the second sub-map to obtain the same target with the same semantics in the first sub-map and the second sub-map. Then, the spatial size ratio of the same target between the first sub-map and the second sub-map is calculated, and the spatial size ratio is determined as the scale factor of the first sub-map. Similarly, the scale factor of each sub-map can be determined. The above-mentioned way accurately locates the same target with the same semantics in adjacent sub-maps through semantic matching processing, and accurately recovers the scale factor by calculating the spatial size ratio of the same target between adjacent sub-maps.

[0080] In some embodiments, the spatial size of each target in each submap can be determined in the following way: the image pixels of any target in the image frames used to construct any submap and the camera poses at the time when the image frames are used to construct any submap are triangulated to obtain a plurality of spatial map points of any target in any submap; the plurality of spatial map points of any target in any submap are enclosed by a bounding box, and the size of the bounding box is determined as the spatial size of any target in any submap.

[0081] Here, taking any target (named as target T) in any submap (named as submap M) as an example, the process of determining the spatial size is described.

[0082] It is considered that the same feature point of the target T will appear in different two-dimensional coordinates (i.e. different image pixel coordinates) in different image frames as the camera moves, therefore, the image pixels of the target T in the image frames used to construct the submap M and the camera poses at the time when the image frames are used to construct the submap M are triangulated to restore a plurality of spatial map points of the target T in the submap M, wherein the spatial map points are also called 3D map points. The triangulation process uses geometric and mathematical methods to improve the accuracy of the obtained spatial map points by optimizing the solving method.

[0083] Then, the plurality of spatial map points of the target T in the submap M are enclosed by a bounding box, i.e. the bounding box that can enclose all the spatial map points of the target T and has the smallest size is determined, wherein the shape of the bounding box is not limited, for example, it can be a hexahedron. Then, the size of the bounding box is determined as the spatial size of the target T in the submap M. In this way, the spatial size of the target can be calculated more simply, and the calculation accuracy can also be guaranteed as much as possible.

[0084] In step 104, the reprojection errors of the camera at a plurality of time instants are calculated according to the scale factors of the plurality of submaps.

[0085] Here, the reprojection error refers to the error between the image pixel in the image frame and the reprojection point of the corresponding spatial map point (i.e. the image pixel obtained by projecting the spatial map point to the image), wherein the reprojection point needs to be calculated according to the scale factor at the corresponding time instant. In the case where the scale factor of each submap and the time instants corresponding to each submap are known, the reprojection errors of the camera at a plurality of time instants can be calculated.

[0086] In step 105, the reprojection error model is solved to obtain the optimized camera poses of the camera at a plurality of time instants and the optimized spatial map points in the map; wherein the reprojection error model is used to minimize the reprojection errors at a plurality of time instants.

[0087] Here, a re-projection error model is constructed, and the re-projection error model is used to minimize the re-projection errors at multiple time instants. By solving the re-projection error model, the optimized camera poses of the camera at multiple time instants and the optimized spatial map points in the map are obtained, so as to realize optimization of the initially constructed map and realize more accurate mapping.

[0088] After obtaining the optimized camera poses of the camera at multiple time instants and the optimized spatial map points in the map, further applications can be performed, for example, the optimized camera poses of the camera at multiple time instants and the optimized spatial map points in the map are used for path planning and driving control, so as to improve the reliability of intelligent driving.

[0089] As shown in Figure 3A , the embodiment of the present application obtains multiple image frames obtained by the camera at multiple time instants and camera poses of the camera at multiple time instants estimated by an inertial measurement unit (IMU); divides a map constructed according to the multiple image frames and the camera poses of the camera at multiple time instants to obtain multiple sub-maps; determines spatial sizes of targets in each sub-map and determines a scale factor of each sub-map according to a spatial size ratio of the same target between adjacent sub-maps; calculates re-projection errors of the camera at multiple time instants according to the scale factors of the multiple sub-maps; and solves a re-projection error model to obtain optimized camera poses of the camera at multiple time instants and optimized spatial map points in the map. The re-projection error model is used to minimize the re-projection errors at multiple time instants. The embodiment of the present application considers that scale factors of adjacent time instants usually do not differ much, so the entire map is divided into multiple sub-maps and the scale factor of each sub-map is calculated. In this way, the optimization problem of global mapping is changed into optimization problems of several sub-maps, so that the optimized camera poses and the optimized spatial map points can be calculated stably and robustly without specific constraints on the moving track.

[0090] In some embodiments, referring to Figure 3B , Figure 3B is a flowchart of a visual mapping method provided by the embodiment of the present application, Figure 3A The step 104 shown in FIG. 1 can be implemented by steps 201 to 203, which will be described in combination with the steps.

[0091] In step 201, a spatial map point observed by the camera at any time instant is re-projected according to an intrinsic matrix of the camera, a camera pose of the camera at the time instant and a scale factor of a sub-map corresponding to the time instant, to obtain a re-projection point of the spatial map point.

[0092] Here, taking any time instant (named as time instant k) as an example, the process of calculating the re-projection error of the camera at the time instant k is described.

[0093] First, the spatial map points observed by the camera at time k are reprojected based on the camera's intrinsic parameter matrix, the camera's pose at time k, and the scale factor of the submap corresponding to time k (the time corresponding to each submap can be determined when the submap is divided). This yields the reprojected points of the spatial map points observed by the camera at time k. The camera's intrinsic parameter matrix can be pre-calibrated.

[0094] It's worth noting that the camera pose describes the camera's position and orientation in the world coordinate system. It typically consists of a rotation matrix and a translation vector. The camera pose is used to project points in the world coordinate system onto the camera coordinate system. The camera's intrinsic parameter matrix describes the camera's internal geometry and is used to project points in the camera coordinate system onto the camera's imaging plane. However, these points are still points in the imaging plane, not image pixels. The scale factor converts points in the imaging plane into image pixels, accurately representing objects in three-dimensional space in the image.

[0095] It is worth noting that the camera pose at time k and the spatial map point observed by the camera at time k are the variables to be optimized in the embodiments of the present application.

[0096] In step 202, image pixels corresponding to spatial map points are determined in an image frame captured at any moment.

[0097] Here, for a spatial map point observed by the camera at time k, the corresponding image pixel point is determined in the image frame captured at time k.

[0098] In step 203, the error between the reprojection point of the spatial map point and the image pixel point corresponding to the spatial map point is determined as the reprojection error of the camera at any moment.

[0099] Here, for a spatial map point observed by the camera at time k, the error between the reprojection point of the spatial map point and the image pixel point corresponding to the spatial map point is calculated as the reprojection error of the camera at time k.

[0100] It is worth noting that, considering that the number of spatial map points observed by the camera at time k usually includes multiple, the error between the reprojection point of each spatial map point and the image pixel point corresponding to the spatial map point can be calculated, and then all errors are fused (such as summation or other processing) to obtain the reprojection error of the camera at time k.

[0101] like Figure 3BAs shown, the embodiment of the application reprojects the space map point observed by the camera at any time according to the intrinsic matrix of the camera, the camera pose of the camera at any time and the scale factor of the corresponding sub-map at any time, to obtain a reprojected point of the space map point; determines the image pixel point corresponding to the space map point in the image frame shot at any time; and determines the error between the reprojected point of the space map point and the image pixel point corresponding to the space map point as the reprojecting error of the camera at any time. The above reprojecting processing of the space map point in combination with the intrinsic matrix of the camera, the camera pose and the scale factor can accurately measure the reprojecting error of the camera, which is helpful for subsequent optimization of the camera pose and the space map point.

[0102] In the following, an exemplary application of the embodiment of the application in an actual application scenario will be described. For the convenience of understanding, the intelligent driving (such as intelligent parking) task in an indoor parking lot closed scene is taken as an example for description.

[0103] As an example, the embodiment of the application provides a visual mapping method as shown in the following flowchart, mainly including three processing stages of data preprocessing, front end (visual odometry) and back end, which will be described respectively. Figure 4

[0104] In the data preprocessing stage, the video sequence obtained by the camera shooting is subjected to image preprocessing to obtain a plurality of image frames corresponding to different times, and the image preprocessing such as converting the image into an RGB format for subsequent processing, or can also include noise removal and the like, which is not limited; in addition, the IMU and the camera need to be calibrated in advance. In the front end stage, the image features (i.e. identifying the feature points in the image) of the image obtained by the camera shooting are extracted, and then the key image frames are screened out, which are used to construct the map; at the same time, the camera pose of the camera at each time is estimated through the data measured by the IMU. In the back end stage, the space map points in the map are calculated by triangulation, and the optimized camera pose and the optimized space map points are obtained by solving the reprojecting error model.

[0105] In the embodiment of the application, the reprojecting error model is as shown below:

[0106]

[0107] wherein, (R k ,t k ) represents the camera pose at time k, R k represents the rotation matrix in the camera pose, t k represents the translation vector in the camera pose; K represents the intrinsic matrix of the camera; X i represents a certain space map point observed by the camera at time k; x i ​Indicates the image frame captured at time k and X i The corresponding image pixel point; π represents the projection operation, which is used to realize the projection from the imaging plane to the image. As an example, the embodiment of the present application provides the following Figure 5 The projection diagram shown, Figure 5 P1, P2 and P3 in represent spatial map points, p1 represents the reprojection point of P1 in a certain image frame, and so on. Figure 5 The formula in does not show the camera's intrinsic parameter matrix K.

[0108] By solving the reprojection error model, the optimal camera pose {(R k ,t k ),k=1,…,M} and all the optimization space map points {X i ,i=1,…,N}.

[0109] It is worth noting that the transformation performed by the projection operation π is as follows:

[0110]

[0111] Furthermore, the projection operation π can be expressed as follows:

[0112]

[0113] Where d represents the characteristic scale (essentially the inverse of the scale factor above). Based on this, the reprojection error model can be updated as follows:

[0114]

[0115] It is worth noting that the characteristic scale d of adjacent moments k Usually the difference is not big, so the optimization problem of global mapping can be transformed into the optimization problem of several sub-maps, each sub-map shares the same d k .

[0116] Based on the above reprojection error model, robust visual-IMU mapping can be achieved, for example, by the following steps.

[0117] Step 1: Map division is performed based on the driving trajectory measured by the IMU (corresponding to the movement trajectory above).

[0118] The accuracy of visual mapping in spatial map point recovery often depends on the baseline length of observations of the same spatial map point in adjacent images, i.e., the distance of different image frames in space. Different image frames that have observations of the same spatial map point have relatively consistent scale factors when the distance is recovered. Therefore, the entire driving trajectory given by the IMU is divided into multiple segmented trajectories according to the distance, which is equivalent to dividing the entire map into multiple sub-maps, as shown in FIG. 1. For each sub-map divided, there is a relatively consistent scale factor in the sub-map. Figure 6

[0119] Step 2: Identify the same target in adjacent sub-maps.

[0120] In each sub-map (segmented trajectory), some typical semantic objects (such as a column, a fire hydrant, a vehicle, etc.) are identified, and the same target with the same semantics in adjacent sub-maps is obtained through semantic matching.

[0121] Step 3: Calculate the scale factor of each sub-map.

[0122] For example, the scale factor of a sub-map is calculated as follows: Figure 1

[0123]

[0124] where Size 目标in子地图1 represents the 3D box size (corresponding to the spatial size in the foregoing description) of a target extracted from a sub-map; and Size 目标in子地图2 represents the 3D box size of the same target extracted from another sub-map. The size of a 3D bounding box used to surround the target can be taken as the 3D box size of the target, as shown in FIG. 2. Figure 1 Figure 2 Figure 7

[0125] Step 4: Optimize the camera pose and the spatial map point.

[0126] The following re-projection error model is solved:

[0127]

[0128] where d I = 1 / s I , and s I represents the scale factor of the Ith sub-map.

[0129] By solving the re-projection error model, the optimized camera pose at multiple time points and all optimized spatial map points in the map can be obtained, so that the constructed map is more accurate.

[0130] The embodiments of the present application have at least the following technical effects:​​​​​

[0131] 1) Only images captured by the camera (i.e., visual information) and data measured by the IMU are required. There is no need to introduce sensors that require external communication and obtain external data, and there is no need to implement loop detection. It has a wide range of applications, for example, it can be used in closed and isolated scenarios such as indoor parking lots.

[0132] 2) Using the scale consistency within each submap as a constraint to determine the optimal scale can achieve robust mapping and obtain better mapping results.

[0133] The following continues to describe the exemplary structure of the visual mapping device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the visual mapping device 455 of the memory 450 may include: an acquisition module 4551, used to acquire multiple image frames captured by a camera at multiple moments, and a camera pose of the camera at multiple moments estimated by an inertial measurement unit (IMU); a division module 4552, used to divide a map constructed based on the multiple image frames and the camera pose of the camera at multiple moments to obtain multiple submaps; a scale factor determination module 4553, used to determine the spatial size of the target in each submap, and determine the scale factor of each submap based on the spatial size ratio of the same target between adjacent submaps; an error calculation module 4554, used to calculate the reprojection error of the camera at multiple moments based on the scale factors of the multiple submaps; and an optimization module 4555, used to optimize the reprojection error at multiple moments to obtain optimized spatial map points in the map.

[0134] In some embodiments, the error calculation module 4554 is further used to: reproject the spatial map points observed by the camera at any moment based on the camera's intrinsic parameter matrix, the camera pose at any moment, and the scale factor of the corresponding sub-map at any moment to obtain the reprojection points of the spatial map points; determine the image pixel points corresponding to the spatial map points in the image frame taken at any moment; and determine the error between the reprojection points of the spatial map points and the image pixel points corresponding to the spatial map points as the reprojection error of the camera at any moment.

[0135] In some embodiments, the division module 4552 is further used to: divide the movement trajectory measured by the IMU according to the distance to obtain multiple segmented trajectories; and divide the map into sub-maps corresponding to the multiple segmented trajectories.

[0136] In some embodiments, the scale factor determination module 4553 is further configured to: perform semantic matching on the target in the first submap and the target in the second submap to obtain a same target in the first submap and the second submap; and determine the scale factor of the first submap according to a spatial size ratio of the same target between the first submap and the second submap.

[0137] In some embodiments, the scale factor determination module 4553 is further configured to: perform triangulation on image pixel points of any target in a plurality of image frames used to construct any submap and camera poses at corresponding time points of the plurality of image frames to obtain a plurality of spatial map points of any target in any submap; and perform bounding box processing on the plurality of spatial map points of any target in any submap, and determine a size of the bounding box as a spatial size of any target in any submap.

[0138] In some embodiments, the visual mapping device 455 further includes a screening module configured to: perform screening processing on the plurality of image frames to obtain a plurality of key image frames; and use the plurality of key image frames and camera poses at corresponding time points of the plurality of key image frames to construct the map.

[0139] Embodiments of the present application provide a computer program product or a computer program, which includes executable instructions stored in a computer readable storage medium. A processor of an electronic device reads the executable instructions from the computer readable storage medium, and the processor executes the executable instructions to cause the electronic device to implement the visual mapping method described above in the embodiments of the present application.

[0140] Embodiments of the present application provide a computer readable storage medium storing executable instructions, wherein the executable instructions, when executed by a processor, cause the processor to implement the visual mapping method provided by the embodiments of the present application.

[0141] In some embodiments, the computer readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disc, or CD-ROM, etc.; or various devices including one or any combination of the above memories.

[0142] In some embodiments, the executable instructions can be in the form of a program, software, software module, script, or code, written in any form of programming language (including a compiled or interpreted language, or a declarative or procedural language), and can be deployed in any form, including being deployed as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0143] By way of example, executable instructions can correspond to a file in a file system, but are not necessarily limited thereto. The file can be stored in a portion of a file that holds other programs or data, for example, one or more scripts stored in a markup language, such as Hyper Text Markup Language (HTML), one or more scripts stored in Hyper Text Preprocessor (HTP) files and associated common gateway interface (CGI) form definitions, or one or more scripts stored in object code format for execution by an interpreter. The executable procedures can be stored in a single file, or across multiple files in a file system, for example, files that store code that is downloaded via the World Wide Web or other network from server or other remote source.

[0144] By way of example, executable instructions can be deployed to be executed on one electronic device or multiple electronic devices that are located at one site, or that are distributed across multiple sites and that are interconnected by a communication network.

[0145] The above merely provides example embodiments of the present application, but is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, and improvement made within the spirit and scope of the present application shall fall within the protection scope of the present application.

Claims

1. A visual mapping method, characterized in that: include: Acquire multiple image frames captured by a camera at multiple moments, and camera poses of the camera at the multiple moments estimated by an inertial measurement unit (IMU); Dividing a map constructed according to the multiple image frames and the camera poses of the camera at the multiple moments to obtain multiple submaps; Determine the spatial size of the object in each submap, and determine the scale factor of each submap based on the ratio of the spatial sizes of the same object between adjacent submaps; Calculating reprojection errors of the camera at the multiple time instants according to the scale factors of the multiple submaps; A reprojection error model is solved to obtain optimized camera poses of the camera at the multiple moments and optimized spatial map points in the map; wherein the reprojection error model is used to minimize the reprojection errors at the multiple moments.

2. The method according to claim 1, characterized in that The calculating the reprojection errors of the camera at the multiple time instants according to the scale factors of the multiple submaps includes: Reprojecting the spatial map points observed by the camera at any moment according to the camera's intrinsic parameter matrix, the camera pose at any moment, and the scale factor of the submap corresponding to the camera at any moment to obtain reprojected points of the spatial map points; Determining the image pixel corresponding to the spatial map point in the image frame captured at any time; An error between a reprojection point of the spatial map point and an image pixel corresponding to the spatial map point is determined as a reprojection error of the camera at any moment.

3. The method according to claim 1, characterized in that The map constructed according to the multiple image frames and the camera poses of the camera at the multiple moments is divided to obtain multiple sub-maps, including: The moving trajectory measured by the IMU is divided according to the distance to obtain a plurality of segmented trajectories; The map is divided into sub-maps corresponding to the plurality of segmented trajectories.

4. The method according to claim 1, wherein Before determining the scale factor of each submap based on the spatial size ratio of the same object between adjacent submaps, the method further includes: Performing semantic matching on the target in the first submap and the target in the second submap to obtain the same target in the first submap and the second submap; The second submap is the next submap of the first submap; The step of determining the scale factor of each submap based on the spatial size ratio of the same object between adjacent submaps includes: A scale factor of the first submap is determined according to a spatial size ratio of the same object between the first submap and the second submap.

5. The method according to claim 1, wherein Determining the spatial size of the target in each sub-map includes: Performing triangulation processing based on image pixel points of any target in a plurality of image frames used to construct any submap and camera poses at corresponding moments in the plurality of image frames used to construct the any submap, to obtain a plurality of spatial map points of the any target in the any submap; A plurality of spatial map points of any target in any sub-map are enclosed by a bounding box, and a size of the bounding box is determined as a spatial size of any target in any sub-map.

6. The method according to claim 1, characterized in that After acquiring a plurality of image frames captured by the camera at a plurality of moments and the camera poses of the camera at the plurality of moments estimated by an inertial measurement unit (IMU), the method further includes: Performing screening processing on the multiple image frames to obtain multiple key image frames; The multiple key image frames and the camera poses at the corresponding moments of the multiple key image frames are used to construct a map.

7. A visual mapping device, characterized in that: include: An acquisition module is used to acquire multiple image frames captured by a camera at multiple times, and the camera poses of the camera at the multiple times estimated by an inertial measurement unit (IMU); a partitioning module, configured to partition a map constructed based on the plurality of image frames and the camera poses at the plurality of moments to obtain a plurality of submaps; A scale factor determination module is used to determine the spatial size of the object in each submap and determine the scale factor of each submap based on the spatial size ratio of the same object between adjacent submaps; an error calculation module, configured to calculate the reprojection errors of the camera at the multiple moments according to the scale factors of the multiple submaps; The optimization module is used to optimize the reprojection errors at the multiple moments to obtain optimized spatial map points in the map.

8. An electronic device, characterized in that: include: a memory for storing executable instructions; A processor, configured to implement the method according to any one of claims 1 to 6 when executing the executable instructions stored in the memory.

9. A computer-readable storage medium, characterized in that Executable instructions are stored, and when executed by a processor, they are used to implement the method described in any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises executable instructions for implementing the method according to any one of claims 1 to 6 when executed by a processor.

Images