Data association method for semantic VI-SLAM in indoor parking environment

Abstract

The present invention relates to a data association method for semantic VI-SLAM in an indoor parking environment, comprising the following steps: obtaining surround image data of an indoor parking environment, extracting surround semantic objects through semantic segmentation; extracting geometric information and semantic information of the surround semantic objects, respectively calculating geometric errors and semantic errors between the surround semantic objects and pre-acquired surround semantic signs; obtaining geometric error weights and semantic error weights by calculating error detection confidence; constructing an association matrix between surround semantic objects and surround semantic signs, and obtaining an optimal association matrix through tight coupling optimization based on the geometric errors, semantic errors, geometric error weights and semantic error weights, thereby realizing data association of semantic VI-SLAM. Compared with the prior art, the present invention has the advantages of improving the performance of semantic VI-SLAM systems, etc.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a data association method for semantic VI-SLAM in an indoor parking environment. Background Art

[0002] Semantic VI-SLAM (Visual-Inertial Simultaneous Localization and Mapping) is a type of SLAM system that uses visual and inertial sensors to perform real-time positioning and mapping in unknown environments. Unlike traditional SLAM, semantic VI-SLAM adds semantic landmarks to the map, enabling it to identify and locate semantic objects such as parking spaces, speed bumps, and parking space numbers in scenarios such as indoor parking lots, thereby improving the environmental perception capabilities of the autonomous driving system. The core of this system is to build a semantic map and determine the vehicle's position in the environment.

[0003] In semantic VI-SLAM, data association refers to the process of matching the currently observed semantic objects with the semantic landmarks in the existing map. When the system observes m semantic objects at time t, and there are n semantic landmarks in the map, data association needs to estimate the association between each observed object and the corresponding landmark in the map. Effective data association is the key to the precise positioning of the system and is essential for maintaining stable navigation. Incorrect data association will reduce the accuracy of the system's pose estimation, causing the system to diverge, which in turn affects the accuracy and reliability of navigation.

[0004] Existing semantic VI-SLAM usually relies on low-level geometric features (such as corners, lines, and surface patches) to solve the data association problem, but such methods still have the following shortcomings:

[0005] (1) Low-level geometric features are easily affected by changes in their appearance caused by changes in camera view during driving.

[0006] (2) Over time, the accumulation of uncertainty in camera poses and map landmark locations may lead to seemingly reasonable but erroneous associations of currently observed semantic objects.

[0007] (3) Although the appearance of ground semantic objects in indoor parking scenes remains consistent from a top-down perspective, existing data association methods rarely utilize this stable surround semantic feature and fail to fully exploit its advantages. Summary of the invention

[0008] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a data association method for semantic VI-SLAM in an indoor parking environment, so as to solve or partially solve the problem of inaccurate association matching between semantic objects and semantic labels in an indoor parking environment, resulting in unsatisfactory performance of the semantic VI-SLAM system.

[0009] The purpose of the present invention can be achieved by the following technical solutions:

[0010] The present invention provides a data association method for semantic VI-SLAM in an indoor parking environment, comprising the following steps:

[0011] Acquire surround view image data of the indoor parking environment and extract surround view semantic objects through semantic segmentation;

[0012] Extracting geometric information and semantic information of the surround view semantic object, and respectively calculating a geometric error and a semantic error between the surround view semantic object and a pre-acquired surround view semantic marker;

[0013] By calculating the error detection confidence, the geometric error weight and the semantic error weight are obtained;

[0014] An association matrix between surround view semantic objects and surround view semantic markers is constructed, and based on the geometric error, semantic error, geometric error weight and semantic error weight, an optimal association matrix is ​​obtained through tight coupling optimization to achieve data association of semantic VI-SLAM.

[0015] As a preferred technical solution, obtaining surround view image data of an indoor parking environment and extracting surround view semantic objects through semantic segmentation includes the following steps:

[0016] Acquire multi-source images in the indoor parking scene, and obtain surround view image data of the indoor parking environment through transformation based on a pre-built transformation matrix;

[0017] The surround view image data is processed through semantic segmentation to obtain a plurality of surround view semantic objects.

[0018] As a preferred technical solution, the surround view semantic objects include parking spaces, speed bumps and parking space numbers.

[0019] As a preferred technical solution, the construction process of the transformation matrix includes the following steps:

[0020] The camera intrinsic parameters of multi-source images are calibrated to identify the feature points of the shared field of view of adjacent cameras on the chessboard, and the relative position and posture of the camera and the ground are obtained by solving the PnP problem.

[0021] A transformation matrix of the multi-source images is obtained based on the relative positions and postures.

[0022] As a preferred technical solution, the process of extracting the geometric information and semantic information of the surround view semantic object and respectively calculating the geometric error and semantic error between the surround view semantic object and the pre-acquired surround view semantic marker comprises the following steps:

[0023] Extracting geometric features of the surround view semantic object to obtain geometric distribution information of the surround view semantic object;

[0024] Building a Gaussian distribution model based on the geometric distribution information to obtain geometric error information between the surround look semantic object and the surround look semantic marker;

[0025] Constructing a CNN feature learning network, taking the segmented image of the surround view semantic object as input, and obtaining the semantic features of the surround view semantic object;

[0026] Based on the semantic features, error information between the look-around semantic object and the look-around semantic marker is calculated.

[0027] As a preferred technical solution, the geometric error is calculated using the following formula:

[0028]

[0029] in, Indicates at time At that time, Lookaround semantics With Lookaround semantic object The geometric error, and Respectively in time The camera pose and covariance matrix of , Indicates the mean , the variance is Gaussian distribution.

[0030] As a preferred technical solution, the semantic error is calculated using the following formula:

[0031]

[0032] in, Indicates at time At that time, Lookaround semantics With Lookaround semantic object The semantic error of and They are the covariance matrices corresponding to the semantic features and time of the surround semantic object.

[0033] As a preferred technical solution, the process of constructing the association matrix between the look-around semantic objects and the look-around semantic markers includes the following steps:

[0034] Constructing the moment correlation matrix ,in , The time When , the number of surround semantic objects in the surround image data and the number of surround semantic landmarks in the semantic map are is the matrix element in the data incidence matrix, representing the The look-around semantic object and The relevance of the look-around semantic markers, The value is 0 or 1. When the value is 0, it means there is no association, and when the value is 1, it means there is an association, and the association matrix satisfies .

[0035] As a preferred technical solution, the optimal correlation matrix is ​​obtained by solving the following problems:

[0036]

[0037] in, represents the optimal incidence matrix, , Represent the geometric error weight and semantic error weight respectively, , They represent geometric error and semantic error respectively.

[0038] As a preferred technical solution, the optimal correlation matrix is ​​obtained by solving the Newton method, Gauss-Newton method or Levenberg-Marquardt method.

[0039] Compared with the prior art, the present invention has at least the following beneficial effects:

[0040] Improving the performance of semantic VI-SLAM systems: Considering that in an indoor parking environment, since the appearance of each surround object is roughly the same when viewed from top to bottom, the present invention integrates the geometric and semantic information of semantic objects detected in the surround image into a tightly coupled model, so that the proposed invention is more robust when the vehicle is driving in a complex indoor parking environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Flow chart of a data association method for semantic VI-SLAM in an indoor parking environment in an embodiment;

[0042] Figure 2 A flowchart of calculating the geometric error and semantic error between look-around semantic objects in an embodiment;

[0043] Figure 3 Schematic diagram of an electronic device in an embodiment. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0045] Example 1

[0046] In view of the problems existing in the above-mentioned prior art, this embodiment provides a data association method for semantic VI-SLAM in an indoor parking environment, see Figure 1 , the method comprises the following steps:

[0047] Step S1, obtaining a surround RGB image of an indoor parking environment through a surround camera system, and extracting surround semantic objects using a semantic segmentation method.

[0048] In this embodiment, the surround view camera system includes four wide-angle fisheye cameras located at the front, rear, left and right. The four wide-angle fisheye cameras are used to obtain surround view RGB images of the indoor parking environment. The camera model is ov10640-490, the resolution is 416416, and the frequency is 10Hz.

[0049] Preferably, step S1 may include steps S101-S104:

[0050] Step S101, using the chessboard method to calibrate the internal parameters of four wide-angle fisheye cameras, and constructing a PnP (Perspective-n-Point) problem through the feature points of the shared field of view of two adjacent wide-angle fisheye cameras on the chessboard, to solve the relative position of the wide-angle fisheye camera and the ground;

[0051] Step S102, solving the transformation matrix of the fisheye RGB image according to the relative posture in step S101;

[0052] Step S103, transforming the fisheye RGB images generated by four wide-angle fisheye cameras located in the indoor parking scene into surround view RGB images according to the internal reference of step S101 and the transformation matrix of step S102;

[0053] Step S104, using a semantic segmentation algorithm to process the surround view RGB image acquired in step S103 to segment surround view semantic objects such as parking spaces, speed bumps, and parking space numbers.

[0054] Preferably, step S101 and step S102 belong to the system offline process, and their parameters can be solved in advance.

[0055] Preferably, step S103 and step S104 belong to an online process of the system, which generates a surround view RGB image online and extracts surround view semantic objects in the surround view RGB image.

[0056] Step S2, calculating the geometric error and semantic error between the surround view semantic objects according to the geometric information and semantic information of the surround view semantic objects.

[0057] Preferably, see Figure 2 , step S2 may include steps S201-S204:

[0058] Step S201 : extracting geometric features for each surround view semantic object to obtain the geometric distribution of the surround view semantic object.

[0059] Step S202: construct a Gaussian distribution model for the geometric distribution of the surround view semantic objects in step S201 to obtain the geometric errors between the surround view semantic objects. The specific contents include:

[0060] At time , the probability distribution of the th ...

[0061]

[0062] in, and are the camera pose and covariance matrix at time, respectively;

[0063] Therefore, the geometric error between look-around semantic objects is:

[0064]

[0065] Step S203: construct a CNN feature learning network for the surround view semantic object to obtain the semantic features of the surround view semantic object.

[0066] Step S204, feature representation is performed on the semantic features of step S203 to obtain the semantic errors between the surrounding semantic objects. The specific contents include:

[0067] In time , No. Lookaround semantics flag The corresponding Lookaround semantic objects The semantic error is expressed as:

[0068]

[0069] in, and They are the feature representation of the surround semantic object and the time The corresponding covariance matrix.

[0070] Step S3, calculating the error detection confidence of the surround semantic object to obtain the optimized weights of the geometric error and the semantic error.

[0071] Considering the different types of errors and measurement reliability involved in step S2, step S3 will calculate the detection confidence of the geometric error and the semantic error for the surround semantic object respectively, and calculate the error weight according to the error detection confidence.

[0072] Step S4, constructing a data association matrix for the look-around semantic object, combining the geometric error, semantic error and optimization weight, and solving the optimal data association matrix through tightly coupled optimization.

[0073] Preferably, step S4 may include steps S401-S404:

[0074] S401, in the time correlation matrix Constructing the data association matrix ,in:

[0075] It's time The number of surround semantic objects in the surround RGB image when ;

[0076] It's time , the number of glance semantic landmarks in the semantic map;

[0077] is the matrix element in the data incidence matrix, representing the Look around semantic objects and the relevance of semantic markers;

[0078] The value is 0 or 1. When the value is 0, it means there is no association, and when the value is 1, it means there is an association.

[0079] Data Correlation Matrix Constrained to a single lookaround semantic object is only associated with a single semantic flag, and each row can only have one element that is 1, that is:

[0080]

[0081] Step S402, constructing an optimization objective function based on the geometric error, semantic error and error weight obtained in steps S2 and S3, and optimizing the data association matrix of step S401, that is:

[0082]

[0083] in, is the optimal data association matrix solved by tightly coupled optimization.

[0084] The solution of the tightly coupled optimization method is solved by gradient descent method, including Newton's method, Gauss-Newton method or Levenberg-Marquardt method.

[0085] To verify the effectiveness of this method, this method is embedded in VIS SLAM (Shao et al., A tightly-coupled semantic SLAM system with visual, inertial and surround-view sensors for autonomous indoor parking, 2020) to evaluate the difference in system performance when using the data association method of semantic VI-SLAM in an indoor parking environment with and without this method.

[0086] Among them, VIS SLAM It is a tightly coupled semantic VI-SLAM system suitable for indoor parking environments.

[0087] VIS SLAM Deployed on an electric vehicle equipped with a forward-looking camera, IMU, and ring camera system.

[0088] The electric vehicle will be driven at a speed of 20 kilometers per hour in a typical indoor parking lot. At the same time, the perception data of the front-view camera, IMU and surround camera system will be recorded during the driving process of the electric vehicle for data analysis in subsequent experiments.

[0089] The real trajectory of electric vehicles in indoor parking lots is pre-calculated by tracking artificial markers (AprilTags QR codes) in indoor parking lots. In addition, to ensure the accuracy of the real trajectory, the coordinates of the artificial markers will be measured and recorded by a high-precision ETS (Electronic Total Station).

[0090] (1) Qualitative results of the experimental example:

[0091] The indoor parking environment consists of two rows of parking spaces. Each row consists of 12 parking spaces, and two consecutive parking spaces are adjacent to each other. Each parking space is represented by a unique parking space ID. In addition, there are two speed bumps in this indoor parking environment. SLAM The optimization results lead to the following conclusions:

[0092] 1. When the data association method of this embodiment is not included in VIS SLAM During the optimization process, since the scale of IMU estimation is difficult to achieve absolute accuracy, not all semantic objects can be correctly associated in the constructed map.

[0093] 2. When the data association method of this embodiment is incorporated into VIS SLAM During the optimization process, it is able to cluster different semantic objects of the same parking space, and these different semantic objects can be correctly associated to construct a unique semantic landmark in the map.

[0094] 3. When the data association method of this embodiment is incorporated into VIS SLAM During the optimization process, the estimated trajectory and the ground truth trajectory roughly coincide with each other, and the positioning result is highly accurate, which proves the effectiveness of the data association method.

[0095] (2) Quantitative results of experimental examples:

[0096] In order to quantitatively evaluate the SLAM The absolute trajectory error is defined as the difference between the trajectory obtained by the data association method of semantic VI-SLAM in the indoor parking environment of this embodiment and the actual trajectory during the optimization process. , as follows:

[0097]

[0098] in, is the total number of camera frames, represents the translation part of the camera pose, and are the true camera pose and the estimated camera pose for all camera frames, respectively.

[0099] According to the absolute trajectory error calculation formula, calculate the VIS SLAM The absolute trajectory error results of the optimization process without and with the data association method of this embodiment are shown in Table 1:

[0100] Table 1 Comparison of absolute trajectory error results

[0101]

[0102] Among them, SLAM ∅ VIS that does not adopt the data association method of this embodiment SLAM , its absolute trajectory error is 0.335m. SLAM DA VIS representing the data association method of this embodiment SLAM , and its absolute trajectory error is 0.221m.

[0103] By comparing the absolute trajectory errors of the two, the data association method of this embodiment can significantly reduce the positioning error in semantic VI-SLAM, and its magnitude can reach 34%.

[0104] In addition, by comparing SLAM ∅ and SLAM DA The average processing time of the data association method is only increased by 0.6ms, which is only a small part of the overall average processing time. Therefore, the data association method is resource-saving and suitable for inclusion in any semantic VI-SLAM system.

[0105] In order to further evaluate the data association method for VIS SLAM In order to improve the performance of the proposed method, in addition to the absolute trajectory error, the revisit error is also introduced as a performance evaluation indicator.

[0106] Among them, the revisit error quantifies the change of positioning results at different times. When it is difficult to obtain an accurate true trajectory, the revisit error can be used as a useful positioning evaluation indicator. In order to calculate the revisit error, three test points are first selected in the map, and then the real position of the electric vehicle when it revisits the test point is counted according to the number of laps driven by the electric vehicle. The difference between the real position and the position of the electric vehicle using the data association method VIS is calculated. SLAM The difference between the estimated positions of The experimental results are shown in Table 2:

[0107] Table 2 Comparison of estimated position difference results

[0108]

[0109] The unit of Table 2 is m, and the statistics are based on the data association method VIS SLAM The revisit error is about 0.081m.

[0110] In order to further prove the effectiveness of the optimization strategy of this method, an ablation experiment is conducted. The experimental results are shown in Table 3:

[0111] Table 3 Comparison of ablation experiment results

[0112]

[0113] Among them, SLAM ∅ Represents VIS that does not use this data association method SLAM , its absolute trajectory error is 0.335m and the revisit error is 0.185m. SLAM G VIS represents the method of using only geometric error tightly coupled optimization data association SLAM , its absolute trajectory error is 0.270m and the revisit error is 0.132m. SLAM SVIS represents the method of optimizing data association using only semantic error tight coupling SLAM , its absolute trajectory error is 0.285m and the revisit error is 0.102m. DA VIS stands for Data Association Method SLAM , its absolute trajectory error is 0.221m and the revisit error is 0.081m.

[0114] According to Table 3, compared with SLAM ∅ In comparison, SLAM G and SLAM S The absolute trajectory errors of SLAM are significantly reduced by 0.065m and 0.050m respectively, proving the effectiveness of geometric error and semantic error in data association. DA The absolute trajectory error and revisit error are greatly improved, which proves the effectiveness of the data association method of this embodiment.

[0115] In summary, the experiments demonstrate the effectiveness of the data association method for semantic VI-SLAM in indoor parking environments.

[0116] The method integrates the geometric and semantic information of surround view semantic objects detected in the surround view image into a tightly coupled model. Since each surround view object has roughly the same appearance when viewed from top to bottom, the method provided by the embodiment is more robust when the vehicle is driving in a complex indoor parking environment. In addition, experiments conducted in a realistic indoor parking environment show that the method significantly improves the performance of the semantic VI-SLAM system in the indoor parking environment.

[0117] Example 2

[0118] See also Figure 3 This embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the data association method for semantic VI-SLAM in an indoor parking environment as described in Example 1.

[0119] like Figure 3 As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1Of course, in addition to the software implementation, the present invention does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0120] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0121] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0122] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A data association method for semantic VI-SLAM in indoor parking environment, characterized in that: The steps include: Acquire surround view image data of the indoor parking environment and extract surround view semantic objects through semantic segmentation; Extracting geometric information and semantic information of the surround view semantic object, and respectively calculating a geometric error and a semantic error between the surround view semantic object and a pre-acquired surround view semantic marker; By calculating the error detection confidence, the geometric error weight and the semantic error weight are obtained; Constructing an association matrix between surround view semantic objects and surround view semantic landmarks, and obtaining an optimal association matrix through tight coupling optimization based on the geometric error, semantic error, geometric error weight, and semantic error weight, thereby realizing data association of semantic VI-SLAM. The process of constructing the association matrix between look-around semantic objects and look-around semantic landmarks includes the following steps: Build The correlation matrix of the moment ,in , The time When , the number of surround semantic objects in the surround image data and the number of surround semantic landmarks in the semantic map are is the matrix element in the data incidence matrix, representing the The look-around semantic object and The association of the lookaround semantic flags is 0 or 1. When the value is 0, it means there is no association, and when the value is 1, it means there is an association, and the association matrix satisfies , The optimal incidence matrix is ​​obtained by solving the following problem: ， in, represents the optimal incidence matrix, , Represent the geometric error weight and semantic error weight respectively, , They represent geometric error and semantic error respectively.

2. The data association method for semantic VI-SLAM in an indoor parking environment according to claim 1, characterized in that: Acquiring surround view image data of an indoor parking environment and extracting surround view semantic objects through semantic segmentation includes the following steps: Acquire multi-source images in the indoor parking scene, and obtain surround view image data of the indoor parking environment through transformation based on a pre-built transformation matrix; The surround view image data is processed through semantic segmentation to obtain a plurality of surround view semantic objects.

3. The data association method for semantic VI-SLAM in an indoor parking environment according to claim 2, characterized in that: The surround view semantic objects include parking spaces, speed bumps and parking space numbers.

4. The data association method for semantic VI-SLAM in an indoor parking environment according to claim 2, characterized in that: The construction process of the transformation matrix includes the following steps: The camera intrinsic parameters of multi-source images are calibrated to identify the feature points of the shared field of view of adjacent cameras on the chessboard, and the relative pose between the camera and the ground is obtained by solving the PnP problem. A transformation matrix of the multi-source images is obtained based on the relative positions and postures.

5. The data association method for semantic VI-SLAM in an indoor parking environment according to claim 1, characterized in that: The process of extracting the geometric information and semantic information of the surround view semantic object and respectively calculating the geometric error and semantic error between the surround view semantic object and the pre-acquired surround view semantic marker comprises the following steps: Extracting geometric features of the surround view semantic object to obtain geometric distribution information of the surround view semantic object; Building a Gaussian distribution model based on the geometric distribution information to obtain geometric error information between the surround look semantic object and the surround look semantic marker; Constructing a CNN feature learning network, taking the segmented image of the surround view semantic object as input, and obtaining the semantic features of the surround view semantic object; Based on the semantic features, semantic error information between the look-around semantic object and the look-around semantic marker is calculated.

6. The data association method for semantic VI-SLAM in an indoor parking environment according to claim 5, characterized in that: The geometric error is calculated using the following formula: ， in, Indicates at time At that time, Lookaround semantics With Lookaround semantic object The geometric error, and Respectively in time The camera pose and covariance matrix of , It means the mean , the variance is Gaussian distribution.

7. The data association method for semantic VI-SLAM in an indoor parking environment according to claim 5, characterized in that: The semantic error is calculated using the following formula: ， in, Indicates at time At that time, Lookaround semantics With Lookaround semantic object The semantic error and are the semantic features and time of the surround semantic object The corresponding covariance matrix.

8. The data association method for semantic VI-SLAM in an indoor parking environment according to claim 1, characterized in that: The optimal incidence matrix is ​​obtained by Newton's method, Gauss-Newton method or Levenberg-Marquardt method.

Images