A vehicle positioning method and device based on visual SLAM
By integrating visual odometer and lane line semantic information, the problem of insufficient robustness of visual SLAM algorithm in autonomous driving is solved, and positioning accuracy and stability is improved. It is suitable for autonomous driving scenarios without lidar vehicles.
Patent Information
- Application Number
- CN202211246923.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-10-12
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-10-12
AI Technical Summary
The existing visual SLAM algorithms are not robust enough in autonomous driving, especially under light changes and noise interference, and cannot ensure positioning accuracy at the decimeter level, and vehicles lacking lidar sensors cannot perform effective positioning.
By obtaining image information during vehicle driving, visual odometer positioning information and lane line semantic information are extracted, and fused to obtain final positioning information, the similarity calculation of SIFT feature descriptor and significant vector is used to enhance vision Robustness and accuracy of positioning.
It improves the accuracy and robustness of visual SLAM positioning, can be stable in a variety of autonomous driving scenarios, makes up for the shortcomings of traditional visual SLAM algorithms, and is suitable for vehicles without lidar sensors.
Smart Images

Figure CN115493612B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of autonomous driving technology, and in particular to a vehicle positioning method based on visual SLAM and a vehicle positioning device based on visual SLAM. Background Art
[0002] With the rapid development of autonomous driving in recent years, its application scenarios are also expanding. At present, the ADAS industry has higher and higher requirements for positioning accuracy, not only requiring accuracy at the decimeter level, but also requiring positioning stability in various autonomous driving scenarios such as sunlight, darkness, rain and snow.
[0003] At present, the visual SLAM positioning algorithm has been valued and promoted by the autonomous driving industry due to its low cost and fast computing speed, and has become the main research direction of multi-source fusion positioning. However, compared with sensors such as lidar, cameras have their own shortcomings, that is, they are more sensitive to light and the features they form are easily disturbed by noise. Therefore, the traditional visual SLAM algorithm is not robust enough and cannot cover the entire scene, which makes the SLAM algorithm limited in the field of autonomous driving.
[0004] The existing technology solution is a positioning solution that integrates vision and lidar, and compensates for the shortcomings of each sensor through the integration of multiple sensors.
[0005] However, many of the existing vehicles may not be hit by the laser radar, so the above method cannot be used to locate the vehicle.
[0006] Therefore, it is hoped that a technical solution can be provided to solve or at least alleviate the above-mentioned deficiencies of the prior art. Summary of the invention
[0007] The object of the present invention is to provide a vehicle positioning method based on visual SLAM to at least solve one of the above technical problems.
[0008] One aspect of the present invention provides a vehicle positioning method based on visual SLAM, the vehicle positioning method based on visual SLAM comprising:
[0009] Acquire image information of the current frame and image information of the previous frame during vehicle driving;
[0010] Obtain visual odometer positioning information based on the image information of the current frame and the image information of the previous frame of the current frame;
[0011] Process the image information of the current frame to obtain the lane line semantic positioning information;
[0012] The lane line semantic positioning information and the visual odometer positioning information are fused to obtain final positioning information.
[0013] Optionally, acquiring visual odometer positioning information according to image information of the current frame and image information of a frame before the current frame includes:
[0014] Acquire SITF feature descriptor similarity according to the image information of the current frame and the image information of the previous frame of the current frame;
[0015] Acquire similarity of salient vectors according to image information of the current frame and image information of a frame before the current frame;
[0016] Acquire combined similarity information according to the similarity of the SITF feature descriptors and the similarity of the salient vectors;
[0017] The visual odometry positioning information is obtained by combining the similarity information.
[0018] Optionally, the acquiring combined similarity information according to the similarity of the SITF feature descriptors and the similarity of the salient vectors includes:
[0019] The following formula is used to obtain the combined similarity information:
[0020] D r =exp(d m +d s );in,
[0021] D r To combine similarity information; d m is the SITF feature descriptor similarity; d s is the similarity of the significant vector; exp refers to the power of e.
[0022] Optionally, the processing of the image information of the current frame to obtain the lane line semantic positioning information includes:
[0023] Acquiring map information, wherein the map information includes lane line semantic features;
[0024] Extract lane line point features from the image information of the current frame;
[0025] The lane line semantic positioning information is obtained according to the lane line point features and the lane line semantic features.
[0026] Optionally, fusing the lane semantic positioning information and the visual odometer positioning information to obtain final positioning information includes:
[0027] Acquire lane line confidence according to the lane line semantic positioning information;
[0028] The lane line confidence, the lane line semantic positioning information and the visual odometer positioning information are fused to obtain the final positioning information.
[0029] Optionally, the fusing the lane line confidence, the lane line semantic positioning information and the visual odometer positioning information to obtain the final positioning information includes:
[0030] When the lane line confidence is greater than a first preset threshold, the lane line semantic positioning information and the visual odometer positioning information are fused.
[0031] Optionally, the fusing the lane line confidence, the lane line semantic positioning information and the visual odometer positioning information to obtain the final positioning information includes:
[0032] When the lane line confidence is less than a first preset threshold and greater than a second preset threshold, obtaining the lane line confidence of the current frame and the lane line confidence of a preset number of frames after the frame;
[0033] According to the lane line confidence of the current frame and the lane line confidence of the preset number of frames before the frame, it is determined whether to fuse the lane line confidence and the visual odometer positioning information. If so, then
[0034] The lane line semantic positioning information and the visual odometer positioning information are fused.
[0035] Optionally, the fusing the lane line confidence, the lane line semantic positioning information and the visual odometer positioning information to obtain the final positioning information includes:
[0036] When the lane line confidence is less than a second preset threshold, generating a weight information for the lane line confidence;
[0037] Fusion is performed according to the weight and the visual odometer positioning information.
[0038] Optionally, the lane confidence and the visual odometer positioning information are fused using the following formula:
[0039] K=PH T (HPH T +V) -1
[0040] x=K(h(x f )-h(x p ))
[0041] P = (I - KH) P; where
[0042] Where x and P are the fused state and covariance matrix, K is the Kalman gain, h is the function mapping from observation space to state space, H is the Jacobian matrix of function h, and V is the observation error; x f Represents the predicted posture state; x p represents the observed posture state, I is the unit matrix, and T represents the transpose of the matrix.
[0043] The present application also provides a vehicle positioning device based on visual SLAM, and the vehicle positioning device based on visual SLAM includes:
[0044] An image information acquisition module, wherein the image information acquisition module is used to acquire image information of a current frame and image information of a frame before the current frame during the vehicle's driving process;
[0045] A visual odometer positioning information acquisition module, wherein the visual odometer positioning information acquisition module is used to acquire visual odometer positioning information based on image information of a current frame and image information of a frame before the current frame;
[0046] A lane line semantic positioning information acquisition module, wherein the lane line semantic positioning information acquisition module is used to process the image information of the current frame to obtain the lane line semantic positioning information;
[0047] A fusion module is used to fuse the lane line semantic positioning information and the visual odometer positioning information to obtain final positioning information.
[0048] Beneficial Effects
[0049] The vehicle positioning method based on visual SLAM of the present application utilizes the raw data (image data) output by the camera sensor and the perception output data (semantic data) as the input source of the SLAM algorithm, and respectively adds the saliency information and the lane line semantic information into the SLAM framework, thereby enhancing the visual semantic information representation capability of the visual positioning SLAM, making up for the shortcomings of the existing methods, and being able to better extract the effective information used for positioning in the image, thereby further improving the accuracy and robustness of the visual SLAM positioning. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 It is a flow chart of a vehicle positioning method based on visual SLAM according to an embodiment of the present application.
[0051] Figure 2 It is a schematic diagram of an electronic device capable of implementing the vehicle positioning method based on visual SLAM according to an embodiment of the present application.
[0052] Figure 3It is a detailed flowchart of a vehicle positioning method based on visual SLAM according to an embodiment of the present application. DETAILED DESCRIPTION
[0053] In order to make the purpose, technical scheme and advantages of the implementation of this application clearer, the technical scheme in the embodiment of this application will be described in more detail below in conjunction with the drawings in the embodiment of this application. In the drawings, the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The described embodiments are part of the embodiments of this application, not all of them. The embodiments described below with reference to the drawings are exemplary and are intended to be used to explain this application, and should not be construed as limitations on this application. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. The embodiments of this application are described in detail below in conjunction with the drawings.
[0054] Figure 1 It is a flow chart of a vehicle positioning method based on visual SLAM according to an embodiment of the present application. Figure 3 It is a detailed flowchart of a vehicle positioning method based on visual SLAM according to an embodiment of the present application.
[0055] like Figure 1 as well as Figure 3 The vehicle localization method based on visual SLAM shown includes:
[0056] Step 1: Obtain image information of the current frame and image information of the previous frame during vehicle driving;
[0057] Step 2: Obtain visual odometer positioning information based on the image information of the current frame and the image information of the previous frame of the current frame;
[0058] Step 3: Process the image information of the current frame to obtain the lane line semantic positioning information;
[0059] Step 4: Fuse the lane line semantic positioning information and the visual odometer positioning information to obtain the final positioning information.
[0060] The vehicle positioning method based on visual SLAM of the present application utilizes the original data output by the camera sensor (for example, image data to obtain visual odometer positioning information) and the perception output data (perception based on the original image to obtain lane line semantic positioning information) as the input source of the SLAM algorithm, and adds the saliency information and lane line semantic information to the SLAM framework respectively, thereby enhancing the visual semantic information representation capability of the visual positioning SLAM, making up for the shortcomings of the existing methods, and being able to better extract effective information from the image for positioning, thereby further improving the accuracy and robustness of visual SLAM positioning.
[0061] In this embodiment, obtaining visual odometer positioning information according to image information of the current frame and image information of a frame before the current frame includes:
[0062] Obtain SITF feature descriptor similarity based on the image information of the current frame and the image information of the previous frame of the current frame;
[0063] Obtaining the similarity of the salient vectors according to the image information of the current frame and the image information of the previous frame of the current frame;
[0064] Obtain combined similarity information based on the similarity of SITF feature descriptors and the similarity of salient vectors;
[0065] The visual odometry positioning information is obtained by combining the similarity information.
[0066] In this embodiment, the following scheme is adopted to obtain the similarity of SITF feature descriptors according to the image information of the current frame and the image information of the previous frame of the current frame:
[0067] First, SIFT features are obtained through dense sampling. This feature descriptor has good stability and invariance, can adapt to changes in rotation, scale, and brightness, and can be unaffected by changes in perspective, affine transformation, and noise to a certain extent. SIFT features can be expressed as:
[0068]
[0069] in,
[0070] is the magnitude of the gradient of each point L(x, y); is the direction of each point L(x, y).
[0071] Extract local blocks from the image in sequence and each block is a SIFT descriptor. n ]∈R D×N Represents the obtained SIFT descriptor. iis the descriptor extracted from the i-th block. The saliency value of each pixel in the local block is calculated by the PQFT method. The PQFT (Phase Spectrum of Quaternion Fourier Transform) model obtains the spatiotemporal saliency map of the image by calculating the phase spectrum of the quaternion Fourier transform of the image. The quaternion image can be expressed as follows:
[0072] q(t)=M(t)+RG(t)μ1+BY(t)μ2+I(t)μ3;
[0073] By calculating the inverse phase information q′(t), we can get
[0074] q′(t)=ρ0(t)+ρ1(t)μ1+ρ2(t)μ2+ρ3(t)μ3; where,
[0075] RG(t) represents the red / green neurons in the human visual cortex in the physiological structure of the human visual system, and BY(t) represents the blue / yellow neurons in the human visual cortex in the physiological structure of the human visual system. I(t) is the brightness feature of the input image F(t) at time t, and M(t) is the motion feature of the input image F(t) at time t;
[0076] q(t) is a quaternion;
[0077] μ i , i=1,2,3
[0078]
[0079] q'(t) is the information obtained by performing the inverse Fourier transform on Q(t).
[0080] The spatiotemporal saliency mapping is sM(t)=g*‖q′(t)‖ 2 The PQFT method can be used to calculate the salient vector of each SIFT descriptor, that is, M = [m1, m2, ..., m N ]∈R S×N .
[0081] At this point, we have extracted the SIFT feature descriptors and the corresponding salient vectors in the image through the feature extraction algorithm for matching and positioning. Finally, in the matching process between the previous and next frames, we calculate the similarity of the SIFT feature descriptors and the salient vectors respectively, and match the feature points based on the similarity. First, the Euclidean distance is used to calculate the similarity of each SIFT feature descriptor and the similarity of the salient vector corresponding to the saliency. The similarity formula of the salient vector is as follows:
[0082] d m =D(mk , m k+1 )=Σ(m kj -m (k+1)j ) 2 ;in,
[0083] d m Represents the similarity of the salient vector, k is the previous frame, k+1 is the current frame, and the larger the value, the smaller the similarity of the corresponding point cloud.
[0084] After obtaining the similarity of SITF feature descriptors and the similarity of salient vectors respectively, the following formula is used to obtain the combined similarity information:
[0085] D r =exp(d m +d s );in,
[0086] D r To combine similarity information; d m is the similarity of the significant vector; d s is the similarity of SITF feature descriptors; exp refers to the power of e.
[0087] In this embodiment, the image generated by the autonomous driving camera is divided into foreground and background. The foreground is often an area with very rich features in autonomous driving, while the background contains a lot of redundant information. For the positioning system, adding saliency information during matching helps to improve the matching effect. In saliency detection, the saliency value of the foreground is often higher than the saliency value of the background; therefore, it can be considered that the information contained in features with similar saliency is often more similar. Therefore, when the visual odometer is matched in SLAM, if the saliency of the features of the previous and next frames is similar, the probability of matching will increase.
[0088] In this embodiment, processing the image information of the current frame to obtain the lane line semantic positioning information includes:
[0089] Obtaining map information, which includes lane line semantic features;
[0090] Extract lane line point features from the image information of the current frame;
[0091] The lane line semantic positioning information is obtained based on the lane line point features and the lane line semantic features.
[0092] The following describes in detail how to obtain the lane line semantic positioning information:
[0093] Positioning is performed through lane line matching. First, we need to deploy high-precision maps on the car. High-precision maps can be provided by map providers such as AutoNavi and Baidu, or they can be collected by ourselves. The map contains the semantic information required for positioning. Lane line matching uses the semantic information output by the current camera to select candidate lane line point features from the map. Those that meet the threshold are candidate features in the map.
[0094] The position of the z-axis needs to be judged: At the same time, it is necessary to judge the positions of the two planes, which must meet the following requirements: Finally, to judge the ground direction, the following conditions must be met:
[0095] where y lm is the location of the lane feature in the map, g lm is the plane normal vector of the point feature on the lane line. th is the threshold parameter, where th dir ,Th d And Th h They are the threshold parameters for direction, height and plane distance respectively.
[0096] Perform a nearest neighbor search on all currently observed lane line point features and map candidate features, and those that meet the threshold are the matched map features and observed features.
[0097] The distance between the observed lane line point feature and the lane line point feature in the map is:
[0098] The direction difference of the lane line is: ‖g lm ×(R new ·g b )‖, the feature pair with the smallest distance and satisfying the threshold condition is the matched observation feature and map feature. In this embodiment, b is the abbreviation of body, which represents the carrier coordinate system. m is the abbreviation of measurement, which represents measurement. R, t represents rotation and translation. Each map feature has coordinates in the map coordinate system. By matching with the map semantic features, the semantic positioning information of the vehicle's lane line can be obtained.
[0099] In this embodiment, the lane semantic positioning information and the visual odometer positioning information are integrated to obtain the final positioning information, including:
[0100] Obtain lane line confidence based on lane line semantic positioning information;
[0101] The lane line confidence, the lane line semantic positioning information and the visual odometer positioning information are fused to obtain the final positioning information.
[0102] In this embodiment, the lane line confidence, the lane line semantic positioning information and the visual odometer positioning information are integrated to obtain the final positioning information, including:
[0103] When the lane line confidence is greater than a first preset threshold, the lane line semantic positioning information and the visual odometer positioning information are fused.
[0104] In this embodiment, the lane line confidence can be obtained through the camera sensing end, which is a prior art and will not be described in detail here.
[0105] In this embodiment, the final positioning information is obtained by fusing the lane line confidence, the lane line semantic positioning information and the visual odometer positioning information, including:
[0106] When the lane line confidence is less than a first preset threshold and greater than a second preset threshold, obtaining the lane line confidence of the current frame and the lane line confidence of a preset number of frames after the frame;
[0107] According to the lane line confidence of the current frame and the lane line confidence of the preset number of frames before the frame, it is determined whether to fuse the lane line confidence and the visual odometer positioning information. If so, then
[0108] The lane line semantic positioning information and the visual odometer positioning information are fused.
[0109] In this embodiment, the lane line confidence, the lane line semantic positioning information and the visual odometer positioning information are integrated to obtain the final positioning information, including:
[0110] When the lane line confidence is less than a second preset threshold, generating a weight information for the lane line confidence;
[0111] Fusion is performed based on weights and visual odometer positioning information.
[0112] In this embodiment, the lane line confidence and the visual odometer positioning information are fused using the following formula:
[0113] K=PH T (HPH T +V) -1 x=K(h(x f )-h(x p ))P=(I-KH)P;wherein,
[0114] Where x and P are the state and covariance matrix after fusion, K is the Kalman gain, h is the function mapping from observation space to state space, H is the Jacobian matrix of function h, and V is the observation error;.
[0115] For example, multi-source fusion is performed on the lane line semantic positioning information and the visual odometer positioning information.
[0116] We use Kalman filtering to perform fusion. And we distinguish different scenarios for autonomous driving. When the lane line is clear and stable, the lane line has higher robustness. The current fusion scheme is judged by the confidence of the lane line. When the confidence of the lane line information is ≥0.7 (the first preset threshold), we fuse the saliency feature positioning information and the lane line positioning information. When the lane line confidence is <0.7 (the first preset threshold) but >0.3 (the second preset threshold), we first evaluate the lane line recognition effect of the first 5 frames. If it is continuously recognized and there is no jump, we multiply the lane line matching result by the weight and fuse it with the saliency feature positioning information. When the lane line confidence is ≤0.3 (the second preset threshold), we only use the saliency feature positioning information as the positioning source, but it is necessary to ensure the consistency of different positioning sources to ensure that the jump between frames before and after positioning does not exceed 10cm. This application uses the following formula for fusion:
[0117] K=PH T (HPH T +V) -1
[0118] x=K(h(x f )-h(x p ))
[0119] P = (I - KH) P; where
[0120] x and P are the states and covariance matrices after fusion, K is the Kalman gain, h is the function mapping the observation space to the state space, H is the Jacobian matrix of function h, and V is the observation error. By fusing the positioning results of feature matching and the positioning results of lane line semantic matching through the above method, a stable positioning effect can be achieved. At the same time, the result obtained by this method is also the theoretically optimal estimate.
[0121] The present application also provides a vehicle positioning device based on visual SLAM, which includes an image information acquisition module, a visual odometer positioning information acquisition module, a lane line semantic positioning information acquisition module and a fusion module; wherein,
[0122] The image information acquisition module is used to acquire the image information of the current frame and the image information of the previous frame of the current frame during the vehicle driving process;
[0123] The visual odometer positioning information acquisition module is used to obtain the visual odometer positioning information according to the image information of the current frame and the image information of the previous frame of the current frame;
[0124] The lane line semantic positioning information acquisition module is used to process the image information of the current frame to obtain the lane line semantic positioning information;
[0125] The fusion module is used to fuse the lane line semantic positioning information and the visual odometer positioning information to obtain the final positioning information.
[0126] It can be understood that the above description of the method is also applicable to the description of the device.
[0127] The present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, the above-mentioned vehicle positioning method based on visual SLAM is implemented.
[0128] The present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the above-mentioned vehicle positioning method based on visual SLAM.
[0129] Figure 2 It is an exemplary structural diagram of an electronic device capable of implementing a vehicle positioning method based on visual SLAM provided according to an embodiment of the present application.
[0130] like Figure 2 As shown, the electronic device includes an input device 501, an input interface 502, a central processor 503, a memory 504, an output interface 505, and an output device 506. The input interface 502, the central processor 503, the memory 504, and the output interface 505 are interconnected through a bus 507, and the input device 501 and the output device 506 are connected to the bus 507 through the input interface 502 and the output interface 505, respectively, and then connected to other components of the electronic device. Specifically, the input device 504 receives input information from the outside, and transmits the input information to the central processor 503 through the input interface 502; the central processor 503 processes the input information based on the computer executable instructions stored in the memory 504 to generate output information, temporarily or permanently stores the output information in the memory 504, and then transmits the output information to the output device 506 through the output interface 505; the output device 506 outputs the output information to the outside of the electronic device for use by the user.
[0131] That is to say, Figure 2 The electronic device shown can also be implemented as comprising: a memory storing computer executable instructions; and one or more processors, which can implement the combination when executing the computer executable instructions. Figure 1 Described is a vehicle localization method based on visual SLAM.
[0132] In one embodiment, Figure 2 The electronic device shown can be implemented to include: a memory 504 configured to store executable program code; and one or more processors 503 configured to run the executable program code stored in the memory 504 to execute the vehicle positioning method based on visual SLAM in the above embodiment.
[0133] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0134] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0135] Computer readable media include permanent and non-permanent, removable and non-removable, and the media can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), data versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.
[0136] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0137] In addition, it is obvious that the word "comprising" does not exclude other units or steps. Multiple units, modules or devices stated in the device claim can also be implemented by one unit or the overall device through software or hardware.
[0138] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and a module, a program segment or a part of a code includes one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes identified in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or the total flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0139] The processor referred to in this embodiment may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.
[0140] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the device / terminal equipment by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0141] In this embodiment, if the module / unit integrated in the device / terminal equipment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. Although the present application is disclosed as above in terms of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.
[0142] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0143] In addition, it is obvious that the word "comprising" does not exclude other units or steps. Multiple units, modules or devices stated in the device claim can also be implemented by one unit or the overall device through software or hardware.
[0144] Although the present invention has been described in detail above with general descriptions and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements may be made thereto based on the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all fall within the scope of protection claimed by the present invention.
Claims
1. A vehicle positioning method based on visual SLAM, characterized in that: The vehicle positioning method based on visual SLAM includes: Acquire image information of the current frame and image information of the previous frame during vehicle driving; Obtain visual odometer positioning information based on the image information of the current frame and the image information of the previous frame of the current frame; Process the image information of the current frame to obtain the lane line semantic positioning information; The lane line semantic positioning information and the visual odometer positioning information are integrated to obtain final positioning information; The obtaining of visual odometer positioning information according to the image information of the current frame and the image information of the previous frame of the current frame comprises: Acquire SITF feature descriptor similarity according to the image information of the current frame and the image information of the previous frame of the current frame; Acquire similarity of salient vectors according to image information of the current frame and image information of a frame before the current frame; Acquire combined similarity information according to the similarity of the SITF feature descriptors and the similarity of the salient vectors; The visual odometry positioning information is obtained by combining the similarity information.
2. The vehicle positioning method based on visual SLAM as claimed in claim 1, characterized in that, The obtaining of combined similarity information according to the similarity of the SITF feature descriptors and the similarity of the salient vectors includes: The following formula is used to obtain the combined similarity information: D r =exp(d m +d s );in, D r To combine similarity information; d m is the SITF feature descriptor similarity; d s is the similarity of the significant vector; exp is the power of exponent e.
3. The vehicle positioning method based on visual SLAM as claimed in claim 2, characterized in that, The processing of the image information of the current frame to obtain the lane line semantic positioning information includes: Acquiring map information, wherein the map information includes lane line semantic features; Extract lane line point features from the image information of the current frame; The lane line semantic positioning information is obtained according to the lane line point features and the lane line semantic features.
4. The vehicle positioning method based on visual SLAM as described in any one of claims 1 to 3, characterized in that: Fusing the lane semantic positioning information and the visual odometer positioning information to obtain final positioning information includes: Acquire lane line confidence according to the lane line semantic positioning information; The lane line confidence, the lane line semantic positioning information and the visual odometer positioning information are fused to obtain the final positioning information.
5. The vehicle positioning method based on visual SLAM as claimed in claim 4, characterized in that, The fusing the lane line confidence, the lane line semantic positioning information and the visual odometer positioning information to obtain the final positioning information includes: When the lane line confidence is greater than a first preset threshold, the lane line semantic positioning information and the visual odometer positioning information are fused.
6. The vehicle positioning method based on visual SLAM as claimed in claim 5, characterized in that, The fusing the lane line confidence, the lane line semantic positioning information and the visual odometer positioning information to obtain the final positioning information includes: When the lane line confidence is less than a first preset threshold and greater than a second preset threshold, obtaining the lane line confidence of the current frame and the lane line confidence of a preset number of frames after the frame; According to the lane line confidence of the current frame and the lane line confidence of the preset number of frames before the frame, it is determined whether to fuse the lane line confidence and the visual odometer positioning information. If so, then The lane line semantic positioning information and the visual odometer positioning information are fused.
7. The vehicle positioning method based on visual SLAM as claimed in claim 6, characterized in that: The fusing the lane line confidence, the lane line semantic positioning information and the visual odometer positioning information to obtain the final positioning information includes: When the lane line confidence is less than a second preset threshold, generating a weight information for the lane line confidence; Fusion is performed according to the weight and the visual odometer positioning information.
8. The vehicle positioning method based on visual SLAM as claimed in claim 6, characterized in that, The lane line confidence and the visual odometer positioning information are fused using the following formula: K=PH T (HPH T +V) -1 x=K(h(x f )-h(x p )) P = (I - KH) P; where Where x and P are the fused state and covariance matrix, K is the Kalman gain, h is the function mapping from observation space to state space, H is the Jacobian matrix of function h, and V is the observation error; x f Represents the predicted posture state; x p represents the observed posture state, I is the unit matrix, and T represents the transpose of the matrix.
9. A vehicle positioning device based on visual SLAM, characterized in that: The vehicle positioning device based on visual SLAM comprises: An image information acquisition module, the image information acquisition module is used to acquire image information of a current frame and image information of a frame before the current frame during vehicle driving; A visual odometer positioning information acquisition module, wherein the visual odometer positioning information acquisition module is used to acquire visual odometer positioning information based on image information of a current frame and image information of a frame before the current frame; A lane line semantic positioning information acquisition module, wherein the lane line semantic positioning information acquisition module is used to process the image information of the current frame to obtain the lane line semantic positioning information; A fusion module, the fusion module is used to fuse the lane semantic positioning information and the visual odometer positioning information to obtain final positioning information; The obtaining of visual odometer positioning information according to the image information of the current frame and the image information of the previous frame of the current frame comprises: Acquire SITF feature descriptor similarity according to the image information of the current frame and the image information of the previous frame of the current frame; Acquire similarity of salient vectors according to image information of the current frame and image information of a frame before the current frame; Acquire combined similarity information according to the similarity of the SITF feature descriptors and the similarity of the salient vectors; The visual odometry positioning information is obtained by combining the similarity information.
