Vehicle positioning method, device and equipment, storage medium and program product

Abstract

The invention discloses a vehicle positioning method, device and equipment, a storage medium and a program product, and belongs to the technical field of intelligent driving, and the method comprises the steps: obtaining the current vehicle state information and the current image of a target vehicle at the current moment, and the historical vehicle state information and the historical image of the target vehicle at the previous moment of the current moment; iMU pre-integration processing is carried out on the current vehicle state information and the historical vehicle state information, and a pre-integration result is obtained; processing the current image and the historical image through an image matching model to obtain a matching relation set between the current image and the historical image; determining a fusion constraint function according to the current vehicle state information, the historical vehicle state information, the pre-integration result and the feature point position information of each matched feature point in the matching relation set; and positioning the target vehicle according to the fusion constraint function. According to the invention, the cost is reduced, and the vehicle positioning accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving technology, and in particular to a vehicle positioning method, device, equipment, storage medium and program product. Background Art

[0002] Positioning technology is the core technology for realizing vehicle autonomous driving. It estimates the vehicle's posture information in real time through various sensor data. It is of great significance to the autonomous decision-making, path planning and underlying integrated control of intelligent driving vehicles.

[0003] However, the existing vehicle positioning method based on multi-sensor data fusion requires the use of lidar sensors and image sensors to obtain data, which has high hardware costs. In addition, the data obtained by the image sensor is easily interfered by external factors such as light, and the accuracy of the vehicle positioning results cannot be guaranteed. Summary of the invention

[0004] The present invention provides a vehicle positioning method, device, equipment, storage medium and program product to reduce costs and improve the accuracy of vehicle positioning.

[0005] According to one aspect of the present invention, a vehicle positioning method is provided, the method comprising:

[0006] Obtaining the current vehicle state information and current image of the target vehicle at the current moment, as well as the historical vehicle state information and historical image of the target vehicle at the previous moment at the current moment;

[0007] Perform IMU pre-integration processing on the current vehicle state information and the historical vehicle state information to obtain a pre-integration result;

[0008] The current image and the historical images are processed by an image matching model to obtain a matching relationship set between the current image and the historical images;

[0009] Determine a fusion constraint function according to current vehicle state information, historical vehicle state information, pre-integration results, and feature point position information of each matching feature point in a matching relationship set;

[0010] According to the fusion constraint function, the target vehicle is located.

[0011] According to another aspect of the present invention, a vehicle positioning device is provided, the device comprising:

[0012] A data acquisition module is used to acquire the current vehicle state information and current image of the target vehicle at the current moment, as well as the historical vehicle state information and historical image of the target vehicle at the previous moment at the current moment;

[0013] A pre-integration result determination module is used to perform IMU pre-integration processing on current vehicle state information and historical vehicle state information to obtain a pre-integration result;

[0014] A matching relationship set determination module is used to process the current image and the historical image through an image matching model to obtain a matching relationship set between the current image and the historical image;

[0015] A fusion constraint function determination module is used to determine the fusion constraint function according to the current vehicle state information, the historical vehicle state information, the pre-integration result and the feature point position information of each matching feature point in the matching relationship set;

[0016] The vehicle positioning module is used to locate the target vehicle according to the fusion constraint function.

[0017] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0018] at least one processor; and

[0019] a memory communicatively connected to at least one processor; wherein,

[0020] The memory stores a computer program that can be executed by at least one processor. The computer program is executed by at least one processor so that the at least one processor can execute the vehicle positioning method according to any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the vehicle positioning method according to any embodiment of the present invention when executed.

[0022] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, and when the computer program is executed by a processor, the vehicle positioning method according to any embodiment of the present invention is implemented.

[0023] The technical solution of the embodiment of the present invention is to obtain the current vehicle status information and the current image of the target vehicle at the current moment, and the historical vehicle status information and the historical image of the target vehicle at the previous moment of the current moment; perform IMU pre-integration processing on the current vehicle status information and the historical vehicle status information to obtain a pre-integration result; process the current image and the historical image through an image matching model to obtain a matching relationship set between the current image and the historical image; determine a fusion constraint function based on the current vehicle status information, the historical vehicle status information, the pre-integration result and the feature point position information of each matching feature point in the matching relationship set; and locate the target vehicle based on the fusion constraint function. The above technical scheme obtains the pre-integration result by performing IMU pre-integration processing on the current vehicle status information and the historical vehicle status information, thereby reducing the influence of noise and error on the vehicle position and vehicle posture estimation, avoiding frequent calculations due to re-integration, and reducing the amount of calculation; the current image and the historical image are processed by the image matching model to obtain the matching relationship set between the current image and the historical image, effectively solving the interference of external factors such as lighting on the image, and improving the accuracy of matching two adjacent frames of images in complex scenes; the target vehicle is positioned by fusing the constraint function, making full use of the data from the IMU, image sensor and GPS sensor, improving the accuracy of vehicle positioning, and there is no need to obtain data with the help of a lidar sensor, thereby reducing costs.

[0024] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 is a flow chart of a vehicle positioning method provided according to Embodiment 1 of the present invention;

[0027] Figure 2A is a flow chart of a vehicle positioning method provided according to Embodiment 2 of the present invention;

[0028] Figure 2B is a structural schematic diagram of an image matching model provided according to Embodiment 2 of the present invention;

[0029] Figure 3is a structural schematic diagram of a vehicle positioning device provided according to Embodiment 3 of the present invention;

[0030] Figure 4 It is a structural schematic diagram of an electronic device for implementing the vehicle positioning method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "target", "current", "historical", "first" and "second" in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] In addition, it should be noted that in the technical solution of the present invention, the collection, storage, use, processing, transmission, provision and disclosure of image matching models and feature point location information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0034] Embodiment 1

[0035] Figure 1 This is a flow chart of a vehicle positioning method provided in Embodiment 1 of the present invention. This embodiment can be applied to the situation of accurately positioning an autonomous driving vehicle in a complex scene. The method can be executed by a vehicle positioning device, which can be implemented in the form of hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0036] S101, obtaining current vehicle state information and current image of a target vehicle at a current moment, and historical vehicle state information and historical image of the target vehicle at a previous moment at the current moment.

[0037] The target vehicle refers to the autonomous driving vehicle to be positioned. The current image refers to the image of the surrounding environment of the target vehicle captured at the current moment. The historical image refers to the image of the surrounding environment of the target vehicle captured at the previous moment before the current moment.

[0038] Among them, the current vehicle status information refers to the status information of the target vehicle at the current moment; optionally, the current vehicle status information includes but is not limited to the current vehicle speed, the current vehicle acceleration and the current vehicle angular velocity. Correspondingly, the historical vehicle status information refers to the status information of the target vehicle at the previous moment of the current moment; optionally, the historical vehicle status information includes but is not limited to the historical vehicle acceleration, the historical vehicle angular velocity, the historical GPS observed position and the historical GPS observed attitude. Among them, GPS (Global Positioning System) is a medium-range circular orbit satellite navigation system based on artificial earth satellites, providing high-precision radio navigation and positioning services.

[0039] Specifically, the current vehicle state information and current image of the target vehicle at the current moment can be obtained through the inertial measurement unit (IMU), image sensor and GPS sensor installed in the target vehicle, and the historical vehicle state information and historical image of the target vehicle at the previous moment at the current moment can be obtained from the data storage unit in the target vehicle. It should be noted that in order to ensure the consistency of time when different sensors measure data, a unified clock source is used to synchronize the time of different sensors (i.e., IMU, image sensor and GPS sensor).

[0040] S102, performing IMU pre-integration processing on the current vehicle state information and the historical vehicle state information to obtain a pre-integration result.

[0041] Among them, the current vehicle state information includes the current vehicle position and the current vehicle posture; the historical vehicle state information includes the historical vehicle position, the historical vehicle speed and the historical vehicle posture; the pre-integration result includes the position change and the posture change.

[0042] Specifically, IMU pre-integration processing can be performed on the current vehicle position, historical vehicle position, historical vehicle speed and historical vehicle posture to obtain the position change; IMU pre-integration processing can be performed on the historical vehicle posture and the current vehicle posture to obtain the posture change.

[0043] More specifically, the current vehicle position, historical vehicle position, historical vehicle speed, and historical vehicle posture are processed by IMU pre-integration to obtain the position change:

[0044]

[0045] Among them, i represents the previous moment of the current moment, j represents the current moment, and α ij Represents the position change, R i represents the historical vehicle posture, Represents R i The transpose of j represents the current vehicle position, p i represents the historical vehicle position, v i represents the historical vehicle speed, Δt ij It represents the time interval between the current moment and the previous moment, and g represents the acceleration due to gravity.

[0046] At the same time, the historical vehicle posture and the current vehicle posture are processed by IMU pre-integration through the following formula to obtain the posture change:

[0047]

[0048] Among them, i represents the previous moment of the current moment, j represents the current moment, and δ ij Indicates the amount of posture change, R i represents the historical vehicle posture, Represents R i The transpose of R j Indicates the current vehicle posture.

[0049] It should be noted that the pre-integration result also includes the speed change. Optionally, the current vehicle speed, historical vehicle speed and historical vehicle posture can be processed by IMU pre-integration to obtain the speed change by the following formula:

[0050]

[0051] Among them, i represents the previous moment of the current moment, j represents the current moment, β ij Indicates the speed change, R i represents the historical vehicle posture, Represents R i The transpose of v j represents the current vehicle speed, v i represents the historical vehicle speed, Δt ij It represents the time interval between the current moment and the previous moment, and g represents the acceleration due to gravity.

[0052] It can be understood that compared with the traditional IMU integration method, that is, the method of integrating the current vehicle status information and the historical vehicle status information to obtain the vehicle position and vehicle posture, the current vehicle status information and the historical vehicle status information are subjected to IMU pre-integration processing, so that the pre-integration result is only related to the current vehicle status information at the current moment and the historical vehicle status information at the previous moment. This not only reduces the impact of noise and error on the estimation of vehicle position and vehicle posture, but also avoids frequent calculations caused by re-integration, thereby reducing the amount of calculation.

[0053] S103: Process the current image and the historical images through an image matching model to obtain a matching relationship set between the current image and the historical images.

[0054] The image matching model can be obtained by training the machine learning model through sample images of two adjacent frames. The matching relationship set refers to a set used to record the matching feature points in the current image and the historical image. The matching feature points refer to the feature points that are exactly the same in the current image and the historical image.

[0055] Specifically, the current image and the historical images can be preprocessed according to the image input format of the image matching model to obtain the preprocessed current image and the historical images; the preprocessed current image and the historical images are input into the image matching model, and the matching relationship set between the current image and the historical images is obtained after processing by the image matching model.

[0056] S104, determining a fusion constraint function according to the current vehicle state information, the historical vehicle state information, the pre-integration result, and the feature point position information of each matching feature point in the matching relationship set.

[0057] The current vehicle state information also includes the current GPS observation position and the current GPS observation posture. The feature point position information includes the position information of the matching feature points in the image coordinate system and the position information of the matching feature points in the three-dimensional space. It should be noted that the position information of the matching feature points in the three-dimensional space can be obtained by triangulation method. The fusion constraint function refers to a function used to assist vehicle positioning.

[0058] Specifically, the vehicle position constraint can be determined according to the current vehicle position, historical vehicle position, historical vehicle posture, historical vehicle speed, position change, and the time interval between the current moment and the previous moment; the vehicle posture constraint can be determined according to the current vehicle posture, historical vehicle posture and posture change; the GPS error can be determined according to the current vehicle position, current vehicle posture, current GPS observation position and current GPS observation posture; based on a preset camera projection model, the reprojection error can be determined according to the current vehicle position, current vehicle posture and the feature point position information of each matching feature point in the matching relationship set; the fusion constraint function can be determined according to the vehicle position constraint, vehicle posture constraint, GPS error and reprojection error.

[0059] More specifically, the vehicle position constraint is determined by the following formula based on the current vehicle position, historical vehicle position, historical vehicle posture, historical vehicle speed, position change, and the time interval between the current moment and the previous moment:

[0060] f p =p j -p i -R i α ij +v i Δt ij -0.5gΔt ij 2 ;

[0061] Among them, i represents the previous moment of the current moment, j represents the current moment, and f p represents the vehicle position constraint, p j represents the current vehicle position, p i represents the historical vehicle position, R i represents the historical vehicle posture, α ij Represents the position change, v i represents the historical vehicle speed, Δt ij It represents the time interval between the current moment and the previous moment, and g represents the acceleration due to gravity.

[0062] Afterwards, the vehicle posture constraint is determined according to the current vehicle posture, historical vehicle posture and posture change through the following formula:

[0063]

[0064] Among them, i represents the previous moment of the current moment, j represents the current moment, and f R represents the vehicle posture constraint, R j Represents the current vehicle posture, R i represents the historical vehicle posture, Represents R i The transpose ofij Indicates the amount of posture change.

[0065] Afterwards, the GPS error is determined by the following formula based on the current vehicle position, current vehicle attitude, current GPS observation position and current GPS observation attitude:

[0066]

[0067] Among them, j represents the current time, f gps represents GPS error, p j Indicates the current vehicle position, p gps Indicates the current GPS observation position, R j Represents the current vehicle posture, R gps Indicates the current GPS observation attitude, Represents R gps The transpose of .

[0068] Afterwards, based on the preset camera projection model, the reprojection error is determined by the following formula according to the current vehicle position, the current vehicle posture and the feature point position information of each matching feature point in the matching relationship set:

[0069]

[0070] Among them, f rep represents the reprojection error, m represents the mth matching feature point in the matching relationship set, M represents the total number of matching feature points in the matching relationship set, f represents the position information of the mth matching feature point in the matching relationship set in the image coordinate system, r represents the matching relationship set, Ω() represents the preset camera projection model, R c is the current vehicle posture in the image coordinate system, R cT Represents R c The transpose of c It is the representation of the current vehicle position in the image coordinate system.

[0071] Then, the vehicle position constraint, vehicle attitude constraint, GPS error and reprojection error are fused to obtain the fusion constraint function, namely:

[0072] f fuse =f p +f R +f gps +f rep ;

[0073] Among them, f fuse represents the fusion constraint function, f p represents the vehicle position constraint, f R represents the vehicle posture constraint, f gps represents the GPS error, frep Represents the reprojection error.

[0074] S105: Positioning the target vehicle according to the fusion constraint function.

[0075] Specifically, the LM method (Levenberg-Marquardt method) can be used to iteratively optimize the fusion constraint function to continuously reduce the value of the fusion constraint function; when the value of the fusion constraint function is less than a preset threshold, or the number of iterations reaches a preset number threshold, the optimization of the fusion constraint function is stopped, and the vehicle position estimation value and vehicle posture estimation value obtained at this time are used as the final vehicle position and final vehicle posture of the target vehicle. Among them, the preset threshold and the preset number threshold can be pre-set according to actual business needs or the experience of those skilled in the art, and the embodiment of the present invention does not specifically limit them.

[0076] The technical solution of the embodiment of the present invention is to obtain the current vehicle status information and the current image of the target vehicle at the current moment, and the historical vehicle status information and the historical image of the target vehicle at the previous moment of the current moment; perform IMU pre-integration processing on the current vehicle status information and the historical vehicle status information to obtain a pre-integration result; process the current image and the historical image through an image matching model to obtain a matching relationship set between the current image and the historical image; determine a fusion constraint function based on the current vehicle status information, the historical vehicle status information, the pre-integration result and the feature point position information of each matching feature point in the matching relationship set; and locate the target vehicle based on the fusion constraint function. The above technical scheme performs IMU pre-integration processing on the current vehicle status information and the historical vehicle status information to obtain the pre-integration result, thereby reducing the influence of noise and error on the vehicle position and vehicle posture estimation, avoiding frequent calculations due to re-integration, and reducing the amount of calculation; the current image and the historical image are processed by the image matching model to obtain the matching relationship set between the current image and the historical image, effectively solving the interference of external factors such as lighting on the image, and improving the accuracy of matching two adjacent frames of images in complex scenes; the target vehicle is positioned by fusing the constraint function, making full use of the data from the IMU, image sensor and GPS sensor, improving the accuracy of vehicle positioning, and there is no need to use a lidar sensor to obtain data, thereby reducing costs.

[0077] Embodiment 2

[0078] Figure 2AThis is a flow chart of a vehicle positioning method provided in Example 2 of the present invention. Based on the above example, this example further optimizes "processing the current image and the historical image through the image matching model to obtain the matching relationship set between the current image and the historical image" and provides an optional implementation scheme. It should be noted that for the parts not described in detail in the example of the present invention, reference can be made to the relevant descriptions of other examples. It should also be noted that see Figure 2B The image matching model 20 includes a first feature extraction module 21, a second feature extraction module 22, a matching degree calculation module 23 and a matching degree optimization module 24; wherein the first feature extraction module 21 is connected to the second feature extraction module 22, the second feature extraction module 22 is connected to the matching degree calculation module 23; the matching degree calculation module 23 is connected to the matching degree optimization module 24. Figure 2A As shown, the method includes:

[0079] S201, obtaining the current vehicle state information and current image of the target vehicle at the current moment, and the historical vehicle state information and historical image of the target vehicle at the previous moment at the current moment.

[0080] S202, performing IMU pre-integration processing on the current vehicle state information and the historical vehicle state information to obtain a pre-integration result.

[0081] S203. Perform feature extraction on the current image and the historical image through the first feature extraction module in the image matching model to obtain a first feature map pair and a third feature map pair.

[0082] Among them, the first feature extraction module uses ResNet50 (Residual Network 50) as the basic network, cancels the first pooling operation in ResNet50, and only retains the convolution parameters of the first three stages of ResNet50. In other words, the first feature extraction module includes three convolution layers, namely the first convolution layer, the second convolution layer, and the third convolution layer.

[0083] It should be noted that the first feature map pair includes the first feature map corresponding to the current image and the first feature map corresponding to the historical image, and the third feature map pair includes the third feature map corresponding to the current image and the third feature map corresponding to the historical image.

[0084] Specifically, feature extraction can be performed on the current image and the historical image through the first convolution layer in the first feature extraction module in the image matching model to obtain a first feature map pair; then, feature extraction can be performed on the first feature map pair through the second convolution layer in the first feature extraction module in the image matching model to obtain a second feature map pair; wherein the second feature map pair includes a second feature map corresponding to the current image and a second feature map corresponding to the historical image; then, feature extraction can be performed on the second feature map pair through the third convolution layer in the first feature extraction module in the image matching model to obtain a third feature map pair.

[0085] It should be noted that before the current image and the historical image are input into the image matching model, the image sizes of the current image and the historical image need to be unified to the input image size required by the image matching model. It should also be noted that the image size of each first feature image in the first feature image pair is The input image size is, and the image size of each second feature map in the second feature map pair is The input image size is, and the image size of each third feature map in the third feature map pair is The input image size.

[0086] S204. Perform feature extraction on the third feature map pair through the second feature extraction module in the image matching model to obtain a related feature map pair.

[0087] Among them, the second feature extraction module is based on the improved Transformer model, including self-attention layer, cross-attention layer and upsampling layer.

[0088] Specifically, internal correlation features can be extracted from each third feature map in the third feature map pair through the self-attention layer in the second feature extraction module in the image matching model to obtain a first internal correlation feature map and a second internal correlation feature map; then, feature extraction is performed on the first internal correlation feature map and the second internal correlation feature map through the cross attention layer in the second feature extraction module in the image matching model to obtain a first cross feature map and a second cross feature map; then, the first cross feature map and the second cross feature map are upsampled respectively through the upsampling layer in the second feature extraction module in the image matching model to obtain a correlation feature map pair; wherein the correlation feature map pair includes a correlation feature map corresponding to the first cross feature map and a correlation feature map corresponding to the second cross feature map.

[0089] S205 , performing matching degree calculation on the relevant feature graph pairs through the matching degree calculation module in the image matching model to obtain a feature matching degree matrix.

[0090] Among them, the matching degree calculation module includes a cross attention layer and a feature fusion layer.

[0091] Specifically, the cross-attention layer in the matching degree calculation module in the image matching model can be used to perform feature extraction on each related feature map in the related feature map pair to obtain the extracted related feature map pair; then, the feature fusion layer in the matching degree calculation module in the image matching model can be used to perform matching degree calculation on the extracted related feature map pairs to obtain a feature matching degree matrix.

[0092] S206: Process the first feature map pair and the feature matching matrix through a matching degree optimization module in the image matching model to obtain a matching relationship set between the current image and the historical image.

[0093] Among them, the matching optimization module includes feature clipping layer, cross attention layer and feature fusion layer.

[0094] Specifically, the feature points in the feature matching matrix can be mapped to each first feature map in the first feature map pair through four times upsampling, and then each first feature map is feature cropped through the feature cropping layer in the matching optimization module in the image matching model to obtain the cropped first feature map; then, the feature of the cropped first feature map is extracted through the cross attention layer in the matching optimization module in the image matching model to obtain the optimized first feature map; further, the matching degree of the optimized first feature map is calculated through the feature fusion layer in the matching optimization module in the image matching model, so as to obtain the matching relationship set between the current image and the historical image, which is recorded as Among them, M represents the total number of matching feature points in the matching relationship set, Represents the first matching feature point in the matching relationship set. Indicates the position of the first matching feature point in the historical image, Indicates the position of the first matching feature point in the current image. Correspondingly, Represents the second matching feature point in the matching relationship set. Indicates the position of the second matching feature point in the historical image, Indicates the position of the second matching feature point in the current image; represents the Mth matching feature point in the matching relationship set, represents the position of the Mth matching feature point in the historical image, Indicates the position of the Mth matching feature point in the current image.

[0095] S207, determining a fusion constraint function according to the current vehicle state information, the historical vehicle state information, the pre-integration result, and the feature point position information of each matching feature point in the matching relationship set.

[0096] S208. Positioning the target vehicle according to the fusion constraint function.

[0097] The technical solution of the embodiment of the present invention is as follows: by obtaining the current vehicle state information and the current image of the target vehicle at the current moment, as well as the historical vehicle state information and the historical image of the target vehicle at the previous moment of the current moment; performing IMU pre-integration processing on the current vehicle state information and the historical vehicle state information to obtain a pre-integration result; performing feature extraction on the current image and the historical image by a first feature extraction module in an image matching model to obtain a first feature map pair and a third feature map pair; performing feature extraction on the third feature map pair by a second feature extraction module in the image matching model to obtain a related feature map pair; performing matching degree calculation on the related feature map pair by a matching degree calculation module in the image matching model to obtain a feature matching degree matrix; processing the first feature map pair and the feature matching degree matrix by a matching degree optimization module in the image matching model to obtain a matching relationship set between the current image and the historical image; determining a fusion constraint function according to the current vehicle state information, the historical vehicle state information, the pre-integration result and the feature point position information of each matching feature point in the matching relationship set; and locating the target vehicle according to the fusion constraint function. The above technical solution processes the current image and historical images through an image matching model to obtain a matching relationship set between the current image and historical images, effectively solving the interference of external factors such as lighting on the image, and improving the accuracy of matching two adjacent frames of images in complex scenes, thereby improving the accuracy of the matching relationship set.

[0098] Embodiment 3

[0099] Figure 3 This is a schematic diagram of the structure of a vehicle positioning device provided in the third embodiment of the present invention. This embodiment can be applied to the situation of accurately positioning an autonomous driving vehicle in a complex scene. The device can be implemented in the form of hardware and / or software and can be configured in an electronic device. Figure 3 As shown, the device comprises:

[0100] The data acquisition module 301 is used to acquire the current vehicle state information and current image of the target vehicle at the current moment, and the historical vehicle state information and historical image of the target vehicle at the previous moment at the current moment;

[0101] A pre-integration result determination module 302 is used to perform IMU pre-integration processing on the current vehicle state information and the historical vehicle state information to obtain a pre-integration result;

[0102] A matching relationship set determination module 303 is used to process the current image and the historical image through an image matching model to obtain a matching relationship set between the current image and the historical image;

[0103] A fusion constraint function determination module 304 is used to determine a fusion constraint function according to current vehicle state information, historical vehicle state information, pre-integration results, and feature point position information of each matching feature point in a matching relationship set;

[0104] The vehicle positioning module 305 is used to position the target vehicle according to the fusion constraint function.

[0105] The technical solution of the embodiment of the present invention is to obtain the current vehicle status information and the current image of the target vehicle at the current moment, and the historical vehicle status information and the historical image of the target vehicle at the previous moment of the current moment; perform IMU pre-integration processing on the current vehicle status information and the historical vehicle status information to obtain a pre-integration result; process the current image and the historical image through an image matching model to obtain a matching relationship set between the current image and the historical image; determine a fusion constraint function based on the current vehicle status information, the historical vehicle status information, the pre-integration result and the feature point position information of each matching feature point in the matching relationship set; and locate the target vehicle based on the fusion constraint function. The above technical scheme obtains the pre-integration result by performing IMU pre-integration processing on the current vehicle status information and the historical vehicle status information, thereby reducing the influence of noise and error on the vehicle position and vehicle posture estimation, avoiding frequent calculations due to re-integration, and reducing the amount of calculation; the current image and the historical image are processed by the image matching model to obtain the matching relationship set between the current image and the historical image, effectively solving the interference of external factors such as lighting on the image, and improving the accuracy of matching two adjacent frames of images in complex scenes; the target vehicle is positioned by fusing the constraint function, making full use of the data from the IMU, image sensor and GPS sensor, improving the accuracy of vehicle positioning, and there is no need to obtain data with the help of a lidar sensor, thereby reducing costs.

[0106] Optionally, the current vehicle state information includes the current vehicle position and the current vehicle posture; the historical vehicle state information includes the historical vehicle position, the historical vehicle speed and the historical vehicle posture; the pre-integration result includes the position change amount and the posture change amount;

[0107] The pre-integration result determination module 302 is specifically used for:

[0108] Perform IMU pre-integration processing on the current vehicle position, historical vehicle position, historical vehicle speed and historical vehicle posture to obtain the position change;

[0109] Perform IMU pre-integration processing on the historical vehicle posture and the current vehicle posture to obtain the posture change.

[0110] Optionally, the image matching model includes a first feature extraction module, a second feature extraction module, a matching degree calculation module and a matching degree optimization module; wherein the first feature extraction module is connected to the second feature extraction module, the second feature extraction module is connected to the matching degree calculation module; the matching degree calculation module is connected to the matching degree optimization module;

[0111] The matching relationship set determination module 303 is specifically used for:

[0112] Extract features from the current image and the historical image using a first feature extraction module in the image matching model to obtain a first feature map pair and a third feature map pair;

[0113] Extract features from the third feature map pair using a second feature extraction module in the image matching model to obtain a related feature map pair;

[0114] The matching degree calculation module in the image matching model is used to calculate the matching degree of the relevant feature graph pairs to obtain a feature matching degree matrix;

[0115] The first feature map pair and the feature matching matrix are processed by a matching degree optimization module in the image matching model to obtain a matching relationship set between the current image and the historical image.

[0116] Optionally, the current vehicle status information also includes the current GPS observation position and the current GPS observation attitude;

[0117] The fusion constraint function determination module 304 is specifically used for:

[0118] Determine the vehicle position constraint based on the current vehicle position, historical vehicle position, historical vehicle posture, historical vehicle speed, position change, and the time interval between the current moment and the previous moment;

[0119] Determine vehicle posture constraints based on current vehicle posture, historical vehicle posture and posture change;

[0120] Determine the GPS error based on the current vehicle position, the current vehicle attitude, the current GPS observed position and the current GPS observed attitude;

[0121] Based on a preset camera projection model, a reprojection error is determined according to the current vehicle position, the current vehicle posture and the feature point position information of each matching feature point in the matching relationship set;

[0122] The fusion constraint function is determined according to the vehicle position constraint, vehicle attitude constraint, GPS error and reprojection error.

[0123] The vehicle positioning device provided in the embodiment of the present invention can execute the vehicle positioning method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing each vehicle positioning method.

[0124] According to an embodiment of the present invention, the present invention also provides an electronic device, a readable storage medium and a computer program product.

[0125] Embodiment 4

[0126] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0127] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, ROM 12 and RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0128] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0129] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a vehicle positioning method.

[0130] In some embodiments, the vehicle positioning method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the vehicle positioning method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the vehicle positioning method in any other appropriate manner (e.g., by means of firmware).

[0131] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0132] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0133] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0134] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0135] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0136] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0137] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0138] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A vehicle positioning method, characterized in that: include: Acquire the current vehicle state information and current image of the target vehicle at the current moment, as well as the historical vehicle state information and historical image of the target vehicle at the previous moment at the current moment; Performing IMU pre-integration processing on the current vehicle state information and the historical vehicle state information to obtain a pre-integration result; Processing the current image and the historical images through an image matching model to obtain a matching relationship set between the current image and the historical images; Determining a fusion constraint function according to the current vehicle state information, the historical vehicle state information, the pre-integration result, and the feature point position information of each matching feature point in the matching relationship set; The target vehicle is positioned according to the fusion constraint function.

2. The method according to claim 1, characterized in that The current vehicle state information includes the current vehicle position and the current vehicle posture; the historical vehicle state information includes the historical vehicle position, the historical vehicle speed and the historical vehicle posture; the pre-integration result includes the position change amount and the posture change amount; The performing IMU pre-integration processing on the current vehicle state information and the historical vehicle state information to obtain a pre-integration result includes: Performing IMU pre-integration processing on the current vehicle position, the historical vehicle position, the historical vehicle speed, and the historical vehicle posture to obtain a position change; Perform IMU pre-integration processing on the historical vehicle posture and the current vehicle posture to obtain a posture change amount.

3. The method according to claim 1, characterized in that The image matching model includes a first feature extraction module, a second feature extraction module, a matching degree calculation module and a matching degree optimization module; wherein the first feature extraction module is connected to the second feature extraction module, the second feature extraction module is connected to the matching degree calculation module; the matching degree calculation module is connected to the matching degree optimization module; The processing of the current image and the historical image by using an image matching model to obtain a matching relationship set between the current image and the historical image includes: Performing feature extraction on the current image and the historical image by using a first feature extraction module in an image matching model to obtain a first feature map pair and a third feature map pair; Extracting features from the third feature map pair using a second feature extraction module in the image matching model to obtain a related feature map pair; Performing matching degree calculation on the relevant feature graph pairs by a matching degree calculation module in the image matching model to obtain a feature matching degree matrix; The first feature map pair and the feature matching matrix are processed by a matching degree optimization module in the image matching model to obtain a matching relationship set between the current image and the historical image.

4. The method according to claim 2, characterized in that: The current vehicle state information also includes the current GPS observation position and the current GPS observation attitude; The determining of the fusion constraint function according to the current vehicle state information, the historical vehicle state information, the pre-integration result and the feature point position information of each matching feature point in the matching relationship set includes: Determine a vehicle position constraint according to the current vehicle position, the historical vehicle position, the historical vehicle posture, the historical vehicle speed, the position change, and the time interval between the current moment and the previous moment; Determining a vehicle posture constraint according to the current vehicle posture, the historical vehicle posture, and the posture change amount; Determining a GPS error based on the current vehicle position, the current vehicle attitude, the current GPS observed position, and the current GPS observed attitude; Based on a preset camera projection model, determining a reprojection error according to the current vehicle position, the current vehicle posture and feature point position information of each matching feature point in the matching relationship set; A fusion constraint function is determined according to the vehicle position constraint, the vehicle attitude constraint, the GPS error and the reprojection error.

5. A vehicle positioning device, characterized in that: include: A data acquisition module, used to acquire the current vehicle state information and current image of the target vehicle at the current moment, and the historical vehicle state information and historical image of the target vehicle at the previous moment at the current moment; A pre-integration result determination module is used to perform IMU pre-integration processing on the current vehicle state information and the historical vehicle state information to obtain a pre-integration result; A matching relationship set determination module, used to process the current image and the historical image through an image matching model to obtain a matching relationship set between the current image and the historical image; A fusion constraint function determination module, used to determine a fusion constraint function according to the current vehicle state information, the historical vehicle state information, the pre-integration result and the feature point position information of each matching feature point in the matching relationship set; The vehicle positioning module is used to position the target vehicle according to the fusion constraint function.

6. The device according to claim 5, characterized in that The current vehicle state information includes the current vehicle position and the current vehicle posture; the historical vehicle state information includes the historical vehicle position, the historical vehicle speed and the historical vehicle posture; the pre-integration result includes the position change amount and the posture change amount; The pre-integration result determination module is specifically used for: Performing IMU pre-integration processing on the current vehicle position, the historical vehicle position, the historical vehicle speed, and the historical vehicle posture to obtain a position change; Perform IMU pre-integration processing on the historical vehicle posture and the current vehicle posture to obtain a posture change amount.

7. The device according to claim 5, characterized in that The image matching model includes a first feature extraction module, a second feature extraction module, a matching degree calculation module and a matching degree optimization module; wherein the first feature extraction module is connected to the second feature extraction module, the second feature extraction module is connected to the matching degree calculation module; the matching degree calculation module is connected to the matching degree optimization module; The matching relationship set determination module is specifically used to: Performing feature extraction on the current image and the historical image by using a first feature extraction module in an image matching model to obtain a first feature map pair and a third feature map pair; Extracting features from the third feature map pair using a second feature extraction module in the image matching model to obtain a related feature map pair; Performing matching degree calculation on the relevant feature graph pairs by a matching degree calculation module in the image matching model to obtain a feature matching degree matrix; The first feature map pair and the feature matching matrix are processed by a matching degree optimization module in the image matching model to obtain a matching relationship set between the current image and the historical image.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the vehicle positioning method according to any one of claims 1 to 4.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the vehicle positioning method according to any one of claims 1 to 4 when executed.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the vehicle positioning method according to any one of claims 1 to 4 is implemented.

Images