Positioning method, device, vehicle, medium and program product

Abstract

This application is applied to the field of vehicle positioning, and provides a positioning method, device, vehicle, medium and program product. The method includes: when the vehicle-mounted computer detects a request for positioning the vehicle, the vehicle-mounted computer can identify the trajectory pose of the vehicle in a period of time by obtaining the driving distance and angle of the vehicle in real time. Moreover, the vehicle-mounted computer can also collect real-time feature information during the driving process of the vehicle. Then, the vehicle-mounted computer will determine the environmental feature information that matches the collected feature information from multiple environmental feature information of the parking lot map, and use the pose corresponding to the matching environmental feature information as the feature pose. Finally, the vehicle-mounted computer combines the trajectory pose and the feature pose to determine the pose of the vehicle. Furthermore, relatively accurate vehicle pose information can be obtained, and functions such as parking based on vehicle positioning can be completed.

Description

Technical Field

[0001] The present application relates to the field of vehicle positioning, and in particular, to a positioning method, device, vehicle, medium, and program product. Background Art

[0002] The in-vehicle unit has automatic parking functions such as automated valet parking (AVP). The in-vehicle unit needs to obtain the position and / or angle (hereinafter collectively referred to as pose) of the vehicle in the parking lot, and combine the existing parking lot map to control the vehicle to drive to the target position, for example, control the vehicle to drive to the designated parking space.

[0003] However, due to poor signal in the parking lot, for example, when the vehicle is in the underground parking lot, the in-vehicle unit cannot obtain accurate positioning information in the parking lot. Due to the inability to accurately obtain the real-time positioning information of the vehicle, the in-vehicle unit may drive the vehicle to the wrong position during the automatic parking process, and even cause a parking accident. Summary of the Invention

[0004] In view of this, the present application provides a positioning method, device, vehicle, medium, and program product.

[0005] In a first aspect, a positioning method is provided, which is applied to an in-vehicle unit device in a vehicle. The method includes: detecting a positioning request for requesting to obtain the pose information of the vehicle at a target moment; obtaining a plurality of trajectory poses of the vehicle within a first time period, and environmental feature information collected at each trajectory pose; determining, from an environmental feature map, feature poses corresponding to the environmental feature information of each trajectory pose, where the environmental feature map includes environmental feature information corresponding to different poses; determining, according to the plurality of trajectory poses and the plurality of feature poses, multiple sets of pose pairs for calculating a target trajectory, where the first pose pair in the multiple sets of pose pairs includes a first trajectory pose and a first feature pose collected at a first moment; fitting, according to the multiple sets of pose pairs, to obtain the target trajectory of the vehicle within the first time period; and obtaining the pose information of the vehicle at the target moment according to the target trajectory, the trajectory pose of the vehicle at the target moment, and the feature pose at the target moment.

[0006] In the above solution, when the in-vehicle computer detects a request to locate the vehicle, for example, when it detects a request for automatic parking, the in-vehicle computer can obtain the trajectory poses corresponding to each moment in the driving trajectory of the vehicle within a certain period of time. Moreover, the in-vehicle computer can also collect real-time feature information during the driving process of the vehicle to obtain the feature information collected at each moment. Then, the in-vehicle computer will determine, from multiple environmental feature information in the parking lot map, the environmental feature information that matches the collected feature information, and use the pose corresponding to the matched environmental feature information as the feature pose. Finally, the in-vehicle computer will combine the trajectory pose and the feature pose to determine the pose of the vehicle, obtaining relatively accurate vehicle pose information, and then can complete functions such as parking based on the positioning information of the vehicle. For a solution that only uses feature matching for positioning, there are many repeated scenarios in the parking lot, and the positioning based on feature matching is inaccurate. However, this solution combines the trajectory pose and the feature pose obtained based on feature matching, and can obtain relatively accurate positioning information.

[0007] Combined with the first aspect, in some implementation manners, the method further includes: obtaining the pose increment of the vehicle at each preset time interval through the chassis sensor and / or inertial measurement unit of the vehicle within the first time period; determining multiple trajectory poses of the vehicle within the first time period according to the multiple pose increments of the vehicle within the first time period.

[0008] In the above solution, the in-vehicle computer can obtain the driving trajectory of the vehicle through the chassis sensor and / or inertial measurement unit, and then obtain the trajectory pose. For scenarios with poor signals such as indoor parking lots or underground parking lots, the trajectory pose obtained through the chassis sensor and / or inertial measurement unit will be more accurate.

[0009] Combined with the first aspect, in some implementation manners, the method further includes: determining multiple candidate feature information whose similarity with the first environmental feature information corresponding to the first trajectory pose among each trajectory pose is greater than the first threshold from the multiple environmental feature information included in the environmental feature map; obtaining the neighborhood feature information collected at adjacent time intervals with each candidate feature information from the environmental feature map; selecting, from the multiple candidate feature information, the target environmental feature information whose similarity with the first environmental feature information of the corresponding neighborhood feature information is greater than the second threshold according to the similarity between the first environmental feature information and each neighborhood feature information; using the pose corresponding to the target environmental feature information as the first feature pose corresponding to the first environmental feature information according to the environmental feature map.

[0010] In the above solution, the in-vehicle computer matches the first environmental feature information collected with the feature information in the environmental feature map. After obtaining multiple candidate feature information with relatively high similarity, in order to further screen out the optimal match, it will also match the neighborhood feature information of the candidate feature information with the first environmental feature information collected by the in-vehicle computer. The candidate feature information corresponding to the neighborhood feature information with relatively high similarity to the first environmental feature information is used as the feature information corresponding to the first environmental feature information. Furthermore, the pose where the candidate feature information is collected can be used as the pose corresponding to the first environmental feature information. During the matching process, not only individual feature information is considered, but also the neighborhood information of the feature information, which can improve the accuracy of the feature pose. This method can reduce the positioning error caused by environmental changes or sensor errors.

[0011] Combined with the first aspect, in some implementation manners, the method further includes: determining a map trajectory corresponding to a second feature pose and a third feature pose among multiple feature poses, where the map trajectory is the driving trajectory between the environmental feature information collected when the second feature pose is collected and the environmental feature information collected when the third feature pose is collected during the construction of the environmental feature map; determining a first relative pose change according to the second feature pose and the third feature pose, and determining a second relative pose change according to the map trajectory; when the ratio corresponding to the first relative pose change and the second relative pose change is less than a preset ratio, determining a second pose pair including the second feature pose and the second trajectory pose corresponding to the second feature pose, and determining a third pose pair including the third feature pose and the third trajectory pose corresponding to the third feature pose.

[0012] In the above solution, the pose pairs participating in the calculation of the target trajectory are also screened. If the pose change between adjacent feature poses has a large difference from the path during the corresponding construction of the environmental feature map, it can be determined that there is a large error in the corresponding feature pose. When the ratio of the pose change between adjacent feature poses to the pose change during the corresponding construction of the environmental feature map is small, it can be determined that the corresponding feature pose and trajectory pose can be used to calculate the target trajectory. This method helps to exclude incorrect pose data, helps to generate a more accurate calculation formula for the target trajectory, and improves the accuracy of positioning.

[0013] Combined with the first aspect, in some implementation manners, the second threshold is determined by the similarity between two environmental feature information belonging to the first type of environmental feature information in the environmental feature map and the similarity between two environmental feature information belonging to the second type of environmental feature information, where the difference in the poses of the two environmental feature information belonging to the first type of environmental feature information collected is greater than the pose difference, and the difference in the poses of the two environmental feature information belonging to the second type of environmental feature information collected is less than or equal to the pose difference.

[0014] In the above solution, the similarity threshold can be set according to the actual situation of the constructed environmental feature map. The environmental feature information with a relatively large pose during the construction of the environmental feature map is classified as the first type of environmental feature information, which is used to represent the environmental feature information collected in different scenarios. The environmental feature information with a relatively large pose during the construction of the environmental feature map is classified as the second type of environmental feature information, which is used to represent the environmental feature information collected in similar scenarios. By calculating the similarity between the environmental feature information in the first type of environmental feature information and the second type of environmental feature information, the similarity threshold can be determined, so that the determined similarity threshold conforms to the actual situation of the environmental feature map.

[0015] In combination with the first aspect, in some implementation manners, the method further includes: updating the environmental feature map according to the pose change trajectory within the first time period and the environmental feature information corresponding to each pose in the pose change trajectory.

[0016] In combination with the first aspect, in some implementation manners, the method further includes: adding the second environmental feature information collected at the second moment in the first time period and the trajectory pose for collecting the second environmental feature information to the environmental feature map; and, corresponding to the matching between the third environmental feature information collected at the third moment in the first time period and the fourth environmental feature information in the environmental feature map, replacing the fourth environmental feature information with the third environmental feature information.

[0017] In a second aspect, the present application provides a vehicle-mounted device, including a processor and a memory. The memory is used to store instructions executed by one or more processors of the vehicle-mounted device, and the processor is one of the processors of the vehicle-mounted device and is used to execute the instructions. When the processor executes the instructions, the method described in the first aspect and various implementation manners of the first aspect is executed.

[0018] In a third aspect, the present application provides a vehicle, including the vehicle-mounted device described in the second aspect.

[0019] In a fourth aspect, the present application provides a computer-readable storage medium, in which instructions are stored. When the instructions run on a positioning device, the method described in the first aspect and various implementation manners of the first aspect is executed.

[0020] In a fifth aspect, the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computing device, the computing device executes the method described in the first aspect and various implementation manners of the first aspect.

[0021] The beneficial effects of the above second aspect to fifth aspect can refer to the beneficial effects of the first aspect described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments.

[0023] Figure 1 It is a schematic diagram of a constructed parking lot map provided by an embodiment of the present application;

[0024] Figure 2 It is a schematic diagram of identifying image features provided by an embodiment of the present application;

[0025] Figure 3 It is a schematic diagram of positioning through feature comparison provided by an embodiment of the present application;

[0026] Figure 4 It is a schematic diagram of the structure of a positioning system provided by an embodiment of the present application;

[0027] Figure 5 It is a schematic diagram of determining neighborhood feature information provided by an embodiment of the present application;

[0028] Figure 6 It is a schematic diagram of determining a pose pair provided by an embodiment of the present application;

[0029] Figure 7 It is a schematic diagram of determining positioning considering odometer constraints and VPR constraints provided by an embodiment of the present application;

[0030] Figure 8 It is a schematic diagram of updating a feature map provided by an embodiment of the present application;

[0031] Figure 9 It is a schematic diagram of the structure of a map construction system provided by an embodiment of the present application;

[0032] Figure 10 It is a schematic diagram of determining a similarity threshold provided by an embodiment of the present application;

[0033] Figure 11 It is a schematic diagram of the flowchart of a positioning method provided by an embodiment of the present application;

[0034] Figure 12 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0035] The illustrative embodiments of the present application include, but are not limited to, positioning methods, devices, vehicles, media, and program products.

[0036] As described above, the in-vehicle device included in the vehicle may have an automatic parking function, which can control the vehicle to drive to a designated parking space according to the existing parking lot map and the pose of the current vehicle.

[0037] In some examples, the in-vehicle unit can be positioned through a global navigation satellite system (GNSS) and / or in combination with real-time kinematic (RTK) technology.

[0038] However, in scenarios such as underground parking lots or indoor parking lots, the GNSS signal is weak, and the in-vehicle unit cannot obtain an accurate position of the vehicle, resulting in errors in vehicle positioning. When the in-vehicle unit performs real-time positioning of the vehicle based on GNSS and / or RTK and conducts automatic parking, there will be a problem of inaccurate positioning due to poor GNSS signal, which may further lead to parking failure.

[0039] In other examples, the in-vehicle unit can also determine the position of the vehicle by collecting the feature information around the current environment and matching it with a pre-constructed parking lot feature map. The feature information is obtained by processing the original sensor data through a visual place recognition (VPR) deep neural network model. The original sensor data can be RGB images collected by an in-vehicle camera or point cloud data collected by a lidar, etc.

[0040] Among them, the construction of the parking lot map can be completed by the in-vehicle unit of the current vehicle or other devices. Taking the in-vehicle unit as an example, the in-vehicle unit can be positioned through a global navigation satellite system (GNSS) and / or in combination with real-time kinematic (RTK) technology. Moreover, at different poses, the in-vehicle unit will also collect environmental features. For example, the in-vehicle unit can collect environmental features through sensors such as cameras and / or lidars and identify environmental features such as road information (e.g., lane lines, lane driving directions, etc.), parking space information (e.g., parking space numbers), and obstacle information (e.g., pillars, walls, guardrails, etc.). The in-vehicle unit can combine the collected environmental features with the pose information corresponding to the environmental features to construct a parking lot map as shown in Figure 1 the following.

[0041] In addition, the in-vehicle unit can also extract feature points from the environmental features collected when constructing the parking lot map and identify edges, corners, etc. in the environment. For example, the in-vehicle unit can process the images collected by the in-vehicle camera into grayscale images and determine the areas where the gray values of the pixel points in the grayscale images change significantly as edges, and determine the intersection points of two edges as corners. For example, Figure 2 the left side shows an environmental feature map collected by an in-vehicle unit. By extracting the features in the environmental feature map, the corners included in the environmental feature map can be identified.Figure 2 On the right side is Figure 2 an enlarged schematic diagram of the pixel points in the area shown by the white box on the left side, Figure 2 and in the schematic diagram shown on the right side, the area where the pixel point color is found to change is the corner point, and the vehicle-mounted computer can identify information such as the edges and corner points existing in the environmental feature map.

[0042] Furthermore, when the vehicle-mounted computer performs automatic parking, it can determine the pose of the vehicle by collecting the environmental feature points near the current pose of the vehicle (hereinafter referred to as the feature points during positioning). For example, as Figure 3 shown, the vehicle-mounted computer can compare the feature points during positioning with the environmental feature points in the parking lot map (hereinafter referred to as the feature points during map building), and determine the feature points during map building that match the feature points during positioning (for example, the feature points during map building with the highest similarity to the feature points during positioning) among the multiple feature points during map building included in the parking lot map, and determine the pose information corresponding to the matching feature points during map building as the current pose of the vehicle.

[0043] However, the above requires comparing the feature points during positioning with all the feature points during map building, which has a large amount of calculation and consumes a lot of computing resources. Moreover, in the actual environment of the parking lot, due to the high repetition of features such as parking spaces and lane lines, errors may occur during the feature comparison process. In addition, for underground parking lots or indoor parking lots, the light is weak, and the environmental features collected during map building and / or positioning may not be accurate enough, and it is also easy to collect moving objects in the environment (such as moving vehicles or moving people, etc.). And there may also be a lot of environmental features such as textureless walls and corridors that lack sufficient texture information, resulting in fewer available features being extracted. Furthermore, due to reasons such as inaccurate environmental features collected in the parking lot or fewer available features, the result of the feature comparison is not accurate enough. Since the real-time positioning information of the vehicle cannot be accurately obtained, the vehicle-mounted computer cannot accurately perform automatic parking.

[0044] Furthermore, in order to solve the problem that the vehicle's positioning in the parking lot is inaccurate, resulting in the vehicle-mounted computer being unable to accurately park the vehicle, the present application provides a positioning method. When the vehicle-mounted computer detects a request for positioning the vehicle, for example, when it detects a request for automatic parking, the vehicle-mounted computer can obtain the driving distance and angle of the vehicle in a period of time in real time, and then identify the driving trajectory of the vehicle during this period. Moreover, the pose corresponding to each moment in the driving trajectory of the vehicle during this period can also be referred to as the trajectory pose.

[0045] In addition, the vehicle computer can also collect real-time feature information during the vehicle's driving process and obtain the feature information collected at each moment. Then, the vehicle computer will determine the environmental feature information that matches the collected feature information from the multiple environmental feature information of the parking lot map, and use the posture corresponding to the matching environmental feature information as the characteristic posture. Finally, the vehicle computer will combine the trajectory posture and the characteristic posture to determine the vehicle's posture.

[0046] Among them, the vehicle computer can determine the vehicle posture by combining the trajectory posture and the characteristic posture in the following way: the vehicle computer can use the optimization algorithm to combine the mileage data (i.e., multiple trajectory postures) and the visual matching data (i.e., multiple characteristic postures), and use the trajectory posture and characteristic posture collected at the same time as a set of posture pairs, and then use multiple sets of posture pairs to correct and optimize the vehicle posture estimation algorithm, that is, fit the target trajectory function used to calculate the actual posture of the vehicle. The target trajectory function takes into account the uncertainty of the trajectory position and the characteristic position. The target trajectory function can be specifically obtained by calculating the covariance matrix of the errors of the trajectory position and the characteristic position, and then iteratively solving to obtain the minimized posture error, and then obtaining the final target trajectory function. Then, by bringing the trajectory position and characteristic position at the target moment to be solved into the target trajectory function, the actual posture of the vehicle at the target moment can be obtained.

[0047] In some embodiments, the vehicle computer can obtain information such as the vehicle's driving posture, speed, and displacement through vehicle sensors, such as chassis wheel speed sensors and / or inertial measurement units (IMUs), and then the vehicle computer can determine the vehicle's driving trajectory.

[0048] The method provided in the embodiment of the present application can be applied to vehicle equipment of vehicles, etc. Among them, the above-mentioned vehicles may include but are not limited to road vehicles (such as vehicles), water vehicles (such as ships), and air vehicles (such as airplanes). The above-mentioned vehicles can be vehicles in a broad sense, for example, they can be vehicles (such as commercial vehicles, passenger cars, motorcycles, flying cars, trains, etc.), industrial vehicles (such as forklifts, trailers, tractors, etc.), engineering vehicles (such as excavators, bulldozers, cranes, etc.), agricultural equipment (such as mowers, harvesters, etc.), amusement equipment, toy vehicles, etc.

[0049] It should be understood that the above-mentioned method provided in the embodiment of the present application is applied to the parking lot navigation scenario as an example. In actual application, the method provided in the embodiment of the present application can also be applied to other indoor navigation scenarios, such as shopping malls, factories, etc. Corresponding to different application scenarios, the environmental feature map can also be an environmental feature map of a shopping mall or an environmental feature map of a factory, etc.

[0050] For ease of explanation below, taking the parking lot navigation scenario as an example, the positioning method provided by the embodiments of the present application will be illustrated by way of example.

[0051] Figure 4 FIG. shows a schematic structural diagram of a positioning system. The positioning system includes a wheel speed odometer module, a visual place recognition (VPR) deep neural network module, a VPR feature vector module, a candidate feature retrieval module, a neighborhood feature similarity screening module, a multi-frame pose consistency verification module, a combined positioning pose optimization module, and a feature update module.

[0052] It should be understood that the positioning system may further include more or fewer components than those shown in the figure, or combine certain components, or split certain components, or have different component arrangements. Figure 4 The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0053] The wheel speed odometer module is used to calculate the odometer incremental displacement, and then obtain the driving trajectory of the vehicle. Specifically, when the wheel speed odometer module detects that vehicle positioning is required, it can obtain multiple trajectory poses of the vehicle within a certain time period (for example, the first time period). Specifically, the wheel speed odometer module can be used to obtain information such as the driving attitude, speed, and displacement of the vehicle based on the chassis wheel speed sensor and / or IMU. Then, the wheel speed odometer module can calculate the position and attitude of the vehicle based on the driving attitude, speed, and displacement of the vehicle through incremental calculations such as speed integration and angle integration.

[0054] Among them, the calculation formula for speed integration can refer to the following formula (1), and the calculation formula for angle integration can refer to the following formula (2).

[0055] (1);

[0056] (2);

[0057] Among them, represents the incremental displacement of the vehicle along the driving direction within a certain time interval. represents the incremental displacement of the left wheel of the vehicle within a certain time interval. represents the incremental displacement of the right wheel of the vehicle within a certain time interval.

[0058] represents the angular velocity measured by the gyroscope, that is, the rotation rate of the vehicle around the axis perpendicular to the ground (usually the longitudinal axis of the vehicle). represents the time interval, that is, the time length considered for calculating the displacement and attitude changes. This time interval can be fixed or dynamically adjusted according to the actual situation. Indicates the incremental change in vehicle attitude within a time interval , that is, the rotation angle of the vehicle.

[0059] Moreover, the wheel speed odometer module can also store the vehicle pose and the corresponding timestamp every preset time T1 or every preset distance L1 to obtain the vehicle driving trajectory.

[0060] The VPR feature vector module, candidate feature retrieval module, neighborhood feature similarity screening module, and multi-frame pose consistency verification module are used to obtain multiple feature poses of the vehicle within the first time period when it is detected that vehicle positioning is required.

[0061] The VPR deep neural network module is used to send sensor data such as the RGB image I collected by the on-vehicle camera during the first time period or the point cloud data collected by the on-vehicle lidar into the trained VPR deep neural network Θ. For the following description for convenience, the case where the RGB image I is input into the VPR deep neural network Θ is taken as an example.

[0062] The VPR deep neural network Θ can extract key semantic feature information from the collected images and aggregate the extracted feature information into a VPR feature vector F with a fixed dimension. For example, F can be an n-dimensional vector, and each component is a real number, that is, it belongs to the real number field . The VPR deep neural network module can send the images at each moment into the VPR deep neural network Θ in sequence to obtain a set of VPR feature sequences , where , to represent the feature vectors corresponding to the images at different moments. Specifically, reference can be made to the following formulas (3) and (4).

[0063] (3);

[0064] (4);

[0065] In some embodiments, the VPR deep neural network Θ may include a feature extraction backbone network and a feature aggregation network using a cross-attention mechanism. Among them, the feature extraction backbone network is used to extract key semantic feature information from the input image, such as edges, textures, shapes, and colors. The feature aggregation network is used to convert the extracted semantic feature information into a dedicated feature for visual scene recognition to obtain the VPR feature vector F.

[0066] The candidate feature retrieval module is used to obtain the K candidate feature information with the highest similarity in the parking lot map according to the VPR feature vector collected at a certain pose of the vehicle during the first time period . Among them, the candidate feature retrieval module can calculate the similarity with each feature information in the parking lot map, and then sort each similarity to obtain the top K candidate feature information with the highest similarity . Alternatively, the feature information with a similarity greater than the first threshold can also be used as candidate feature information.

[0067] The neighborhood feature similarity screening module is used to screen the K candidate features according to the neighborhood feature information of the K candidate feature information to obtain the optimal matching feature information of the K candidate feature information in this pose. .

[0068] For example, based on the collected feature image, the VPR deep neural network module outputs the corresponding feature vector After that, the candidate feature retrieval module can determine multiple candidate feature information with the highest similarity to the feature vector . Then for each candidate feature information , at least one characteristic information (hereinafter referred to as neighborhood feature information ) collected from the environmental feature map for this candidate feature information in adjacent frames will also be determined. And the similarity between the feature vector and the neighborhood feature information will be calculated. If the similarity between the feature vector and the neighborhood feature information is low, it is determined that the candidate feature information does not match the feature vector . Furthermore, the candidate feature information with a higher matching degree between the neighborhood feature information and the feature vector can be used as the optimal matching feature information of the feature vector . .

[0069] For example, as Figure 5 shown, for the candidate feature information of the i-th frame included in , the preset time T2 before and after the candidate feature information can be used as the neighborhood interval. Furthermore, the feature information collected in the VPR trajectory map at the preset time T2 before and after the candidate feature is the neighborhood feature information of the candidate feature , such as the candidate feature of the i-th frame The neighborhood feature information may include the feature information of the j-th frame and the feature information of the l-th frame .

[0070] For another example, the similarity between the feature information and the neighborhood feature information can be the Euler distance. For the feature information with the Euler distances from multiple neighborhood feature information all less than a preset distance, the candidate feature information can be determined as the optimal matching feature information . .

[0071] Then, by calculating the similarity between each neighborhood feature information and , the candidate features with the similarity between the neighborhood feature information and lower than the similarity threshold H (as an example of the second threshold) are determined as false recalls. Or, sort the similarities between the neighborhood feature information and , and use the candidate feature with the highest similarity between the neighborhood feature information and as the optimal matching feature information . Among them, the calculation method of the similarity threshold H can also refer to the description of the similarity confidence threshold calculation module in the following Figure 9 .

[0072] The multi-frame pose consistency checking module is used to perform a sliding window match on the poses (hereinafter referred to as feature poses) corresponding to the multiple feature information obtained by the candidate feature retrieval module and the neighborhood feature similarity screening module with the parking lot map, and determine the feature poses that can be used to calculate the target trajectory from the multiple feature poses. Then, use the determined feature poses and the corresponding trajectory poses as a group of pose pairs for the subsequent combined positioning pose optimization module to generate the target trajectory function. For example, Figure 6 the dashed box in

[0073] represents the sliding window. Figure 6 For example, the parking lot map is constructed from the feature information collected by the in-vehicle computer during vehicle driving. The parking lot map can also be called the VPR trajectory map, and thus there is a driving trajectory (hereinafter referred to as the map trajectory) when constructing the parking lot map. The map trajectory includes multiple map poses, and each map pose has the feature information collected under that map pose. As Figure 6As shown, if the environmental feature information in the parking lot map matches the feature information of the feature pose, then the in-vehicle computer can determine that the map pose corresponding to the collected environmental feature information matches the feature pose. The in-vehicle computer will also determine the map pose corresponding to each feature pose among the multiple feature poses screened by the neighborhood feature similarity screening module. Among them, the trajectory formed by the multiple feature poses can also be called a positioning trajectory.

[0074] Then, the in-vehicle computer will calculate the relative pose transformation between adjacent feature poses , and calculate the relative pose transformation between the map poses corresponding to this group of adjacent feature poses . And, by calculating and the ratio , if this is less than the ratio threshold, it is determined that this group of feature poses and the trajectory poses corresponding to the feature poses can be used to calculate the target trajectory. Among them, the trajectory pose corresponding to the feature pose refers to the pose obtained by the odometer module at the same moment as this feature pose. For example, Figure 6 the and between the adjacent trajectory poses and feature poses in the dashed box in

[0075] are both less than the ratio threshold, and thus can both be used to calculate the target trajectory. , and the relative pose transformation between the corresponding map poses Meet the condition that is less than the threshold, and then these four feature poses and the corresponding trajectory poses within the window can be used as pose pairs.

[0076] The combined positioning pose optimization module is used to fit a target trajectory function for calculating the vehicle pose according to multiple groups of pose pairs determined by the multi-frame pose consistency verification module. Among them, the target trajectory function takes into account the errors between the trajectory pose and the feature pose. As Figure 7 shown, p1, p2, and p3 can be used to represent the vehicle pose, odo is used to represent the trajectory pose constraint (also called the key frame odometer pose constraint), and vpr is used to represent the feature pose constraint (also called the VPR observation constraint).

[0077] Optionally, the target trajectory function can be obtained by the Gauss-Newton method or the Levenberg-Marquardt algorithm based on multiple sets of pose pairs. For example, the target trajectory function consists of the covariance matrix of the trajectory pose noise, the covariance matrix of the feature pose noise, the inter-frame pose residuals of the trajectory poses, and the inter-frame pose residuals of the feature poses in the multiple sets of pose pairs obtained from the multiple sets of pose pairs. Finally, the target trajectory function obtained through the combined positioning pose optimization module is substituted with the trajectory pose and feature pose collected at the target time, and then the actual pose of the vehicle at the target time can be solved.

[0078] The feature update module is used to update the historical parking lot map according to the current positioning result.

[0079] Among them, as Figure 8 shown, updating the parking lot map includes that if the actual positioning of the vehicle can be obtained and multiple VPR features are collected at the positioning location, the feature information of the positioning in the historical parking lot map (also known as the original trajectory of map building) can be replaced with the latest collected VPR features.

[0080] If a map that was not covered by the historical parking lot is collected according to the positioning and / or driving trajectory (also known as the positioning trajectory) of the current vehicle, the positioning trajectory and its feature information can be added to the parking lot map.

[0081] In some embodiments, since the maps of current navigation applications do not cover most parking lots, especially for parking lots in newly built communities or shopping malls, etc., the in-vehicle computer can also autonomously construct a parking lot map. Among them, the specific process of the in-vehicle computer constructing the parking lot map can refer to as Figure 9 shown.

[0082] Figure 9 A map building system of an in-vehicle computer is shown. The system can include a wheel speed odometer module, a VPR deep neural network module, a trajectory-VPR feature timestamp association module, a positive and negative sample association module, and a similarity confidence threshold calculation module.

[0083] Among them, the wheel speed odometer module is used to obtain the vehicle wheel speed odometer, and then calculate the vehicle odometer incremental displacement. The VPR deep neural network module is used to generate VPR feature vectors based on the VPR deep neural network for the images collected by the in-vehicle camera. Moreover, for multiple frames of images input by the in-vehicle camera, the VPR deep neural network module can also correspondingly generate multiple VPR feature vectors. The descriptions of the wheel speed odometer module and the VPR deep neural network module can also refer to the relevant descriptions in the foregoing Figure 4 and will not be elaborated here.

[0084] The trajectory-VPR feature timestamp association module is used to find the corresponding driving trajectory points for each VPR feature vector with timestamps and trajectory poses, and obtain multiple corresponding feature poses.

[0085] Moreover, since the VPR features with adjacent spatial positions have low distinctiveness, the associated VPR feature vectors can also be downsampled according to the cumulative trajectory distance. In the curved road scenarios with large curvature changes, the sampling density can also be increased to ensure the scene coverage of VPR features under large viewing angle changes.

[0086] The positive and negative sample association module is used to classify the multiple collected VPR feature vectors into positive samples and negative samples for subsequent calculation of the similarity threshold H. Among them, for two trajectory features with relatively small odometer position and attitude during map building, their corresponding feature poses can be regarded as a group of positive samples; for two trajectory features with large differences in position and attitude during map building / from different parking lots, their corresponding feature poses can be regarded as a group of negative samples. Optionally, for the case where there is a pre-constructed historical feature map (also called historical VPR map) in the current scene, the positive and negative sample association module can also obtain the data of the historical VPR map.

[0087] Moreover, the positive and negative sample association module will calculate the similarity (such as cosine similarity) between any two VPR feature vectors to obtain the similarity results of each group of samples (including positive samples and negative samples). For example, the similarity results can be represented by values in the interval [0,1]. Among them, the closer the similarity result is to 1, the higher the similarity between the two feature vectors. Since the similarity threshold H is required in the neighborhood feature similarity screening module to determine whether the neighborhood feature information is similar to the currently collected feature, due to various factors such as lighting and weather conditions and the distribution of dynamic and static targets, there are huge differences in the feature space distributions of VPR features in different parking lots, and a general confidence threshold cannot be used for judgment. Furthermore, the similarity confidence threshold calculation module can generate the similarity threshold H for the current scene according to the currently collected feature information.

[0088] The similarity confidence threshold calculation module is used to generate the similarity threshold H according to the similarity of each group of positive samples and the similarity of each group of negative samples. Among them, the similarity confidence threshold calculation module can first be used to calculate the mean and standard deviation of the similarities of all positive samples, and the mean of the similarities of all negative samples,

[0089] and then generate the preliminary thresholds for positive and negative samples. Among them, the negative sample-oriented threshold is obtained by the mean and standard deviation and the negative sample range control coefficient is obtained. The negative sample guiding threshold is based on the "upper confidence bound" of negative samples, which can ensure that most negative samples are lower than this value. The positive sample guiding threshold is obtained through the mean value and the standard deviation and the negative sample range control coefficient is obtained. The positive sample guiding threshold is based on the "lower confidence bound" of positive samples, which can ensure that most positive samples are higher than this value.

[0090] Furthermore, the similarity threshold H can be obtained through the negative sample guiding threshold, the negative sample coefficient, the positive sample guiding threshold, and the positive sample coefficient. For example, as Figure 10 shown, the similarity threshold H is equal to the product of the negative sample guiding threshold and the negative sample coefficient, plus the product of the positive sample guiding threshold and the positive sample coefficient. Moreover, the sum of the negative sample coefficient and the positive sample coefficient is 1, and the negative sample coefficient and the positive sample coefficient can also be adjusted according to the number of positive and negative samples. For example, the more the number of positive samples, the larger the positive sample coefficient.

[0091] Optionally, the above data can also be exported as a serialized binary file and stored in the local storage medium of the vehicle terminal.

[0092] Next, the positioning method provided by this application will be introduced in detail. Figure 11 According to the embodiments of this application, a schematic flowchart of the implementation process of a positioning method is provided. As mentioned above, this method is applied to the above vehicle-mounted device. As Figure 11 shown, this method specifically includes:

[0093] S1101: A positioning request is detected.

[0094] The vehicle-mounted device can detect a request from the user to obtain the pose information of the vehicle at the target moment, such as when the user turns on the autonomous navigation function.

[0095] S1102: Obtain multiple trajectory poses of the vehicle within the first time period, and the environmental feature information collected for each trajectory pose.

[0096] The vehicle-mounted device will obtain information such as the posture, speed, and displacement of the vehicle's driving according to the chassis wheel speed sensor and / or the IMU, and then obtain multiple trajectory poses of the vehicle within the first time period. The vehicle-mounted device will also collect the feature information within the first time period based on on-vehicle cameras, etc. Among them, the first time period can be before or after the target moment, and can also include a period of time at the target moment.

[0097] For the vehicle-mounted device to obtain multiple trajectory poses and environmental feature information, reference can also be made to the relevant descriptions of the aforementioned wheel speed odometer module and the VPR deep neural network module, which will not be elaborated here.

[0098] S1103: Determine the feature poses corresponding to the environmental feature information of each trajectory pose from the environmental feature map.

[0099] The in-vehicle computer can determine the environmental feature information that matches the collected feature information from multiple environmental feature information in the parking lot map, and use the pose corresponding to the matched environmental feature information as the feature pose. Specifically, reference can be made to the relevant descriptions of the aforementioned VPR feature vector module, candidate feature retrieval module, neighborhood feature similarity screening module, and multi-frame pose consistency verification module, which will not be elaborated here.

[0100] S1104: Fit the target trajectory of the vehicle within the first time period based on multiple trajectory poses and multiple feature poses.

[0101] The in-vehicle computer can use an optimization algorithm to combine multiple trajectory poses and multiple feature poses to correct and optimize the vehicle's pose estimation algorithm, that is, fit the target trajectory function for calculating the actual pose of the vehicle. Specifically, reference can also be made to the description of the aforementioned combined positioning pose optimization module, which will not be elaborated here.

[0102] S1105: Obtain the pose information of the vehicle at the target moment based on the target trajectory and the trajectory pose and feature pose of the vehicle at the target moment.

[0103] The in-vehicle computer substitutes the obtained target trajectory function with the trajectory pose and feature pose collected at the target moment, and then can solve the actual pose of the vehicle at the target moment.

[0104] In summary, through the above method, the in-vehicle computer can comprehensively consider the trajectory pose and feature pose to obtain relatively accurate vehicle pose information, and then can complete functions such as parking based on vehicle positioning.

[0105] Figure 12 According to the embodiments of the present application, a schematic diagram of the hardware structure of an in-vehicle computer device is shown.

[0106] As Figure 12 shown, the in-vehicle computer 1200 may include a processor 110, a wireless communication module 120, a mobile communication module 130, a power module 140, an audio module 150, an interface module 160, a main camera 170, a wide-angle camera 171, a memory 180, a sensor module 190, a button 102, a motor 103, and an indicator 104, etc. Among them, the sensor module 190 may include a seat sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a touch sensor, an ambient light sensor, etc.

[0107] It can be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the vehicle machine 1200. In some other embodiments of the present application, the vehicle machine 1200 may include more or fewer components than those illustrated, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0108] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0109] The controller may generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.

[0110] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can save the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be directly called from the above memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0111] In the embodiments of the present application, the processor 110 of the vehicle machine 1200 may complete fetching instructions, executing instructions, etc. through the controller to implement the Figure 11 relevant steps executed by the vehicle machine described above. For specific details, reference may be made to the relevant descriptions in the above embodiments, and details will not be elaborated here.

[0112] The interface module 160 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM card interface, and / or a universal serial bus (USB) interface, etc.

[0113] The power module 140 is used to connect the battery, the charging management module, the processor 110, etc. The power module 140 receives the input from the battery and / or the charging management module and supplies power to the processor 110, the memory 180, the main camera 170, the wide-angle camera 171, the wireless communication module 120, etc.

[0114] The wireless communication module 120 may provide wireless communication solutions applied to the in-vehicle unit 1200, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc.

[0115] The mobile communication module 130 may provide wireless communication solutions applied to the in-vehicle unit 1200, including 2G / 3G / 4G / 5G, etc. The mobile communication module 130 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc.

[0116] The main camera 170 and the wide-angle camera 171 are used to capture static images or videos. An object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transfers the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in standard formats such as RGB and YUV.

[0117] The memory 180 can be used to store computer-executable program code, and the executable program code includes instructions. The internal memory 180 can include a program storage area and a data storage area. Among them, the program storage area can store the operating system, application programs required for at least one function (such as the sound playback function, the image playback function, etc.). The data storage area can store the data created during the use of the vehicle head unit 1200 (such as audio data, phone book, etc.). In addition, the memory 180 can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the vehicle head unit 1200 by running the instructions stored in the memory 180 and / or the instructions stored in the memory provided in the processor.

[0118] The vehicle head unit 1200 can implement audio functions through the audio module 150, speakers, receivers, microphones, and application processors, etc. For example, music playback, recording, etc.

[0119] The audio module 150 is used to convert digital audio information into an analog audio signal for output, and is also used to convert analog audio input into a digital audio signal. The audio module 150 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 150 can be disposed in the processor 110, or some functional modules of the audio module 150 can be disposed in the processor 110.

[0120] The speaker, also known as the "loudspeaker", is used to convert an audio electrical signal into a sound signal. The vehicle head unit 1200 can listen to music or hands-free calls through the speaker 170A.

[0121] The receiver, also known as the "earpiece", is used to convert an audio electrical signal into a sound signal. When the vehicle head unit 1200 answers a call or a voice message, it can listen to the voice through the receiver.

[0122] A microphone, also known as a "microphone" or "transmitter", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak close to the microphone with their mouth to input the sound signal into the microphone. The vehicle head unit 1200 can be provided with at least one microphone 170C. In some other embodiments, the vehicle head unit 1200 can be provided with two microphones 170C, which can not only collect sound signals but also implement a noise reduction function. In some other embodiments, the vehicle head unit 1200 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, and implement functions such as directional recording.

[0123] The headphone jack 170D is used to connect a wired headphone. The headphone jack 170D can be a USB interface or a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0124] The buttons 102 include a power-on button, volume buttons, etc. The buttons 102 can be mechanical buttons or touch buttons. The vehicle head unit 1200 can receive button inputs and generate key signal inputs related to the user settings and function controls of the vehicle head unit 1200.

[0125] The motor 103 can generate vibration prompts. The motor 103 can be used for incoming call vibration prompts or touch vibration feedback. For example, touch operations on different applications (such as taking pictures, playing audio, etc.) can correspond to different vibration feedback effects. Touch operations on different areas of the display screen can also correspond to different vibration feedback effects by the motor 103. Different application scenarios (such as time reminder, receiving messages, alarm clock, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0126] The indicator 104 can be an indicator light, which can be used to indicate the charging status, power change, or messages, missed calls, notifications, etc.

[0127] The embodiment of the present application also provides a vehicle, including the above-mentioned vehicle head unit 1200.

[0128] The embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions run on a positioning device, the methods provided in the above embodiments are executed.

[0129] The embodiments of the present application also provide a computer program product, including computer program code, which causes a computer to execute the positioning methods provided in the above embodiments when the computer program code runs on the computer.

[0130] The embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware, or a combination of these implementation methods. The embodiments of the present application can be implemented as computer program modules or module codes executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memories and / or storage elements), at least one input device, and at least one output device.

[0131] The computer program modules or module codes can be applied to input instructions to execute the various functions described in the present application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of the present application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0132] The module codes can be implemented in a high-level modular language or an object-oriented programming language to communicate with the processing system. When needed, the module codes can also be implemented in assembly language or machine language. In fact, the mechanism described in the present application is not limited to the scope of any specific programming language. In any case, the language can be a compiled language or an interpreted language.

[0133] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more transient or non-transitory machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or via other computer-readable media. Thus, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable form (e.g., for a computer), including but not limited to, floppy disks, optical disks, optical discs, magneto-optical disks, read only memory (ROM), random access memory (RAM), erasable programmable read only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic or optical cards, flash memory, or tangible machine-readable memories for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) in electrical, optical, acoustic, or other forms using the Internet. Thus, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a machine-readable form (e.g., for a computer).

Claims

1. A positioning method, characterized in that: The vehicle machine device is applied to a vehicle, and the method comprises: A positioning request is detected, where the positioning request is used to request obtaining position information of the vehicle at a target time; Acquire multiple trajectory positions of the vehicle in a first time period, and environmental feature information collected at each trajectory position; Determine a characteristic posture corresponding to the environmental characteristic information of each trajectory posture from an environmental characteristic map, wherein the environmental characteristic map includes environmental characteristic information corresponding to different postures, the similarity between the environmental characteristic information of each trajectory posture and the environmental characteristic information of the corresponding characteristic posture is greater than a first threshold, and the similarity between the environmental characteristic information of each trajectory posture and the neighborhood characteristic information collected at adjacent time intervals of the corresponding characteristic posture is greater than a second threshold; According to the multiple trajectory postures and the multiple characteristic postures, a plurality of posture pairs for calculating the target trajectory are determined, wherein the first posture pair in the multiple posture pairs includes the first trajectory posture and the first characteristic posture collected at the first moment, and the second posture pair in the multiple posture pairs includes the second trajectory posture and the second characteristic posture, the first trajectory posture and the second trajectory posture are adjacent trajectory postures, the first characteristic posture and the second characteristic posture are adjacent characteristic postures, and the ratio of the first relative posture change to the second relative posture change is less than a preset ratio, the first relative posture change is determined based on the first characteristic posture and the second characteristic posture, and the second relative posture change is obtained based on the driving trajectory between the environmental feature information corresponding to the first characteristic posture collected when constructing the environmental feature map and the environmental feature information corresponding to the second characteristic posture collected; Fitting a target trajectory of the vehicle in the first time period according to the multiple groups of position and posture pairs; The position and posture information of the vehicle at the target moment is obtained according to the target trajectory, the trajectory and posture of the vehicle at the target moment, and the characteristic posture of the vehicle at the target moment.

2. The positioning method according to claim 1, characterized in that: The obtaining of a plurality of trajectory positions of the vehicle in a first time period includes: Acquire, by means of a chassis sensor and / or an inertial measurement unit of the vehicle, a position increment of the vehicle within each preset time interval during the first time period; A plurality of trajectory postures of the vehicle in the first time period are determined according to a plurality of posture increments of the vehicle in the first time period.

3. The positioning method according to claim 1, characterized in that: The various trajectory postures include a first trajectory posture, and the first trajectory posture corresponds to first environmental feature information. The step of determining a characteristic posture corresponding to the environmental characteristic information of each trajectory posture from the environmental characteristic map includes: Determine, from the plurality of pieces of environmental feature information included in the environmental feature map, a plurality of candidate feature information having a similarity with the first environmental feature information greater than the first threshold; Acquire neighborhood feature information collected at adjacent time intervals with each candidate feature information from the environmental feature map; According to the similarity between the first environmental feature information and each piece of neighborhood feature information, select, from the plurality of candidate feature information, target environmental feature information whose corresponding neighborhood feature information has a similarity with the first environmental feature information greater than the second threshold; According to the environmental feature map, the posture corresponding to the target environmental feature information is used as the first characteristic posture corresponding to the first environmental feature information.

4. The positioning method according to claim 3, characterized in that: The second threshold is determined by the similarity between two pieces of environmental feature information belonging to the first category of environmental feature information and the similarity between two pieces of environmental feature information belonging to the second category of environmental feature information in the environmental feature map, wherein the difference in posture between the two pieces of environmental feature information belonging to the first category of environmental feature information is greater than the difference in posture between the two pieces of environmental feature information belonging to the second category of environmental feature information.

5. The positioning method according to claim 1, characterized in that: The method further comprises: The environmental feature map is updated according to the target trajectory in the first time period and the environmental feature information corresponding to each posture in the target trajectory.

6. The positioning method according to claim 5, characterized in that: The updating of the environmental feature map includes: Adding the second environmental feature information collected at the second moment in the first time period and the trajectory position of collecting the second environmental feature information to the environmental feature map; and The third environmental characteristic information collected at the third moment in the first time period is matched with the fourth environmental characteristic information in the environmental characteristic map, and the fourth environmental characteristic information is replaced by the third environmental characteristic information.

7. A vehicle machine device, characterized in that: include: A memory, used to store instructions executed by one or more processors of the vehicle-mounted device, and a processor, which is one of the processors of the vehicle-mounted device, used to execute the positioning method described in any one of claims 1 to 6.

8. A vehicle, characterized in that: Including the vehicle equipment as described in claim 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, which, when executed on a positioning device, enable the positioning device to execute the positioning method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product comprises: a computer program code, and when the computer program code is run on a computer, the computer is enabled to execute the positioning method according to any one of claims 1 to 6.

Images