Positioning method and device of vehicle, vehicle and medium

Abstract

The application discloses a vehicle position positioning method and device, a vehicle and a medium, and relates to the technical field of vehicles. The method comprises the following steps: acquiring image information around the vehicle; acquiring a target feature form of the image information, wherein the target feature form is a feature form input by a large language model; generating a language description and position information of a current scene feature element based on the large language model and the target feature form; matching the language description and position information of the current scene feature element with a scene feature element map; and in the case of successful matching, determining the geographical position information of the vehicle according to the geographical position information of the successfully matched scene feature element in the scene feature element map and the position information of the current scene feature element. The positioning method enhances the positioning capability in the case of no signal, and improves the positioning speed and reliability of the vehicle.

Description

Technical Field

[0001] This invention relates to the field of vehicle technology, and specifically to a vehicle positioning method, a vehicle positioning device, a computer-readable storage medium, and a vehicle. Background Technology

[0002] Current autonomous driving technologies primarily utilize positioning technologies including satellite positioning, radio signal-based positioning, and visual odometry. Satellite positioning offers advantages such as global coverage, high accuracy, and independence, but is susceptible to obstructions and weather conditions, and its computation is relatively complex. Radio signal-based positioning is unusable in certain scenarios, such as valleys, tunnels, streets with tall buildings, and overpasses. Visual odometry uses continuous images captured by cameras, estimating the camera's trajectory through feature detection and matching. It offers advantages in terms of low cost and rich information, but suffers from computational complexity and accumulated errors. Combining these various methods, vehicle positioning technology has significantly improved its positioning capabilities over the years, achieving accurate positioning in most driving scenarios. However, positioning and navigation in long tunnels, indoor parking lots, and under elevated roads remain challenging.

[0003] In related technologies, patent application number CN202311629689.X, entitled "A Method and System for Autonomous Driving Based on a Large Model," discloses a method for fusing external sensor data, positioning data, and driving data with text statements input by the user in the vehicle via voice, using a large language model to generate driving decisions. This patent only uses a large language model to fuse user input and vehicle visual perception information, failing to fully utilize the multimodal large language model's matching capabilities with the current environment and existing maps. Therefore, it can only be used for local positioning and decision-making, and cannot achieve stable positioning in different scenarios. Patent application number CN202410125554.8, entitled "An Autonomous Driving Positioning Method and System," discloses a method for obtaining vehicle position information using processed visual images, establishing an autoregressive moving average model, calculating linear acceleration values ​​based on the model, and using these acceleration values ​​for spatial combination positioning. This method only uses the vehicle's own image data and acceleration data for navigation, without making full use of the perceptible information around the vehicle, without using deep neural network technology to process the data, and without using map information to enhance the positioning. Therefore, it is difficult to achieve stable positioning in different scenarios. Summary of the Invention

[0004] One objective of this invention is to provide a vehicle location positioning method that utilizes a large language model to extract and match surrounding environmental features, enabling vehicles to locate using previously unavailable environmental features and enhancing positioning capabilities in the absence of signal. Simultaneously, a scene feature map is constructed, and the map is matched with the surrounding environmental features extracted by the large language model to determine the vehicle's geographical location information, further improving the positioning speed and reliability. A second objective is to provide a vehicle location positioning device. A third objective is to provide a computer-readable storage medium. A fourth objective is to provide a vehicle.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A method for locating a vehicle includes: acquiring image information surrounding the vehicle; acquiring target feature forms of the image information, wherein the target feature forms are feature forms input from a large language model; generating linguistic descriptions and location information of current scene feature elements based on the large language model and the target feature forms; matching the linguistic descriptions and location information of the current scene feature elements with a scene feature element map; and, if a match is successful, determining the vehicle's geographical location information based on the geographical location information of the successfully matched scene feature elements in the scene feature element map and the location information of the current scene feature elements.

[0007] According to one embodiment of the present invention, matching the language description and location information of current scene feature elements with a scene feature element map includes: selecting sampling points in the scene feature element map according to the location range of the vehicle, wherein the sampling points are associated with multiple scene feature elements; matching the language description and location information of the current scene feature elements with the text description and geographic location information of the multiple scene feature elements associated with the sampling points, respectively, to obtain a first matching result of the language description of the current scene feature elements with the text description of the multiple scene feature elements associated with the sampling points, and a second matching result of the location information of the current scene feature elements with the geographic location information of the multiple scene feature elements associated with the sampling points; and determining the degree of matching based on the first matching result and the second matching result.

[0008] According to one embodiment of the present invention, the above-described positioning method further includes: determining that the match is successful when the degree of matching is greater than or equal to a preset threshold.

[0009] According to an embodiment of the present invention, the above-described positioning method further includes: determining that the matching has failed if the matching degree is less than a preset threshold; in the case of determining that the matching has failed, obtaining the scene feature element associated with the sampling point corresponding to the maximum value in the first matching result and / or the second matching result, and obtaining the next sampling point associated with the scene feature element; and matching the language description and location information of the current scene feature element with the text description and geographical location information of multiple scene feature elements associated with the next sampling point, respectively.

[0010] According to an embodiment of the present invention, the above positioning method further includes: if all the sampling points selected from the scene feature elements corresponding to the vehicle's location range fail to match, then randomly select sampling points from the scene feature elements for matching.

[0011] According to one embodiment of the present invention, obtaining the target feature form of image information includes: using a convolutional neural network to extract features from the image information, and processing the extracted features based on a visual encoder network to obtain visual features of the image information, wherein the visual features represent the semantic information of the image information; and using a deep learning network to map the visual features of the image information to transform the visual features into the target feature form.

[0012] According to one embodiment of the present invention, the convolutional neural network includes one of ResNet, EfficientNet, SwinTransformer, and VoVNetV2, and the deep learning network includes one of multilayer neural network and multilayer attention network.

[0013] According to an embodiment of the present invention, the above positioning method further includes: determining the position information of the current scene feature element based on the depth information of each pixel in the image information and the position information of the current scene feature element in the image, wherein the position information of the current scene feature element is the position information in the vehicle coordinate system.

[0014] According to one embodiment of the present invention, generating a scene feature element map includes: marking the location of scene feature elements in the current frame street view image, wherein the shooting location of the current frame street view image is recorded as a sampling point of the current frame street view image; associating the sampling point location information of the current frame street view image with the location information of the corresponding scene feature elements; marking the location of scene feature elements in the next frame street view image, wherein the shooting location of the next frame street view image is recorded as a sampling point of the next frame street view image, and the distance between the sampling point of the next frame street view image and the sampling point of the current frame street view image is less than a preset distance; associating the sampling point location information of the next frame street view image with the location information of the corresponding scene feature elements; merging the scene feature elements of the next frame street view image with the scene feature elements of the current frame street view image that are the same, to obtain a scene feature element map containing the next frame street view image and the current frame street view image, and so on, to obtain a scene feature element map within a target area.

[0015] According to one embodiment of the present invention, obtaining a large language model includes: acquiring image and text data of a vehicle in autonomous driving mode, wherein the image and text data includes images and text descriptions of the external environment; annotating the image and text data, wherein the annotated objects include objects and markers for positioning reference points; using the image and text data as input, training the large language model using an autoregressive training method to obtain parameters of the large language model; using the annotated image and text data as input, training the large language model again using a supervised training method to obtain parameters of the large language model; adjusting the hyperparameters and training parameters of the autoregressive training method and the supervised training method to optimize the parameters of the large language model, and outputting the final large language model.

[0016] A vehicle location positioning device includes: an image acquisition module for acquiring image information surrounding the vehicle; an acquisition module for acquiring target feature forms of the image information, wherein the target feature forms are feature forms input by a large language model; a generation module for generating linguistic descriptions and location information of current scene feature elements based on the large language model and the target feature forms; a matching module for matching the linguistic descriptions and location information of the current scene feature elements with a scene feature element map; and a determination module for determining the vehicle's geographical location information based on the geographical location information of the successfully matched scene feature elements in the scene feature element map and the location information of the current scene feature elements, if the matching is successful.

[0017] A computer-readable storage medium storing a vehicle location program thereon, which, when executed by a processor, implements the aforementioned vehicle location method.

[0018] A vehicle includes a memory, a processor, and a vehicle positioning program stored in the memory and executable on the processor. When the processor executes the vehicle positioning program, it implements the aforementioned vehicle positioning method.

[0019] The beneficial effects of this invention are:

[0020] (1) The positioning method of the present invention uses a large language model to extract and match the surrounding environmental features, enabling the vehicle to use environmental features that were previously unavailable for positioning, thereby enhancing the positioning capability in the absence of a signal.

[0021] (2) Construct a scene feature element map, match the scene feature element map with the surrounding environment feature elements extracted by the large language model, determine the vehicle's geographical location information, and further improve the vehicle's positioning speed and reliability. Attached Figure Description

[0022] Figure 1 A flowchart of a vehicle location positioning method according to some embodiments of the present invention;

[0023] Figure 2 This is a map of scene feature elements within a target area according to some embodiments of the present invention;

[0024] Figure 3 This is a block diagram of a vehicle positioning device according to some embodiments of the present invention;

[0025] Figure 4 This is a block diagram of a vehicle according to some embodiments of the present invention. Detailed Implementation

[0026] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0027] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0028] The following describes in detail, with reference to the accompanying drawings, the vehicle location positioning method, device, vehicle, and medium according to embodiments of the present invention.

[0029] Figure 1 This is a flowchart illustrating a vehicle positioning method according to some embodiments of the present invention. (Refer to...) Figure 1 The vehicle location positioning method in this application embodiment may include the following steps:

[0030] S110, acquires image information about the area around the vehicle.

[0031] Specifically, the vehicle is equipped with multiple cameras in various directions to acquire image information from all sides of the vehicle, and its field of view can usually cover 360° around the vehicle body.

[0032] S120, Obtain the target feature form of the image information, wherein the target feature form is the feature form input to the large language model.

[0033] Specifically, multiple images captured by each camera are acquired (one image from each camera), and then the image information is processed to obtain the target feature form of the image information. For example, firstly, a visual encoder network is used to process the image features to generate visual features, and then the visual features are input into a mapping layer for processing to obtain the corresponding target feature form of the image information. This target feature form is a feature form that the Big Prophet model can process.

[0034] S130 generates a linguistic description of the current scene feature elements and their location information based on a large language model and target feature forms.

[0035] Specifically, the target feature forms of the image information corresponding to each image are input into a large language model for processing, generating a linguistic description of the current scene feature elements and their location information. The current scene feature elements should be unique, such as signs with location indication functions, and the location information of the current scene feature elements is their location information in the vehicle coordinate system.

[0036] S140, Match the current scene feature elements with the scene feature element map based on the language description and location information of the current scene feature elements.

[0037] Specifically, the current scene features are included in the street view data. Therefore, the street view data can be used to construct a scene feature map. The scene feature map includes the geographical location information of the scene feature and the text description of the scene feature. The scene feature in the scene feature map is matched with the language description and location information of the current scene feature generated by the large language model to obtain the geographical location information of the vehicle.

[0038] S150, if a match is successful, determine the vehicle's geographical location information based on the geographical location information of the successfully matched scene feature features in the scene feature feature map and the location information of the current scene feature features.

[0039] Specifically, if the scene feature elements in the scene feature element map successfully match the linguistic description and location information of the current scene feature elements generated by the large language model, it means that the current scene feature elements around the vehicle are the same as those in the scene feature element map. The location information of the current scene feature elements around the vehicle is their location in the vehicle's coordinate system. Based on this location information, the distance between the vehicle and the current scene feature elements can be calculated. The scene feature elements in the scene feature element map are geographical location information, such as the location, longitude, latitude, and altitude of the scene feature elements on the map. Therefore, based on the geographical location information of the scene feature elements, the vehicle's position relative to the scene feature elements on the map can be determined, and thus the vehicle's geographical location information can be determined. If the scene feature elements in the scene feature element map fail to match the linguistic description and location information of the current scene feature elements generated by the large language model, it means that the current scene feature elements around the vehicle are different from those in the scene feature element map, and the vehicle's geographical location information cannot be determined.

[0040] It should be noted that the input of the large language model is the target feature form of image information and the map information of scene feature elements. The output of the large language model is the geographical location information of the vehicle. The output form of the geographical location information of the vehicle is multiple number strings, each of which can correspond to a word. The geographical location of the vehicle is composed of the words corresponding to each string.

[0041] The positioning method of this invention utilizes a large language model to extract and match features of the surrounding environment, enabling vehicles to locate using environmental features that were previously unavailable, thus enhancing positioning capabilities in the absence of signal. Simultaneously, a scene feature map is constructed, and the scene feature map is matched with the features of the surrounding environment extracted by the large language model to determine the vehicle's geographical location information, further improving the vehicle's positioning speed and reliability.

[0042] In some embodiments, matching the language description and location information of the current scene feature elements with a scene feature element map includes: selecting sampling points in the scene feature element map based on the location range of the vehicle, wherein the sampling points are associated with multiple scene feature elements; matching the language description and location information of the current scene feature elements with the text description and geographic location information of the multiple scene feature elements associated with the sampling points, respectively, to obtain a first matching result of the language description of the current scene feature elements with the text description of the multiple scene feature elements associated with the sampling points, and a second matching result of the location information of the current scene feature elements with the geographic location information of the multiple scene feature elements associated with the sampling points; and determining the degree of matching based on the first matching result and the second matching result.

[0043] Specifically, vehicles are typically equipped with positioning receivers or combined positioning devices, which can provide positioning signals from satellites or base stations. When the positioning signal is good, the vehicle's location range will be relatively accurate, possibly a precise point. In this case, the signal can be used directly as the vehicle's positioning output, but it can also be combined with the positioning method of this invention to achieve more precise positioning. When the positioning signal is poor, only the vehicle's location range can be roughly determined, meaning the vehicle's location range may be a large circular area. In this case, the method of this invention should be relied upon primarily for vehicle positioning.

[0044] Furthermore, sampling points are selected in the scene feature element map based on the vehicle's location range. These sampling points are the locations where the scene feature element map was captured, and each sampling point is associated with multiple scene feature elements, each with its corresponding text description and geographic location information. When matching the current scene feature element's language description and location information with the scene feature element map, the language description of the current scene feature element is matched with the text descriptions of each scene feature element associated with the sampling point to obtain a first matching result. The location information of the current scene feature element is then matched with the geographic location information of each scene feature element associated with the sampling point to obtain a second matching result. The weighted sum of the first and second matching results determines the degree of matching.

[0045] It should be noted that the number of current scene feature elements generated by the large language model can be one or more. When there are multiple current scene feature elements, each current scene feature element should be matched with the scene feature elements associated with the sampling point.

[0046] For example, suppose the current scene feature elements generated by the large language model are A and B, and the scene feature elements associated with the sampling points are A' and B'. When matching the current scene feature elements with the scene feature element map based on their linguistic descriptions and location information, a matching degree is obtained by matching the current scene feature element A with the scene feature element associated with the sampling point A', a matching degree is obtained by matching the current scene feature element A with the scene feature element associated with the sampling point B', a matching degree is obtained by matching the current scene feature element B with the scene feature element associated with the sampling point A', and a matching degree is obtained by matching the current scene feature element B with the scene feature element associated with the sampling point B'.

[0047] In some embodiments, the above positioning method further includes: determining a successful match if the matching degree is greater than or equal to a preset threshold. The preset threshold is determined based on actual circumstances and is not specifically limited here.

[0048] Specifically, a successful match is determined by comparing the degree of matching between the language description and location information of the current scene feature elements and the text description and geographic location information of multiple scene feature elements associated with the sampling point with a preset threshold. For example, if the degree of matching is greater than or equal to the preset threshold, it is determined that the current scene feature element has successfully matched the scene feature element associated with the sampling point. At this time, the location of the vehicle relative to the scene feature element on the map can be determined based on the geographic location information of the scene feature element, and thus the geographic location information of the vehicle can be determined.

[0049] In some embodiments, the above positioning method further includes: determining that the matching has failed if the matching degree is less than a preset threshold; in the case of determining that the matching has failed, obtaining the scene feature element associated with the sampling point corresponding to the maximum value in the first matching result and / or the second matching result, and obtaining the next sampling point associated with the scene feature element; and matching the language description and location information of the current scene feature element with the text description and geographical location information of multiple scene feature elements associated with the next sampling point, respectively.

[0050] Specifically, if the matching degree is less than a preset threshold, it is determined that the current scene feature element fails to match the scene feature element associated with the sampling point. In this case, the first and second matching results calculated during the matching process of the current scene feature element and the scene feature element associated with the sampling point are sorted by size. The scene feature element associated with the sampling point corresponding to the maximum value of the first matching result is obtained, or the scene feature element associated with the sampling point corresponding to the maximum value of the second matching result is obtained, or the scene feature elements associated with the sampling points corresponding to the first and second matching results are obtained simultaneously. The next sampling point associated with the scene feature element in the scene feature element map is then obtained. The language description and location information of the current scene feature element are matched with the text description and geographical location information of multiple scene feature elements associated with the next sampling point, respectively. The specific matching method has been described in detail in the above embodiments and will not be repeated here.

[0051] In some embodiments, the above positioning method further includes: if all the sampling points selected from the scene feature elements corresponding to the vehicle's location range fail to match, then randomly select sampling points from the scene feature elements for matching.

[0052] In some embodiments, obtaining the target feature form of image information includes: using a convolutional neural network to extract features from the image information, wherein visual features characterize the semantic information of the image information, and processing the extracted features based on a visual encoder network to obtain the visual features of the image information; and using a deep learning network to map the visual features of the image information to transform the visual features into the target feature form.

[0053] In some embodiments, the convolutional neural network includes one of the ResNet network, the EfficientNet network, the SwinTransformer network, and the VoVNetV2 network, and the deep learning network includes one of the multilayer neural network and the multilayer attention network.

[0054] For example, when preprocessing image information, a convolutional neural network, such as one of ResNet, EfficientNet, SwinTransformer, or VoVNetV2, is first used to extract features from the image information. Then, a visual encoder network is used to process the extracted features to obtain the visual features of the image information. These visual features are a multi-dimensional feature matrix that can be used to represent the semantic information of the image. Finally, a deep learning network, such as a multi-layer neural network or a multi-layer attention network, is used to map the visual features of the image information, transforming them into a target feature form that can be processed by the large prediction model.

[0055] In some embodiments, the above positioning method further includes: determining the position information of the current scene feature elements based on the depth information of each pixel in the image information and the position information of the current scene feature elements in the image, wherein the position information of the current scene feature elements is the position information in the vehicle coordinate system.

[0056] Specifically, after cameras are installed on the vehicle, extrinsic parameter calibration is required for each camera to be placed in the same coordinate system, i.e., converted to the vehicle's coordinate system. Furthermore, time synchronization processing is needed for the data from each camera to ensure that the data acquisition times are similar and that the external environment does not change significantly within the data acquisition intervals between cameras. For multiple images acquired by the cameras, the depth information of any pixel in each image can be obtained, i.e., the distance information from the pixel to the camera. Since all cameras and sensors have been calibrated in the vehicle's coordinate system, the position of any pixel in the current scene feature element within the vehicle's coordinate system can be determined, thus enabling the determination of the positional information of the current scene feature element.

[0057] For example, a camera calibration method, such as one using image perspective transformation and keypoint matching, can be used to obtain a 3×3 intrinsic parameter matrix for the camera. Based on the transformation relationship from the vehicle coordinate system to the camera coordinate system, the camera can be calibrated in the vehicle coordinate system. Each camera can be time-synchronized using either synchronous or asynchronous triggering. All cameras use the same clock source, and the timestamp of data acquisition is appended to the camera data. Data processing then performs time matching based on similar times.

[0058] In some embodiments, generating a scene feature map includes: marking the location of scene feature elements in the current frame street view image, wherein the shooting location of the current frame street view image is recorded as a sampling point of the current frame street view image; associating the sampling point location information of the current frame street view image with the location information of the corresponding scene feature elements; marking the location of scene feature elements in the next frame street view image, wherein the shooting location of the next frame street view image is recorded as a sampling point of the next frame street view image, and the distance between the sampling point of the next frame street view image and the sampling point of the current frame street view image is less than a preset distance; associating the sampling point location information of the next frame street view image with the location information of the corresponding scene feature elements; merging the scene feature elements of the next frame street view image with the scene feature elements of the current frame street view image that are the same, to obtain a scene feature map containing the next frame street view image and the current frame street view image, and so on, to obtain a scene feature map within the target area.

[0059] Specifically, when a vehicle is traveling in a difficult-to-locate area, such as a tunnel or overpass, the positioning method of this invention needs to be used for positioning. Therefore, it is necessary to pre-construct a scene feature element map of these difficult-to-locate areas (target areas) so that the language description and location information of the current scene feature elements can be matched with the scene feature element map to determine the vehicle's geographical location information.

[0060] For example, a street view image of the current frame is captured at a certain shooting location, which is recorded as the sampling point of the current frame street view image. Notable scene feature elements (such as distance signs) and their geographical location information are marked from this street view image, such as the location of the scene feature elements on the map, longitude information, latitude information, and altitude information. At the same time, the geographical location information of the sampling point is recorded, such as the location of the sampling point on the map, longitude information, latitude information, and altitude information. The geographical location information of the sampling point of the current frame street view image is associated with the location information of the corresponding scene feature elements, that is, the sampling point of the current frame street view image is connected with the scene feature elements, so that the sampling point of the current frame street view image and the corresponding scene feature elements can be mutually indexed.

[0061] Then, the shooting position is moved to capture the next frame of the street view image. This shooting position is recorded as the sampling point of the next frame. Similarly, noteworthy scene features and their geographical locations are marked in this frame, and the geographical location information of the sampling point is recorded. The geographical location information of the sampling point in the next frame is associated with the location information of the corresponding scene feature features, enabling mutual indexing between the sampling point and the corresponding scene feature features. This involves connecting the sampling point of the next frame with the scene feature features. Since the distance between the sampling point of the next frame and the sampling point of the current frame is less than a preset distance, the scene feature features of the current frame and the next frame will overlap. These identical scene feature features are merged to obtain a scene feature map containing both the next and current frame street view images. This process is repeated to obtain a scene feature map of the target area, as shown below. Figure 2 As shown.

[0062] In some embodiments, obtaining a large language model includes: acquiring image and text data of the vehicle during autonomous driving, wherein the image and text data includes images and text descriptions of the external environment; annotating the image and text data, wherein the annotated objects include objects and markers for positioning reference points; using the image and text data as input, training the large language model using an autoregressive training method to obtain parameters of the large language model; using the annotated image and text data as input, training the large language model again using a supervised training method to obtain parameters of the large language model; adjusting the hyperparameters and training parameters of the autoregressive training method and the supervised training method to optimize the parameters of the large language model, and outputting the final large language model.

[0063] Specifically, a large language model should be generated through autoregressive training and supervised training on large-scale data, enabling it to perform the aforementioned location localization task. This large-scale data includes both image and text data from the vehicle's autonomous driving process and annotated image and text data. Image and text data refers to images (e.g., pictures or videos from one or more perspectives of the vehicle during operation) and textual descriptions of the external environment (e.g., descriptions of objects outside the vehicle, road markings, and the overall environment). Annotated image and text data includes objects and markings that serve as localization reference points. In other words, a large language model can be generated based on both the image and text data and the annotated image and text data through autoregressive training and supervised training.

[0064] For example, text and image data are used as input, and an autoregressive training method is used to train the large language model to obtain its parameters. Based on these parameters, labeled text and image data are used as input, and a supervised training method is used to train the large language model again to obtain its parameters. Then, the hyperparameters and training parameters of the autoregressive and supervised training methods are adjusted to optimize the parameters of the large language model, outputting the final large language model. Then, validation image information is input into the final large language model, and the effectiveness of the large language model is verified by comparing the image information of the validation images with the semantic information output by the large language model.

[0065] In summary, the positioning method of this invention utilizes a large language model to extract and match features of the surrounding environment, enabling vehicles to locate using environmental features that were previously unavailable, thus enhancing positioning capabilities in the absence of a signal. Simultaneously, it constructs a scene feature map and matches it with the surrounding environmental features extracted by the large language model to determine the vehicle's geographical location information, further improving the vehicle's positioning speed and reliability.

[0066] Corresponding to the above embodiments, this application also proposes a vehicle positioning device.

[0067] Reference Figure 3The vehicle positioning device 200 includes: an image acquisition module 210, a processing module 220, a generation module 230, a matching module 240, and a determination module 250.

[0068] The system includes several modules: an image acquisition module 210 for acquiring image information surrounding the vehicle; an acquisition module 220 for acquiring target feature forms from the image information, where the target feature forms are the feature forms input from a large language model; a generation module 230 for generating linguistic descriptions and location information of current scene feature elements based on the large language model and the target feature forms; a matching module 240 for matching the linguistic descriptions and location information of the current scene feature elements with a scene feature element map; and a determination module 250 for determining the vehicle's geographical location information based on the geographical location information of the successfully matched scene feature elements in the scene feature element map and the location information of the current scene feature elements, if a match is successful.

[0069] According to one embodiment of the present invention, the matching module 240 is specifically configured to: select sampling points in a scene feature element map according to the location range of the vehicle, wherein the sampling points are associated with multiple scene feature elements; match the language description and location information of the current scene feature element with the text description and geographic location information of the multiple scene feature elements associated with the sampling point, respectively, to obtain a first matching result between the language description of the current scene feature element and the text description of the multiple scene feature elements associated with the sampling point, and a second matching result between the location information of the current scene feature element and the geographic location information of the multiple scene feature elements associated with the sampling point; and determine the degree of matching based on the first matching result and the second matching result.

[0070] According to one embodiment of the present invention, the determining module 250 is specifically used to determine that the matching is successful when the matching degree is greater than or equal to a preset threshold.

[0071] According to one embodiment of the present invention, the determining module 250 is specifically configured to: determine that the matching has failed if the matching degree is less than a preset threshold; in the case of determining that the matching has failed, obtain the scene feature element associated with the sampling point corresponding to the maximum value in the first matching result and / or the second matching result, and obtain the next sampling point associated with the scene feature element; and match the language description and location information of the current scene feature element with the text description and geographical location information of multiple scene feature elements associated with the next sampling point, respectively.

[0072] According to one embodiment of the present invention, the determining module 250 is further configured to, if all the sampling points selected from the scene feature elements corresponding to the vehicle's location range fail to match, randomly select sampling points from the scene feature elements for matching.

[0073] According to one embodiment of the present invention, the image acquisition module 210 preprocesses the image information to obtain the target feature form of the image information. Specifically, it uses a convolutional neural network to extract features from the image information and processes the extracted features based on a visual encoder network to obtain the visual features of the image information; and uses a deep learning network to map the visual features of the image information to transform the visual features into the target feature form.

[0074] According to one embodiment of the present invention, the convolutional neural network includes one of ResNet, EfficientNet, SwinTransformer, and VoVNetV2, and the deep learning network includes one of multilayer neural network and multilayer attention network.

[0075] According to an embodiment of the present invention, the determining module 250 is further configured to determine the position information of the current scene feature element based on the depth information of each pixel in the image information and the position information of the current scene feature element in the image, wherein the position information of the current scene feature element is the position information in the vehicle coordinate system.

[0076] According to one embodiment of the present invention, the generation module 230 generates a scene feature element map, specifically configured to: mark the location of scene feature elements in the current frame street view image, wherein the shooting location of the current frame street view image is recorded as the sampling point of the current frame street view image; associate the sampling point location information of the current frame street view image with the location information of the corresponding scene feature elements; mark the location of scene feature elements in the next frame street view image, wherein the shooting location of the next frame street view image is recorded as the sampling point of the next frame street view image, and the distance between the sampling point of the next frame street view image and the sampling point of the current frame street view image is less than a preset distance; associate the sampling point location information of the next frame street view image with the location information of the corresponding scene feature elements; merge the scene feature elements of the next frame street view image with the scene feature elements of the current frame street view image that are the same, to obtain a scene feature element map containing the next frame street view image and the current frame street view image, and so on, to obtain a scene feature element map within the target area.

[0077] It should be noted that the above explanation of the embodiments and beneficial effects of the vehicle positioning method also applies to the vehicle positioning device of the present invention. To avoid redundancy, it will not be elaborated in detail here.

[0078] Corresponding to the above embodiments, this application also proposes a computer-readable storage medium.

[0079] The present application provides a computer-readable storage medium storing a vehicle location program thereon, which, when executed by a processor, implements the aforementioned vehicle location method.

[0080] It should be noted that the above explanation of the embodiments and beneficial effects of the vehicle positioning method is also applicable to the computer-readable storage medium of the embodiments of the present invention. To avoid redundancy, it will not be elaborated in detail here.

[0081] Corresponding to the above embodiments, this application also proposes a vehicle.

[0082] Reference Figure 4 The vehicle 300 of this application includes a memory 310, a processor 320, and a vehicle location positioning program stored in the memory 310 and executable on the processor 320. When the processor 320 executes the vehicle location positioning program, it implements the aforementioned vehicle location positioning method.

[0083] It should be noted that the above explanation of the embodiments and beneficial effects of the vehicle positioning method also applies to the vehicles in the embodiments of the present invention. To avoid redundancy, they will not be elaborated in detail here.

[0084] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0085] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0086] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0087] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0088] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0089] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

[0090] The above embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention.

Claims

1. A method for locating the position of a vehicle, characterized in that, The method includes: Acquire image information about the area surrounding the vehicle; Obtain the target feature form of the image information, wherein the target feature form is the feature form input by the large language model; Based on the large language model and the target feature form, generate a linguistic description of the current scene feature elements and their location information; The scene feature elements are matched with the scene feature element map based on the language description of the current scene feature elements and the location information. If a match is successful, the geographical location information of the vehicle is determined based on the geographical location information of the successfully matched scene feature features in the scene feature feature map and the location information of the current scene feature feature. The matching of the current scene feature elements with the scene feature element map based on the language description and location information includes: Sampling points are selected in the scene feature element map based on the location range of the vehicle, wherein the sampling points are associated with multiple scene feature elements; The language description and location information of the current scene feature element are matched with the text description and geographic location information of multiple scene feature elements associated with the sampling point, respectively, to obtain a first matching result of the language description of the current scene feature element and the text description of multiple scene feature elements associated with the sampling point, and a second matching result of the location information of the current scene feature element and the geographic location information of multiple scene feature elements associated with the sampling point. The degree of matching is determined based on the first matching result and the second matching result.

2. The vehicle positioning method according to claim 1, characterized in that, The method further includes: If the matching degree is greater than or equal to a preset threshold, the matching is determined to be successful.

3. The vehicle positioning method according to claim 2, characterized in that, The method further includes: If the matching degree is less than the preset threshold, the matching is determined to be unsuccessful; If a match fails, obtain the scene feature element associated with the sampling point corresponding to the maximum value in the first matching result and / or the second matching result, and obtain the next sampling point associated with that scene feature element. The language description and location information of the current scene feature elements are matched with the text description and geographic location information of multiple scene feature elements associated with the next sampling point.

4. The vehicle positioning method according to claim 3, characterized in that, The method further includes: If all the selected sampling points in the scene feature elements within the location range of the vehicle fail to match, then sampling points are randomly selected from the scene feature elements for matching.

5. The vehicle positioning method according to claim 1, characterized in that, Obtaining the target feature form of the image information includes: A convolutional neural network is used to extract features from the image information, and the extracted features are processed based on a visual encoder network to obtain the visual features of the image information, wherein the visual features represent the semantic information of the image information. A deep learning network is used to map the visual features of the image information, transforming the visual features into the target feature form.

6. The vehicle positioning method according to claim 5, characterized in that, The convolutional neural network includes one of ResNet, EfficientNet, SwinTransformer, and VoVNetV2, and the deep learning network includes one of multi-layer neural networks and multi-layer attention networks.

7. The vehicle positioning method according to claim 1, characterized in that, The method further includes: The position information of the current scene feature element is determined based on the depth information of each pixel in the image information and the position information of the current scene feature element in the image. The position information of the current scene feature element is the position information in the vehicle coordinate system.

8. The vehicle positioning method according to claim 1, characterized in that, Generating the scene feature map includes: The scene feature elements of the current frame street view image are marked with their positions, wherein the shooting position of the current frame street view image is recorded as the sampling point of the current frame street view image; Associate the sampling point location information of the current frame street view image with the location information of the corresponding scene feature elements; The scene feature elements of the next frame street view image are marked with their positions. The shooting position of the next frame street view image is recorded as the sampling point of the next frame street view image. The distance between the sampling point of the next frame street view image and the sampling point of the current frame street view image is less than a preset distance. The sampling point location information of the next frame street view image is associated with the location information of the corresponding scene feature elements; The scene feature elements of the next frame street view image are merged with the scene feature elements of the current frame street view image to obtain a scene feature element map containing the next frame street view image and the current frame street view image. This process is repeated to obtain the scene feature element map within the target area.

9. The vehicle positioning method according to claim 1, characterized in that, Obtaining a large language model includes: Acquire the image and text data of the vehicle in autonomous driving mode, wherein the image and text data includes images and text descriptions of the external environment; The graphic data is annotated, wherein the annotated objects include objects and markers used to locate reference points; The image and text data are used as input, and the model is trained using an autoregressive training method to obtain the parameters of the large language model. The labeled image and text data are used as input, and the large language model is trained again using a supervised training method to obtain the parameters of the large language model. The parameters of the large language model are optimized by adjusting the hyperparameters and training parameters of the autoregressive training method and the supervised training method, and the final large language model is output.

10. A vehicle positioning device, characterized in that, The device includes: The image acquisition module is used to acquire image information about the area surrounding the vehicle; The acquisition module is used to acquire the target feature form of the image information, wherein the target feature form is the feature form input by the large language model; The generation module is used to generate a linguistic description of the current scene feature elements and their location information based on the large language model and the target feature form; The matching module is used to match the current scene feature elements with the scene feature element map based on the language description of the current scene feature elements and the location information; The determination module is used to determine the geographical location information of the vehicle based on the geographical location information of the successfully matched scene feature features in the scene feature feature map and the location information of the current scene feature feature when a match is successful. The matching module is specifically used to: select sampling points in the scene feature element map according to the location range of the vehicle, wherein the sampling points are associated with multiple scene feature elements; match the language description and location information of the current scene feature element with the text description and geographical location information of the multiple scene feature elements associated with the sampling point, respectively, to obtain a first matching result between the language description of the current scene feature element and the text description of the multiple scene feature elements associated with the sampling point, and a second matching result between the location information of the current scene feature element and the geographical location information of the multiple scene feature elements associated with the sampling point; and determine the degree of matching based on the first matching result and the second matching result.

11. A computer-readable storage medium, characterized in that, It stores a vehicle location program, which, when executed by a processor, implements the vehicle location method according to any one of claims 1-9.

12. A vehicle, characterized in that, The system includes a memory, a processor, and a vehicle location tracking program stored in the memory and executable on the processor. When the processor executes the vehicle location tracking program, it implements the vehicle location tracking method according to any one of claims 1-9.

Images