Map construction method, map construction system, map construction device, vehicle and storage medium
By combining vehicle-mounted sensors and roadside equipment information, intelligent vehicles build high-precision maps, solving the problems of low map accuracy and environmental impact in the existing technology, and achieving a clear description of road details.
Patent Information
- Application Number
- CN202411462320.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-18
- Publication Date
- 2024-12-06
AI Technical Summary
The maps built in real-time during driving by existing smart vehicles are difficult to achieve high accuracy, especially the road topology description is not accurate enough, and the description of the lane connection relationship and lane center line is lacking, and it is greatly affected by visual obstruction and weather.
By combining the surrounding environment images collected by on-board sensors and road segment information broadcast by roadside equipment, feature extraction models and element extraction models are used for feature encoding and map element extraction, and the target map is constructed after fusing and displaying it on the interface of on-board equipment.
It realizes high-precision map construction, which can clearly reflect detailed information such as roadside lines and center lines, improves the accuracy of map construction, and reduces environmental impacts, such as weather or field of view occlusion.
Smart Images

Figure CN119085691A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of intelligent vehicle technology, and in particular to a map construction method, a map construction system, a device, a vehicle and a storage medium. Background Art
[0002] With the continuous development of intelligent vehicle technology, intelligent vehicles can build vehicle environmental maps online in real time, so that vehicle drivers can refer to the environmental maps to make wise decisions while driving, thereby bypassing obstacles and avoiding accidents, which can improve the safety of intelligent vehicle driving.
[0003] In the related art, when a smart vehicle is driving, a laser radar or an environmental information sensor on the smart vehicle collects the surrounding environmental information in real time, and constructs an environmental map based on the environmental information in real time. However, the above-mentioned method of constructing a map in real time is difficult to construct a map with high accuracy. Summary of the invention
[0004] Based on this, it is necessary to provide a map construction method, a map construction system, a device, a vehicle and a storage medium that can construct a high-precision map in response to the above technical problems.
[0005] In a first aspect, the present application provides a map construction method. The method comprises:
[0006] Obtaining surrounding environment images collected by on-board sensors on the vehicle;
[0007] Receiving the road section information broadcasted by the roadside equipment of the road section where the vehicle is located;
[0008] Constructing a target map according to the surrounding environment image and the road section information;
[0009] The target map is displayed on a display interface on the vehicle-mounted device.
[0010] In one embodiment, constructing a target map according to the surrounding environment image and the road section information includes:
[0011] Inputting the surrounding environment image into a feature extraction model for feature encoding to obtain environment map features; the environment map features include environment depth features, environment bird's-eye view features, and environment content features;
[0012] Inputting the road section information into an element extraction model to extract map elements to obtain roadside map elements; the roadside map elements include line elements, road sign elements, and traffic signal elements;
[0013] The environmental map features and the roadside map elements are input into a map decoder to construct the target map.
[0014] In one embodiment, the feature extraction model includes a feature extraction network, a deep branch network, a feature conversion network, and a content semantic branch network; the surrounding environment image is input into the feature extraction model for feature encoding to obtain the environment map features, including:
[0015] Inputting the surrounding environment image into the feature extraction network to extract global features, so as to obtain global features of the surrounding environment image;
[0016] Inputting the global features and the surrounding environment image into the deep branch network to extract deep features, thereby obtaining environmental deep features of the surrounding environment image;
[0017] Inputting the global features and the surrounding environment image into the feature conversion network for feature conversion to obtain the environmental bird's-eye view features of the surrounding environment image;
[0018] The global features and the surrounding environment image are input into the content semantic branch network to extract semantic features, so as to obtain the environmental content features of the surrounding environment image.
[0019] In one embodiment, the element extraction model includes a line extraction network, a sign extraction network, and a signal light extraction network; the step of inputting the road section information into the element extraction model to extract map elements to obtain roadside map elements includes:
[0020] Inputting the road section information into the line extraction network to extract line elements to obtain the line elements;
[0021] Inputting the road section information into the sign extraction network to extract sign elements to obtain the road indication sign elements;
[0022] The road section information is input into the traffic light extraction network to extract signals, thereby obtaining the traffic signal elements.
[0023] In one embodiment, the step of inputting the environment map features and the roadside map elements into a map decoder to construct the target map includes:
[0024] Fusing the environment map features with the roadside map elements to obtain fused features;
[0025] The fused features are input into a map decoder to construct the target map.
[0026] In one embodiment, the fusing the environment map feature and the roadside map element to obtain the fused feature includes:
[0027] converting the environmental map feature into a first vector, and converting the roadside map element into a second vector;
[0028] The first vector and the second vector are fused to obtain the fused feature.
[0029] In one embodiment, the method further comprises:
[0030] Obtaining location information of the vehicle;
[0031] Determine target intersection information according to the location information of the vehicle and the road section information broadcast by the roadside equipment;
[0032] Constructing a next intersection map according to the target intersection information;
[0033] The next intersection map is displayed on the display interface.
[0034] In a second aspect, the present application also provides a map construction system. The system includes vehicle-mounted sensors, vehicle-mounted equipment, and roadside equipment;
[0035] The vehicle-mounted device is used to execute the method as described in the first aspect.
[0036] In a third aspect, the present application further provides a map construction device, the device comprising:
[0037] An acquisition module is used to acquire the surrounding environment image collected by the vehicle-mounted sensor;
[0038] A receiving module, used for receiving the road section information broadcasted by the roadside equipment of the road section where the vehicle is located;
[0039] A construction module, used to construct a target map according to the surrounding environment image and the road section information;
[0040] A display module is used to display the target map on a display interface on the vehicle-mounted device.
[0041] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:
[0042] Obtaining surrounding environment images collected by on-board sensors on the vehicle;
[0043] Receiving the road section information broadcasted by the roadside equipment of the road section where the vehicle is located;
[0044] Constructing a target map according to the surrounding environment image and the road section information;
[0045] The target map is displayed on a display interface on the vehicle-mounted device.
[0046] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:
[0047] Obtaining surrounding environment images collected by on-board sensors on the vehicle;
[0048] Receiving the road section information broadcasted by the roadside equipment of the road section where the vehicle is located;
[0049] Constructing a target map according to the surrounding environment image and the road section information;
[0050] The target map is displayed on a display interface on the vehicle-mounted device.
[0051] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:
[0052] Obtaining surrounding environment images collected by on-board sensors on the vehicle;
[0053] Receiving the road section information broadcasted by the roadside equipment of the road section where the vehicle is located;
[0054] Constructing a target map according to the surrounding environment image and the road section information;
[0055] The target map is displayed on a display interface on the vehicle-mounted device.
[0056] In the above-mentioned map construction method, map construction system, device, vehicle and storage medium, the vehicle-mounted device obtains the surrounding environment image collected by the vehicle-mounted sensor, and receives the road section information broadcast by the roadside device of the road section where the vehicle is located, and constructs the target map according to the surrounding environment image and the road section information, and displays the target map on the display interface of the vehicle-mounted device. When the vehicle-mounted device in the above-mentioned method constructs the map online, it can obtain the road section information from the roadside device, and construct the target map by combining the road section information and the surrounding environment image collected by the vehicle-mounted sensor, so that the quality of the target map is not affected by the environment where the vehicle is located, such as weather or field of view obstruction, etc., thereby improving the accuracy of constructing the target map. In addition, the fusion of the road section information and the surrounding environment image can be realized to improve the refinement of the target map, so that the constructed target map can clearly reflect the road sideline, road centerline and other detailed information in the road, thereby improving the accuracy of map construction. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] Figure 1 A diagram of an application environment of a map construction method in an embodiment;
[0058] Figure 2 One of the flowcharts of a map construction method in one embodiment;
[0059] Figure 2A One of the schematic diagrams of a display interface for map construction in one embodiment;
[0060] Figure 3 The second flowchart of the map construction method in one embodiment;
[0061] Figure 4 is a schematic diagram of the structure of a feature extraction model in an embodiment;
[0062] Figure 5 The third flowchart of a map construction method in one embodiment;
[0063] Figure 6 A schematic diagram of the structure of an element extraction model in an embodiment;
[0064] Figure 7 The fourth flowchart of the map construction method in one embodiment;
[0065] Figure 8 FIG5 is a flowchart of a map construction method in one embodiment;
[0066] Fig. 8A A second schematic diagram of a display interface for map construction in one embodiment;
[0067] Fig. 9 The sixth flowchart of a map construction method in one embodiment;
[0068] Fig.10 FIG7 is a flowchart of a map construction method in one embodiment;
[0069] Fig.11 A schematic diagram of the structure of a map construction system in one embodiment;
[0070] Fig.12 FIG8 is a flowchart of a map construction method in one embodiment;
[0071] Fig.13 It is a structural block diagram of a map construction device in one embodiment;
[0072] Fig.14 is a structural block diagram of a vehicle in one embodiment;
[0073] Fig.15 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0074] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0075] The map construction method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. It includes a vehicle 102 and multiple roadside devices 104, and the vehicle 102 can communicate with each roadside device 104 through the network. The vehicle 102 is equipped with vehicle-mounted devices, vehicle-mounted sensors, wireless communication modules, positioning modules and processing modules, etc. The vehicle-mounted sensors are used to collect perceived information, that is, the surrounding environment image of the vehicle 102; the positioning module is used to locate the location of the vehicle, that is, the location information of the vehicle; the processing module can be connected to the vehicle-mounted sensor, the wireless communication module, and the positioning module respectively, and the processing module is used to obtain the surrounding environment image collected from the vehicle-mounted sensor, and the processing module is also used to obtain the location information of the vehicle from the positioning module. The processing module can also communicate with each roadside device 104 through the wireless communication module, and obtain the road section information broadcast by each roadside device 104, so as to combine the vehicle's location information, the surrounding environment image and the road section information broadcast by each roadside device 104 to build a target map, and display the target map in real time on the display screen of the vehicle-mounted device of the vehicle 102 for easy viewing by users. Each roadside device 104 is used to monitor the road information within the covered road section in real time and broadcast the road information, wherein the road information may include road topology information, road image information, etc.
[0076] At present, in the field of intelligent vehicle driving, vehicles can build environmental maps of the road sections where the vehicles are located online, that is, intelligent vehicles generate environmental maps of road sections in real time. Online real-time map construction ensures the freshness of map data, even in dynamic environments where road conditions change frequently. It enables vehicles to make wise decisions, bypass obstacles and avoid accidents. In related technologies, when intelligent vehicles are driving, the laser radar or environmental information sensor device on the intelligent vehicles will perceive the surrounding environmental information in real time, and build an environmental map based on this environmental information in real time. However, the above-mentioned method of real-time map construction has the following problems: the description of road topology is not accurate and detailed enough; there is a lack of description of the lane connection relationship and lane centerline in the intersection; the map construction distance is limited due to line of sight obstruction and weather.
[0077] In view of this, an embodiment of the present application provides a map construction method. By combining the road section information broadcast by roadside equipment and the surrounding environment images collected by vehicle-mounted sensors, this method can realize real-time construction of a high-precision map. The map can accurately describe the topological structure of the road and is not easily affected by the limited mapping distance caused by reasons such as line of sight obstruction and weather.
[0078] It should be noted that the beneficial effects brought about by the embodiments of the present application or the technical problems solved are not limited to this one, but may also include other implicit or related problems. For details, please refer to the description of the following embodiments.
[0079] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0080] In one embodiment, Figure 2 As shown, a map construction method is provided, which is applied to Figure 1 The vehicle-mounted device in the example is used to illustrate, including the following steps:
[0081] S201, obtaining a surrounding environment image collected by an on-board sensor of a vehicle.
[0082] Among them, the on-board sensor is used to collect road information of the vehicle's surrounding environment. A vehicle can be equipped with multiple on-board sensors. Different on-board sensors can collect environmental images from different perspectives, so all on-board sensors on the vehicle can collect environmental images from all perspectives, such as 360° environmental images.
[0083] In the embodiment of the present application, a wireless communication module, a processing module and a positioning module are provided on the vehicle-mounted device, and the processing module is connected to the vehicle-mounted sensor, the wireless communication module and the positioning module. When the vehicle is started, the vehicle-mounted device can immediately start the vehicle-mounted sensor on the vehicle to collect the perception information around the vehicle and obtain the surrounding environment image; then, the processing module can obtain the surrounding environment image. One scenario is that the surrounding environment image is sent to the roadside device for storage, so that the roadside device can obtain the road section information based on the surrounding environment image for broadcasting; in another scenario, when the processing module obtains the surrounding environment image, it can immediately perform steps S202-S204 based on this information to generate and display the target map. It should be noted that different vehicle-mounted sensors can collect environmental images from different perspectives, so multiple vehicle-mounted sensors on the vehicle collect environmental images from their respective perspectives, so the method for obtaining the environmental map around the vehicle specifically includes: the processing module can use a corresponding image stitching algorithm or an image processing algorithm to stitch the environmental images collected by all vehicle-mounted sensors from their respective perspectives to generate a stitched 360° surrounding environment image.
[0084] S202, receiving road section information broadcasted by a plurality of roadside devices on the road section where the vehicle is located.
[0085] The road section information includes the road section images collected by each roadside device, and / or the road section information includes the road line information, road centerline information, road boundary line information, road sign information, traffic signal information, etc. on the road section.
[0086] In an embodiment of the present application, when a vehicle is started and travels to a certain road, a plurality of roadside devices are arranged on the section of the road where the road is located, and each roadside device corresponds to a different test section. Each roadside device collects the section information on the corresponding test section in real time or at intervals of a preset time period. Then, in one scenario, when each roadside device collects the section information in real time or at intervals of a preset time period, it broadcasts it immediately. Based on this, when a vehicle travels to the test section corresponding to these roadside devices, the on-board device thereon can receive the section information broadcasted by each roadside device on the section where the vehicle is located through the wireless communication module; in another scenario, each roadside device collects the section information in real time or at intervals of a preset time period. When the road section information is collected at preset time intervals, it can be stored in each roadside device. When a vehicle travels to the test section corresponding to these roadside devices, the on-board device thereon can send a road section information acquisition request to each roadside device on the section where the vehicle is located through the wireless communication module. When each roadside device receives the road section information acquisition request, it can obtain the stored road section information of the corresponding section from its own hardware memory, and broadcast the road section information of each section. Based on this, when a vehicle travels to the test section corresponding to these roadside devices, the on-board device thereon can receive the road section information broadcasted by each roadside device on the section where the vehicle is located through the wireless communication module.
[0087] It should be noted that the road section information stored on the roadside device can be updated after a preset time period. Specifically, the roadside device can collect the road section information of the corresponding road section every preset time period and update the originally stored road section information; or, when the roadside device collects the road section information of the corresponding road section, it can filter the road section information of the corresponding road section and filter out the road section information that meets the requirements, where meeting the requirements means that the road line, road center line, road sign, traffic signal, etc. on the road section corresponding to the road section information are accurate, or the quality of the road image meets the requirements, that is, the road line information, road center line information, road boundary line information, road sign information, and traffic signal information contained in the filtered road section information can accurately describe or indicate the road line, road center line, road boundary line, road sign, traffic signal, etc. The road image is clear, and the road line information, road center line information, road boundary line information, road sign information, and traffic signal information can be extracted from the road image.
[0088] S203, constructing a target map based on the surrounding environment image and road section information.
[0089] In an embodiment of the present application, when the vehicle-mounted device receives the road section information sent by the roadside device, the processing module thereon can call the trained feature extraction model to perform feature extraction on the surrounding environment image to obtain the environmental map features of the surrounding environment image; optionally, when the processing module receives the road section information sent by the roadside device, it can call the trained element extraction model to perform feature extraction on the road section information to obtain the roadside map elements.
[0090] Optionally, when the vehicle-mounted device receives the road section information sent by the roadside device, it can also obtain the vehicle's location information and send the location information to the roadside device. When the roadside device receives the location information, it can determine the specific location of the vehicle on the road section. Alternatively, the roadside device can also detect the location information of vehicles passing through the corresponding road section to determine the specific location of the vehicle on the road section, and then obtain the feature extraction model associated with the specific location of the vehicle, and the element extraction model associated with the road section information. Here, the feature extraction model associated with the location information may include a feature extraction model composed of a feature extraction network, a deep branch network, a feature conversion network, and a content semantic branch network. Please refer to the following Figure 3 The method described in the embodiment; the element extraction model associated with the road section information includes an element extraction model composed of a line extraction network, a sign extraction network, and a signal light extraction network, which can be seen below Figure 4 The method described in the embodiment.
[0091] When the on-board device obtains environmental map features and roadside map elements, the on-board device can input the environmental map features and roadside map elements into a map decoder to construct a target map; when the on-board device obtains environmental map features and roadside map elements, it can fuse the environmental map features and roadside map elements, and construct a map based on the fused features to obtain a target map; optionally, the on-board device converts both the environmental map features and roadside map elements into feature vectors, and constructs a map based on the feature vectors to obtain a target map.
[0092] Optionally, when the vehicle-mounted device receives the road section information broadcasted by each roadside device, since the road section where the vehicle is located may include multiple roadside devices, the vehicle-mounted device may receive the road section information of multiple roadside devices, among which there is unnecessary road section information, that is, the unnecessary road section information refers to the road section information of the roadside device on the road section far away from the vehicle. Based on this, when the vehicle-mounted device receives the road section information broadcasted by each roadside device, the vehicle-mounted device may also receive the location information sent by the vehicle, or detect the location of the vehicle, and may filter the multiple road section information according to the location of the vehicle, filter out the road section information on the road section closest to the location of the vehicle, and then later construct the target map based on the filtered road section information and the surrounding environment image.
[0093] Optionally, when the vehicle-mounted device receives the road section information sent by the roadside device, the processing module thereon can also call the trained image enhancement model, and use the image enhancement model to perform image feature enhancement processing on the surrounding environment image according to the road section information to obtain an enhanced surrounding environment image, and then the processing module calls the trained feature extraction model to extract features from the enhanced surrounding environment image to obtain environmental map features, and finally the vehicle-mounted device can construct a target map based on the environmental map features. Optionally, when the vehicle-mounted device obtains the surrounding environment image in real time, it can send the surrounding environment image to the roadside device in real time, so that the roadside device can store the surrounding environment image as a road section image of the road section for use by the vehicle-mounted devices of other vehicles.
[0094] S204, displaying the target map on the display interface of the vehicle-mounted device.
[0095] In the embodiment of the present application, when the processing module on the vehicle-mounted device completes the construction of the target map, if the display screen on the vehicle-mounted device is not turned on or is in a screen-off state, the processing module can automatically turn on the display interface of the display screen or activate the display interface of the display screen, and then display the target map on the display interface (see Figure 2A Optionally, when the processing module completes the construction of the target map, it may also display the target map on the display interface in response to the user's triggering operation of the corresponding map display control or navigation control on the display interface (see Figure 2A The display interface shown, in which a map of the vehicle's current location is constructed online, wherein any M in the display interface can be a map display control or a navigation control).
[0096] In the above-mentioned map construction method, map construction system, device, equipment, storage medium and program product, the vehicle-mounted equipment obtains the surrounding environment image collected by the vehicle-mounted sensor on the vehicle, and receives the road section information broadcast by the roadside equipment on the road section where the vehicle is located, and constructs the target map according to the surrounding environment image and the road section information, and displays the target map on the display interface on the vehicle-mounted equipment. When the vehicle-mounted equipment in the above-mentioned method constructs the map online, it can obtain the road section information from the roadside equipment, and construct the target map by combining the road section information and the surrounding environment image collected by the vehicle-mounted sensor, so that the quality of the target map is not affected by the environment where the vehicle is located, such as weather or field of view obstruction, etc., thereby improving the accuracy of constructing the target map. In addition, the fusion of the road section information and the surrounding environment image can be realized to improve the refinement of the target map, so that the constructed target map can clearly reflect the road sideline, road centerline and other detailed information in the road, thereby improving the accuracy of map construction.
[0097] In one embodiment, Figure 3As shown, a method for constructing a target map is provided, that is, the above S203 "constructing a target map according to the surrounding environment image and the road section information", including:
[0098] S301, input the surrounding environment image into the feature extraction model for feature encoding to obtain the environment map features.
[0099] Among them, the environmental map features include environmental depth features, environmental bird's-eye view features, and environmental content features. The feature extraction model is a neural network model that has been pre-trained based on images, and the feature extraction model is used to perform multi-dimensional feature extraction on the input image to obtain the features of the image in the depth dimension, bird's-eye view type dimension, and content dimension. Optionally, the feature extraction model can be a neural network model pre-stored on the processing module, or a neural network model sent to the vehicle-mounted device by the roadside device.
[0100] In an embodiment of the present application, when the processing module on the vehicle-mounted device obtains the surrounding environment image, it can call a pre-trained feature extraction model, or obtain the feature extraction model from the roadside device, or receive the feature extraction model sent by the roadside device, and then input the surrounding environment image into the feature extraction model for feature encoding to obtain environmental map features including environmental depth features, environmental bird's-eye view features, and environmental content features.
[0101] S302: Input the road section information into an element extraction model to extract map elements and obtain roadside map elements.
[0102] Among them, the roadside map elements include line elements, road sign elements, and traffic signal elements; the element extraction model is a neural network model pre-trained based on information samples, and the element extraction model is used to extract multi-dimensional elements from the input road section information to obtain line elements, road sign elements, traffic signal elements, etc., wherein the line elements may include a series of map-related line elements such as road line elements, road centerline elements, and road boundary line elements. Optionally, the element extraction model may be a neural network model pre-stored on the processing module, or a neural network model sent by the roadside device to the vehicle-mounted device.
[0103] In an embodiment of the present application, when the processing module on the vehicle-mounted device obtains road section information or obtains filtered road section information, it can call a pre-trained element extraction model, or obtain an element extraction model from a roadside device, or receive an element extraction model sent by a roadside device, and then input the road section information into the element extraction model for map element encoding to obtain at least one map element including line elements, road sign elements, traffic signal elements, etc.
[0104] S303: Input the environmental map features and roadside map elements into a map decoder to construct a target map.
[0105] The map decoder is used to output vectorized map elements, and then construct a map based on the vectorized map elements. The map decoder is a pre-trained image decoder, which can be trained together with the image encoder. The map decoder can be pre-stored in the vehicle-mounted device or sent to the vehicle-mounted device by the roadside device.
[0106] In the embodiment of the present application, when the vehicle-mounted device obtains the environmental map features and roadside map elements based on the aforementioned steps, the environmental map features and roadside map elements can be directly input into a pre-trained map decoder for map construction to obtain a target map. Optionally, when the vehicle-mounted device obtains the environmental map features and roadside map elements, the environmental map features and roadside map elements can be fused, and a map can be constructed based on the fused features to obtain a target map; optionally, the vehicle-mounted device converts both the environmental map features and roadside map elements into feature vectors, and a map is constructed based on the feature vectors to obtain a target map.
[0107] Optionally, since the environmental map features include features of three dimensions: environmental depth features, environmental bird's-eye view features, and environmental content features, and the environmental bird's-eye view features may be inaccurate due to weather or other factors, when the on-board equipment fuses the environmental map features with the roadside map elements, the roadside map elements can be fused with the environmental bird's-eye view features to enhance the environmental bird's-eye view features, and then build the target map based on the fused features, environmental depth features, environmental content features, and road section information.
[0108] The method described in the embodiment of the present application obtains multi-dimensional image features through multi-dimensional feature extraction, and constructs a target map based on the fused multi-dimensional features, which can improve the accuracy of map construction and enable detailed information to be fully displayed in the target map.
[0109] In one embodiment, Figure 4 As shown, a feature extraction model is provided, which includes a feature extraction network 401, a deep branch network 402, a feature conversion network 403, and a content semantic branch network 404; the output of the feature extraction network 401 is connected to the input of the deep branch network 402, the feature conversion network 403, and the content semantic branch network 404 respectively; based on this, a method for obtaining environmental map features is provided, that is, the above S301 "inputting the surrounding environment image into the feature extraction model for feature encoding to obtain environmental map features", such as Figure 5 As shown, including:
[0110] S501, input the surrounding environment image into the feature extraction network to extract global features, and obtain the global features of the surrounding environment image.
[0111] The feature extraction network is used for global feature extraction, and specifically can be a 2D feature extraction network backbone, and the feature extraction network is a pre-trained neural network.
[0112] In an embodiment of the present application, when the vehicle-mounted device inputs the surrounding environment image into the feature extraction model for feature extraction, the surrounding environment image can be first input into the feature extraction network for global feature extraction to obtain the global features of the surrounding environment map.
[0113] S502, inputting the global features and the surrounding environment image into a deep branch network to extract depth features, and obtaining environmental depth features of the surrounding environment image.
[0114] Among them, the deep branch network is used for deep feature extraction and is a pre-trained neural network.
[0115] In an embodiment of the present application, when the vehicle-mounted device obtains the global features based on the aforementioned steps, the global features and the surrounding environment image can be input into the deep branch network for deep feature extraction to obtain the environmental depth features of the surrounding environment image.
[0116] S503, inputting the global features and the surrounding environment image into a feature conversion network for feature conversion to obtain the environment bird's-eye view features of the surrounding environment image.
[0117] The feature conversion network is used for feature conversion, specifically for converting features into bird's eye view features, and is a pre-trained neural network. For example, the feature conversion network may be a 2D to BEV network, which can convert 2D features into bird's eye view features (Bird's eye view, BEV).
[0118] In an embodiment of the present application, when the vehicle-mounted device obtains the global features based on the aforementioned steps, the global features and the surrounding environment image can be input into the feature conversion network for feature conversion to obtain the environmental bird's-eye view features of the surrounding environment image.
[0119] S504, inputting the global features and the surrounding environment image into the content semantic branch network to extract semantic features, and obtaining the environmental content features of the surrounding environment image.
[0120] Among them, the content semantic branch network is used for semantic feature extraction and is a pre-trained neural network.
[0121] In an embodiment of the present application, when the vehicle-mounted device obtains global features based on the aforementioned steps, the global features and the surrounding environment image can be input into a deep branch network for semantic feature extraction to obtain environmental content features of the surrounding environment image.
[0122] The method described in the above embodiment obtains multi-dimensional image features through multi-dimensional feature extraction such as global, depth, image type (bird's eye view), semantics, etc., and constructs a target map based on the fused multi-dimensional features, which can improve the accuracy of map construction and enable the target map to fully display detailed information.
[0123] In one embodiment, Figure 6 As shown, an element extraction model is provided, which includes a line extraction network 601, a sign extraction network 602, and a signal light extraction network 603; the line extraction network 601, the sign extraction network 602, and the signal light extraction network 603 are connected in parallel and are all used to process the input information; based on this, a method for obtaining road section information is provided, that is, the above S302 "input the road section information into the element extraction model to extract map elements, and obtain roadside map elements", such as Figure 7 As shown, including:
[0124] S701, inputting the road section information into the line extraction network to extract line elements and obtain line elements.
[0125] The line extraction network is used to extract line elements, which is a pre-trained neural network. Line elements include road line elements, road center line elements, road boundary line elements and other line elements related to the map.
[0126] In an embodiment of the present application, when the vehicle-mounted device obtains the road section information based on the above steps, the road section information can be input into the line extraction network to extract line elements to obtain a series of map-related line type elements including road line elements, road centerline elements, road boundary line elements, etc.
[0127] Optionally, the line extraction network in the embodiment of the present application may also include multiple branch networks, and each branch network corresponds to its own type of line type. For example, the line extraction network includes a road line extraction branch network, a road centerline extraction branch network, and a road boundary line extraction branch network, and these three branch networks are branch networks connected in parallel, all of which are used to extract corresponding elements of the input road section information. Specifically, the road line extraction branch network is used to extract road line elements to obtain road line elements; the road centerline extraction branch network is used to extract road centerline elements to obtain road centerline elements; and the road boundary line extraction branch network is used to extract road boundary line elements to obtain road boundary line elements.
[0128] S702: Input the road section information into the sign extraction network to extract sign elements, and obtain road sign sign elements.
[0129] The sign extraction network is used to extract sign elements, and is a pre-trained neural network. Road sign elements include various types of road signs, such as through signs, left turn signs, right turn signs, etc.
[0130] In the embodiment of the present application, when the vehicle-mounted device obtains the road section information based on the above steps, the road section information can be input into the identification extraction network to extract identification elements to obtain road indication identification elements.
[0131] S703: Input the road section information into the traffic light extraction network to extract signal elements and obtain traffic signal elements.
[0132] The traffic light extraction network is used to extract signal elements and is a pre-trained neural network. Traffic signal elements include various types of signal light elements, such as traffic light elements, turn signal elements, etc.
[0133] In the embodiment of the present application, when the vehicle-mounted device obtains the road section information based on the aforementioned steps, the road section information can be input into the traffic light extraction network to extract signal elements to obtain traffic signal elements.
[0134] The method described in the above embodiment can obtain detailed information of multiple maps by extracting various types of map elements from road section information, which is convenient for later enhancing the surrounding environment image based on these map elements and then constructing the map, or fusing the environmental map features corresponding to the surrounding environment image to construct the map, thereby improving the accuracy of the constructed map.
[0135] In one embodiment, a method for constructing a target map is provided, namely Figure 2 In the embodiment, S203 "inputs the environmental map features and the roadside map elements into the map decoder to construct a target map", such as Figure 8 As shown, the method includes:
[0136] S801, fuse the environmental map features and the roadside map elements to obtain fused features.
[0137] Optionally, the fused features may include one type of fused features; optionally, the fused features may also include multiple types of fused features, for example, the fused features include fused depth features, fused bird's-eye view features and fused content features; for another example, the fused features include fused bird's-eye view features, environmental depth features and environmental content features; for another example, the fused features include bird's-eye view features, fused environmental depth features and environmental content features; the fused features include bird's-eye view features, environmental depth features and fused environmental content features.
[0138] In the embodiment of the present application, when the processing module on the vehicle-mounted device obtains the environmental map features based on the above steps, the processing module can use the corresponding image fusion algorithm to fuse the environmental map features and the roadside map elements to obtain the fused features; optionally, the processing module can first enhance the environmental map features and the roadside map elements to obtain the processed map features, and finally use the corresponding image fusion algorithm to fuse the processed map features to obtain the fused features. Specifically, since the environmental map features are extracted from the surrounding environment image of the current position of the vehicle, and the roadside map elements are features extracted from the image of the road section at the current position of the vehicle or the road section information, and the surrounding environment image collected by the vehicle and the image of the road section information may not cover the same map area, based on this, in order to improve the accuracy of map construction or refine the detailed information in the map, such as road boundary lines, road lines, road center lines, etc., and also because the images collected by the vehicle-mounted sensors may be limited by the environment in which the vehicle is located, resulting in the problem of missing some necessary features or detailed features in the surrounding environment map, such as the missing of road line features due to rainy days, and the missing of intersection road features due to visual occlusion. In this scenario, the processing module can use roadside map elements to supplement or enhance environmental map features, which can improve the accuracy of vehicle map construction in harsh environments, or highlight detailed information on the constructed map so that the target map can display more information.
[0139] Optionally, the environmental map features include environmental depth features, environmental bird's-eye view features, and environmental content features; based on this, the on-board device can fuse the roadside map elements with these three types of features respectively, for example, fuse the roadside map elements with the environmental depth features to obtain fused depth features, fuse the roadside map elements with the environmental bird's-eye view features to obtain fused bird's-eye view features, and fuse the roadside map elements with the environmental content features to obtain fused content features; ultimately, a fused feature including fused depth features, fused bird's-eye view features, and fused content features is obtained. Optionally, the on-board device can also fuse the roadside map elements with any one of the environmental depth features, environmental bird's-eye view features, and environmental content features, ultimately obtaining a fused feature including bird's-eye view features, fused environmental depth features, and environmental content features, or obtaining a fused feature including bird's-eye view features, environmental depth features, and fused environmental content features. Optionally, since the environmental bird's-eye view features are easily affected by the environment at the vehicle's location and become inaccurate, the on-board equipment can also fuse the roadside map elements with the environmental bird's-eye view features to obtain a fused fused bird's-eye view feature, and finally obtain a fused feature including the fused bird's-eye view feature, the environmental depth feature and the environmental content feature.
[0140] Optionally, since the target map is a vectorized map, when fusing the environmental map features and the roadside map elements, the environmental map features can be specifically converted into a first vector and the roadside map elements can be converted into a second vector first; and the first vector and the second vector are fused to obtain fused features, so as to subsequently construct a vectorized map based on the fused features.
[0141] S802, input the fused features into a map decoder to construct a target map.
[0142] The map decoder is used to output vectorized map elements, and then construct a map based on the vectorized map elements. The map decoder is a pre-trained image decoder, which can be trained together with the image encoder. The map decoder can be pre-stored in the vehicle-mounted device or sent to the vehicle-mounted device by the roadside device.
[0143] In the embodiment of the present application, when the vehicle-mounted device obtains the fused features based on the above steps, the fused features can be directly input into the pre-trained map decoder for map construction to obtain the target map. Based on the above description of S801, the fused features can include fused depth features, fused bird's-eye view features and fused content features. Based on this, the vehicle-mounted device can input the fused depth features, fused bird's-eye view features and fused content features into the map decoder to construct the target map; optionally, the vehicle-mounted device can also input the fused depth features, fused bird's-eye view features and fused content features into the map decoder in sequence to construct a depth map corresponding to the fused depth features, a bird's-eye view map corresponding to the fused bird's-eye view features and a content map corresponding to the fused content features. Finally, the depth map, the bird's-eye view map and the content map are fused to obtain the target map.
[0144] The method described in the embodiment of the present application obtains multi-dimensional image features through multi-dimensional feature extraction, and constructs a target map based on the fused multi-dimensional features, which can improve the accuracy of map construction and enable detailed information to be fully displayed in the target map.
[0145] In one embodiment, the vehicle-mounted device can also trigger the display of the intersection map. Based on this, Figure 2 The method described in the embodiment, as shown in 9, further includes:
[0146] S901, obtaining the vehicle's location information and navigation path.
[0147] The location information includes the current location information of the vehicle (e.g., location name, location description) or the current longitude and latitude information of the vehicle (e.g., longitude and latitude coordinates); the navigation path may refer to the path between the destination address of the vehicle and the starting address of the vehicle, which may be determined by the user triggering the navigation control on the display screen of the vehicle-mounted device or inputting navigation information. The navigation path includes the location information and other related description information of multiple intersections where the vehicle travels, that is, multiple intersections between the destination address and the starting address of the vehicle.
[0148] In the embodiment of the present application, a positioning module is provided on the vehicle-mounted device. When the vehicle starts to travel, the positioning module can locate the position of the vehicle and obtain the position information of the vehicle; accordingly, the vehicle-mounted device can allow the user to perform navigation operations on the vehicle-mounted device, specifically, the corresponding navigation controls can be triggered or navigation information can be input on the display screen of the vehicle-mounted device. For example, the user can input the destination address on the display screen, and the vehicle-mounted device can locate the starting address of the travel by itself, or the user can input the starting address of the travel on the display screen, and the vehicle-mounted device can obtain the path between the destination address of the travel and the starting address of the vehicle.
[0149] S902, determining target intersection information according to the vehicle's location information, the navigation path, and the road section information broadcast by the roadside equipment.
[0150] Among them, the target intersection information includes map information on the next intersection map corresponding to the target intersection information. For example, the target intersection information includes road line elements, road centerline elements, road boundary line elements, traffic light elements and other map-related elements on the next intersection map.
[0151] In the embodiment of the present application, since the road section information broadcasted by the roadside equipment includes the road section information of multiple roadside equipment, and the road sections corresponding to each roadside equipment are different geographical areas, including road sections that are relatively close to the current position of the vehicle and road sections that are relatively far from the current position of the vehicle, the road section information corresponding to the position of the next intersection of the vehicle is included. If it is necessary to obtain the map of the next intersection next, it is necessary to first extract the road section information corresponding to the road section belonging to the next intersection from the road section information broadcasted by each roadside equipment. Based on this, when the vehicle-mounted equipment obtains the vehicle's position information and navigation path based on the above steps, it can first determine the position of the next intersection based on these two information, and extract the road section information corresponding to the road section of the next intersection from the road section information broadcasted by multiple roadside equipment based on the position of the next intersection, that is, the target intersection information. When the vehicle-mounted equipment determines the position of the next intersection based on the vehicle's position information and the navigation path, it can determine the position of the vehicle based on the vehicle's position information, and then find the position of the next intersection closest to the vehicle's position based on the navigation path, and then determine the position of the next intersection.
[0152] S903: construct a next intersection map according to the target intersection information.
[0153] Among them, the next intersection map is an intersection map of the next intersection where the vehicle is expected to travel;
[0154] In an embodiment of the present application, when the vehicle-mounted device obtains the target intersection information based on the aforementioned steps, the corresponding map decoder can be used to construct the next intersection map based on the target intersection information; or, a pixel extraction model can be used to extract map elements based on the target intersection information to obtain intersection map elements, and then the intersection map elements are input into the map decoder to construct the next intersection map.
[0155] S904, displaying the next intersection map on the display interface.
[0156] In the embodiment of the present application, when the processing module on the vehicle-mounted device completes the construction of the next intersection map, if the display screen on the vehicle-mounted device displays the target map at this time, the processing module can automatically display the next intersection map on the target map, for example, displaying the next intersection map in a small window, so that the user can view the situation of the next intersection that the vehicle is expected to travel through the next intersection map, so as to predict the situation of the next intersection in advance, especially the intersection that is more likely to be congested, and can be checked in advance, for example, see Fig. 8A The interface diagram shown in the figure shows that the intersection map is displayed on the map display interface of the vehicle-mounted device in the form of a small window, and the window is provided with a close control and a shrink and hide display control; it can also be displayed simultaneously with the target map based on other methods; optionally, when the processing module on the vehicle-mounted device completes the construction of the next intersection map, if the target map is displayed on the display screen of the vehicle-mounted device at this time, the current display interface can also switch to the next intersection map display, so that the user can fully and clearly view the situation of the next intersection on the display screen. Optionally, if the target map is not displayed on the display screen of the vehicle-mounted device at this time, the processing module can automatically trigger the display screen to display the next intersection map, so that the user can preview the situation of the next intersection on the display screen.
[0157] According to the method described in the above embodiment, when the on-board equipment constructs a map of the vehicle's surrounding environment online, it can also construct an intersection map of the next intersection where the vehicle is expected to travel, so that the user can predict the situation at the next intersection. The on-board equipment constructs the next intersection map based on the road section information broadcast by the roadside equipment. The quality of the next intersection map is not affected by the vehicle's environment, such as weather or field of view obstruction, thereby improving the accuracy of constructing the intersection map and thus improving the safety of vehicle driving.
[0158] In one embodiment, a method for triggering target map display is provided, such as Fig.10 As shown, the method includes:
[0159] S1001, when the vehicle is started, determining whether the vehicle-mounted equipment meets the preset map building conditions.
[0160] Among them, the preset map construction conditions include any of the following: the vehicle-mounted device responds to the user's triggering operation of the map generation control or the navigation control on the display interface of the vehicle-mounted device; the vehicle-mounted device is powered on and started; the vehicle-mounted device detects that the vehicle is driving to the target road; the target road includes any of a curve, a ramp, an intersection, a narrow road, and an accident road.
[0161] S1002, when the vehicle-mounted device meets the preset map building conditions, execute the step of obtaining the surrounding environment image collected by the vehicle-mounted sensor on the vehicle.
[0162] The embodiment of the present application relates to a method for triggering the display of a target map on a display screen of a vehicle-mounted device. Specifically, when the vehicle is started, the first scenario is to determine whether the vehicle-mounted device is powered on. If the vehicle-mounted device is powered on, the vehicle-mounted device can execute the aforementioned Figure 2-Figure 9 The method described in the embodiment constructs a target map and / or a map of the next intersection, and displays the target map and / or the map of the next intersection on the display screen of the vehicle-mounted device. The second scenario is that the vehicle-mounted device has been started, that is, the display screen on the vehicle-mounted device has been turned on, and the user can trigger the map generation control or the navigation control on the display interface of the display screen. At this time, the processing module can determine whether the user has performed a triggering operation on the map generation control or the navigation control on the display interface of the vehicle-mounted device, that is, whether it responds to the user's triggering operation on the map generation control or the navigation control on the display interface of the vehicle-mounted device. If the user triggers the map generation control or the navigation control, the vehicle-mounted device can immediately execute the aforementioned Figure 2-Figure 9 The method described in the embodiment constructs the target map and / or the next intersection map, and displays the target map and / or the next intersection map on the display screen of the vehicle-mounted device. The third scenario is that the corresponding sensor or detection device on the vehicle can detect the road state of the vehicle and send the road state of the vehicle to the processing module of the vehicle-mounted device. When the vehicle-mounted device detects that the vehicle has traveled to the target road according to the road state of the vehicle, the vehicle-mounted device can immediately execute the above-mentioned Figure 2-Figure 9 The method described in the embodiment constructs a target map and / or a next intersection map.
[0163] Based on the methods described in all the above embodiments, the present application also provides a map construction system, such as Fig.11As shown, the system includes: a vehicle-mounted sensor 1101, a vehicle-mounted device 1102, and a roadside device 1103; the vehicle-mounted sensor 1101 and the vehicle-mounted device 1102 are arranged on the vehicle; the vehicle-mounted device 1102 is also provided with a wireless communication module, a processing module and a positioning module, the processing module is respectively connected to the vehicle-mounted sensor 1101, the processing module is also connected to the positioning module, and the processing module is also connected to the roadside device 1103 through the wireless communication module. The functions of the vehicle-mounted sensor 1101, the vehicle-mounted device 1102, and the roadside device 1103 are all explained in the methods described in the aforementioned embodiments. Please refer to the aforementioned description for details and will not be repeated here.
[0164] Based on the methods described in all the above embodiments, the present application embodiment also provides a map construction method, such as Fig.12 As shown, the method includes:
[0165] S1201, obtaining a surrounding environment image collected by a vehicle-mounted sensor, and receiving road section information broadcasted by multiple roadside devices on the road section where the vehicle is located.
[0166] S1202, input the surrounding environment image into the feature extraction model for feature encoding to obtain environment map features.
[0167] Among them, the environmental map features include environmental depth features, environmental bird's-eye view features, and environmental content features.
[0168] S1203: Input the road section information into an element extraction model to extract map elements and obtain roadside map elements.
[0169] Among them, roadside map elements include line elements, road sign elements, and traffic signal elements.
[0170] S1204: Fusing the environmental map features and the roadside map elements to obtain fused features, and inputting the fused features into a map decoder to construct a target map.
[0171] S1205, obtaining the vehicle's location information and navigation path.
[0172] S1206, determining target intersection information based on the vehicle's location information, navigation path, and road section information broadcast by roadside equipment.
[0173] S1207, constructing a next intersection map according to the target intersection information, and displaying the next intersection map on the display interface.
[0174] Each step in the above embodiment has been described above. Please refer to the above description for details and will not be repeated here.
[0175] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0176] Based on the same inventive concept, the embodiment of the present application also provides a map construction device for implementing the map construction method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more map construction device embodiments provided below can refer to the limitations on the map construction method above, and will not be repeated here.
[0177] In one embodiment, Fig.13 As shown, a map construction device is provided, comprising:
[0178] The acquisition module 1301 is used to acquire the surrounding environment image collected by the vehicle-mounted sensor;
[0179] The receiving module 1302 is used to receive the road section information broadcasted by the roadside equipment of the road section where the vehicle is located;
[0180] A construction module 1303, configured to construct a target map according to the surrounding environment image and the road section information;
[0181] The display module 1304 is used to display the target map on the display interface of the vehicle-mounted device.
[0182] In one embodiment, the building block 1303 includes:
[0183] A first feature extraction unit is used to input the surrounding environment image into a feature extraction model for feature encoding to obtain environment map features; the environment map features include environment depth features, environment bird's-eye view features, and environment content features;
[0184] A second feature extraction unit is used to input the road section information into an element extraction model to extract map elements to obtain roadside map elements; the roadside map elements include line elements, road sign elements, and traffic signal elements;
[0185] A construction unit is used to input the environmental map features and the roadside map elements into a map decoder to construct the target map.
[0186] In one embodiment, the first feature extraction unit includes:
[0187] A first extraction subunit, used for inputting the surrounding environment image into the feature extraction network to extract global features, so as to obtain global features of the surrounding environment image;
[0188] A second extraction subunit is used to input the global feature and the surrounding environment image into the deep branch network to extract the deep feature, so as to obtain the environmental deep feature of the surrounding environment image;
[0189] A third extraction subunit is used to input the global features and the surrounding environment image into the feature conversion network for feature conversion to obtain an environmental bird's-eye view feature of the surrounding environment image;
[0190] The fourth extraction subunit is used to input the global features and the surrounding environment image into the content semantic branch network to extract semantic features, so as to obtain the environmental content features of the surrounding environment image.
[0191] In one embodiment, the second feature extraction unit includes:
[0192] a fifth extraction subunit, configured to input the road section information into the line extraction network to extract line elements and obtain the line elements;
[0193] A sixth extraction subunit, configured to input the road section information into the sign extraction network to extract sign elements, and obtain the road indication sign elements;
[0194] The seventh extraction subunit is used to input the road section information into the traffic light extraction network to extract signal elements to obtain the traffic signal elements.
[0195] In one embodiment, the above-mentioned building block comprises:
[0196] A fusion subunit, used for fusing the environment map feature with the roadside map element to obtain a fusion feature;
[0197] The construction subunit is used to input the fusion features into a map decoder to construct the target map.
[0198] In one embodiment, the above-mentioned building block comprises:
[0199] The above-mentioned fusion subunit is specifically used to convert the environmental map feature into a first vector, and convert the roadside map element into a second vector; and fuse the first vector and the second vector to obtain the fused feature.
[0200] In one embodiment, the map construction device further includes:
[0201] A location and path acquisition module, used to acquire the location information and navigation path of the vehicle;
[0202] A first determination module, configured to determine target intersection information according to the location information of the vehicle, the navigation path, and the road section information broadcast by the roadside equipment;
[0203] A road intersection construction module is used to construct a next road intersection map according to the target road intersection information;
[0204] The intersection map display module is used to display the next intersection map on the display interface.
[0205] In one embodiment, the map construction device further includes:
[0206] The second determination module is used to determine whether the vehicle-mounted device meets the preset map construction conditions when the vehicle is started; wherein the preset map construction conditions include any one of the following: the vehicle-mounted device responds to the user's triggering operation of the map generation control or the navigation control on the display interface of the vehicle-mounted device; the vehicle-mounted device is powered on and started; the vehicle-mounted device detects that the vehicle has traveled to a target road; the target road includes any one of a curve, a ramp, an intersection, a narrow road, and an accident road.
[0207] The execution module is used to execute the step of acquiring the surrounding environment image collected by the vehicle-mounted sensor on the vehicle when the vehicle-mounted device meets the preset map building condition.
[0208] Each module in the above-mentioned map construction device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.
[0209] In one embodiment, a vehicle is provided. Fig.14As shown, the vehicle includes an on-board device, and the on-board device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the on-board device is used to provide computing and control capabilities. The memory of the on-board device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the on-board device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a map construction method is implemented. The display screen of the on-board device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the on-board device can be a touch layer covered on the display screen, or a button, a trackball or a touch pad set on the housing of the on-board device, or an external keyboard, touch pad or mouse, etc.
[0210] In one embodiment, a computer device is provided. The computer device may be a vehicle-mounted terminal, and its internal structure diagram may be as follows: Fig.15 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a map construction method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse, etc.
[0211] Those skilled in the art will understand that Fig.15 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0212] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:
[0213] Obtaining surrounding environment images collected by on-board sensors on the vehicle;
[0214] Receiving road section information broadcasted by a plurality of roadside devices on the road section where the vehicle is located;
[0215] Constructing a target map according to the surrounding environment image and the road section information;
[0216] The target map is displayed on a display interface on the vehicle-mounted device.
[0217] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0218] Inputting the surrounding environment image into a feature extraction model for feature encoding to obtain environment map features; the environment map features include environment depth features, environment bird's-eye view features, and environment content features;
[0219] Inputting the road section information into an element extraction model to extract map elements to obtain roadside map elements; the roadside map elements include line elements, road sign elements, and traffic signal elements;
[0220] The environmental map features and the roadside map elements are input into a map decoder to construct the target map.
[0221] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0222] Inputting the surrounding environment image into the feature extraction network to extract global features, so as to obtain global features of the surrounding environment image;
[0223] Inputting the global features and the surrounding environment image into the deep branch network to extract deep features, thereby obtaining environmental deep features of the surrounding environment image;
[0224] Inputting the global features and the surrounding environment image into the feature conversion network for feature conversion to obtain the environmental bird's-eye view features of the surrounding environment image;
[0225] The global features and the surrounding environment image are input into the content semantic branch network to extract semantic features, so as to obtain the environmental content features of the surrounding environment image.
[0226] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0227] Inputting the road section information into the line extraction network to extract line elements to obtain the line elements;
[0228] Inputting the road section information into the sign extraction network to extract sign elements to obtain the road indication sign elements;
[0229] The road section information is input into the traffic light extraction network to extract signal elements to obtain the traffic signal elements.
[0230] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0231] Fusing the environment map features with the roadside map elements to obtain fused features;
[0232] The fused features are input into a map decoder to construct the target map.
[0233] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0234] converting the environmental map feature into a first vector, and converting the roadside map element into a second vector;
[0235] The first vector and the second vector are fused to obtain the fused feature.
[0236] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0237] Obtaining the location information and navigation path of the vehicle;
[0238] Determine target intersection information according to the location information of the vehicle, the navigation path and the road section information broadcast by the roadside equipment;
[0239] Constructing a next intersection map according to the target intersection information;
[0240] The next intersection map is displayed on the display interface.
[0241] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0242] When the vehicle is started, determining whether the vehicle-mounted device meets a preset map building condition;
[0243] When the vehicle-mounted device meets the preset map building condition, the step of acquiring the surrounding environment image collected by the vehicle-mounted sensor on the vehicle is performed;
[0244] The preset map construction condition includes any of the following:
[0245] The in-vehicle device responds to a triggering operation of a map generation control or a navigation control on a display interface of the in-vehicle device by a user;
[0246] The vehicle-mounted device is powered on and started;
[0247] The vehicle-mounted device detects that the vehicle travels to a target road; the target road includes any one of a curve, a ramp, an intersection, a narrow road, and an accident road.
[0248] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
[0249] Obtaining surrounding environment images collected by on-board sensors on the vehicle;
[0250] Receiving road section information broadcasted by a plurality of roadside devices on the road section where the vehicle is located;
[0251] Constructing a target map according to the surrounding environment image and the road section information;
[0252] The target map is displayed on a display interface on the vehicle-mounted device.
[0253] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0254] Inputting the surrounding environment image into a feature extraction model for feature encoding to obtain environment map features; the environment map features include environment depth features, environment bird's-eye view features, and environment content features;
[0255] Inputting the road section information into an element extraction model to extract map elements to obtain roadside map elements; the roadside map elements include line elements, road sign elements, and traffic signal elements;
[0256] The environmental map features and the roadside map elements are input into a map decoder to construct the target map.
[0257] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0258] Inputting the surrounding environment image into the feature extraction network to extract global features, so as to obtain global features of the surrounding environment image;
[0259] Inputting the global features and the surrounding environment image into the deep branch network to extract deep features, thereby obtaining environmental deep features of the surrounding environment image;
[0260] Inputting the global features and the surrounding environment image into the feature conversion network for feature conversion to obtain the environmental bird's-eye view features of the surrounding environment image;
[0261] The global features and the surrounding environment image are input into the content semantic branch network to extract semantic features, so as to obtain the environmental content features of the surrounding environment image.
[0262] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0263] Inputting the road section information into the line extraction network to extract line elements to obtain the line elements;
[0264] Inputting the road section information into the sign extraction network to extract sign elements to obtain the road indication sign elements;
[0265] The road section information is input into the traffic light extraction network to extract signal elements to obtain the traffic signal elements.
[0266] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0267] Fusing the environment map features with the roadside map elements to obtain fused features;
[0268] The fused features are input into a map decoder to construct the target map.
[0269] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0270] converting the environmental map feature into a first vector, and converting the roadside map element into a second vector;
[0271] The first vector and the second vector are fused to obtain the fused feature.
[0272] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0273] Obtaining the location information and navigation path of the vehicle;
[0274] Determine target intersection information according to the location information of the vehicle, the navigation path and the road section information broadcast by the roadside equipment;
[0275] Constructing a next intersection map according to the target intersection information;
[0276] The next intersection map is displayed on the display interface.
[0277] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0278] When the vehicle is started, determining whether the vehicle-mounted device meets a preset map building condition;
[0279] When the vehicle-mounted device meets the preset map building condition, the step of acquiring the surrounding environment image collected by the vehicle-mounted sensor on the vehicle is performed;
[0280] The preset map construction condition includes any of the following:
[0281] The in-vehicle device responds to a triggering operation of a map generation control or a navigation control on a display interface of the in-vehicle device by a user;
[0282] The vehicle-mounted device is powered on and started;
[0283] The vehicle-mounted device detects that the vehicle travels to a target road; the target road includes any one of a curve, a ramp, an intersection, a narrow road, and an accident road.
[0284] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:
[0285] Obtaining surrounding environment images collected by on-board sensors on the vehicle;
[0286] Receiving road section information broadcasted by a plurality of roadside devices on the road section where the vehicle is located;
[0287] Constructing a target map according to the surrounding environment image and the road section information;
[0288] The target map is displayed on a display interface on the vehicle-mounted device.
[0289] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0290] Inputting the surrounding environment image into a feature extraction model for feature encoding to obtain environment map features; the environment map features include environment depth features, environment bird's-eye view features, and environment content features;
[0291] Inputting the road section information into an element extraction model to extract map elements to obtain roadside map elements; the roadside map elements include line elements, road sign elements, and traffic signal elements;
[0292] The environmental map features and the roadside map elements are input into a map decoder to construct the target map.
[0293] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0294] Inputting the surrounding environment image into the feature extraction network to extract global features, so as to obtain global features of the surrounding environment image;
[0295] Inputting the global features and the surrounding environment image into the deep branch network to extract deep features, thereby obtaining environmental deep features of the surrounding environment image;
[0296] Inputting the global features and the surrounding environment image into the feature conversion network for feature conversion to obtain the environmental bird's-eye view features of the surrounding environment image;
[0297] The global features and the surrounding environment image are input into the content semantic branch network to extract semantic features, so as to obtain the environmental content features of the surrounding environment image.
[0298] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0299] Inputting the road section information into the line extraction network to extract line elements to obtain the line elements;
[0300] Inputting the road section information into the sign extraction network to extract sign elements to obtain the road indication sign elements;
[0301] The road section information is input into the traffic light extraction network to extract signal elements to obtain the traffic signal elements.
[0302] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0303] Fusing the environment map features with the roadside map elements to obtain fused features;
[0304] The fused features are input into a map decoder to construct the target map.
[0305] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0306] converting the environmental map feature into a first vector, and converting the roadside map element into a second vector;
[0307] The first vector and the second vector are fused to obtain the fused feature.
[0308] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0309] Obtaining the location information and navigation path of the vehicle;
[0310] Determine target intersection information according to the location information of the vehicle, the navigation path and the road section information broadcast by the roadside equipment;
[0311] Constructing a next intersection map according to the target intersection information;
[0312] The next intersection map is displayed on the display interface.
[0313] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0314] When the vehicle is started, determining whether the vehicle-mounted device meets a preset map building condition;
[0315] When the vehicle-mounted device meets the preset map building condition, the step of acquiring the surrounding environment image collected by the vehicle-mounted sensor on the vehicle is performed;
[0316] The preset map construction condition includes any of the following:
[0317] The in-vehicle device responds to a triggering operation of a map generation control or a navigation control on a display interface of the in-vehicle device by a user;
[0318] The vehicle-mounted device is powered on and started;
[0319] The vehicle-mounted device detects that the vehicle travels to a target road; the target road includes any one of a curve, a ramp, an intersection, a narrow road, and an accident road.
[0320] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.
[0321] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0322] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A map construction method, characterized in that: The method is applied to a vehicle-mounted device, and the method comprises: Obtaining surrounding environment images collected by on-board sensors on the vehicle; Receiving road section information broadcasted by a plurality of roadside devices on the road section where the vehicle is located; Constructing a target map according to the surrounding environment image and the road section information; The target map is displayed on a display interface on the vehicle-mounted device.
2. The method according to claim 1, characterized in that The step of constructing a target map according to the surrounding environment image and the road section information includes: The surrounding environment image is input into a feature extraction model for feature encoding to obtain environment map features, and the road section information is input into an element extraction model for map element extraction to obtain roadside map elements; the environment map features include environment depth features, environment bird's-eye view features, and environment content features; the roadside map elements include line elements, road sign elements, and traffic signal elements; The environmental map features and the roadside map elements are input into a map decoder to construct the target map.
3. The method according to claim 2, characterized in that The feature extraction model includes a feature extraction network, a deep branch network, a feature conversion network, and a content semantic branch network; the element extraction model includes a line extraction network, a sign extraction network, and a signal light extraction network; the surrounding environment image is input into the feature extraction model for feature encoding to obtain environmental map features, and the road section information is input into the element extraction model for map element extraction to obtain roadside map elements, including: Inputting the surrounding environment image into the feature extraction network to extract global features, so as to obtain global features of the surrounding environment image; Inputting the global features and the surrounding environment image into the deep branch network to extract the deep features, so as to obtain the environmental deep features of the surrounding environment image; Inputting the global features and the surrounding environment image into the feature conversion network for feature conversion to obtain the environmental bird's-eye view features of the surrounding environment image; Inputting the global features and the surrounding environment image into the content semantic branch network to extract semantic features, thereby obtaining the environmental content features of the surrounding environment image; Inputting the road section information into the line extraction network to extract line elements to obtain the line elements; Inputting the road section information into the sign extraction network to extract sign elements to obtain the road indication sign elements; The road section information is input into the traffic light extraction network to extract signal elements to obtain the traffic signal elements.
4. The method according to claim 2, characterized in that: The step of inputting the environment map features and the roadside map elements into a map decoder to construct the target map includes: converting the environmental map feature into a first vector, and converting the roadside map element into a second vector; fusing the first vector and the second vector to obtain a fused feature; The fused features are input into a map decoder to construct the target map.
5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Obtaining the location information and navigation path of the vehicle; Determine target intersection information according to the location information of the vehicle, the navigation path and the road section information broadcast by the roadside equipment; Constructing a next intersection map according to the target intersection information; The next intersection map is displayed on the display interface.
6. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: When the vehicle is started, determining whether the vehicle-mounted device meets a preset map building condition; When the vehicle-mounted device meets the preset map building condition, the step of acquiring the surrounding environment image collected by the vehicle-mounted sensor on the vehicle is performed; The preset map construction condition includes any of the following: The in-vehicle device responds to a triggering operation of a map generation control or a navigation control on a display interface of the in-vehicle device by a user; The vehicle-mounted device is powered on and started; The vehicle-mounted device detects that the vehicle travels to a target road; the target road includes any one of a curve, a ramp, an intersection, a narrow road, and an accident road.
7. A map construction system, characterized in that: The system includes vehicle-mounted sensors, vehicle-mounted equipment, and roadside equipment; The vehicle-mounted device is used to execute the method according to any one of claims 1-6.
8. A map construction device, characterized in that: The device comprises: An acquisition module is used to acquire the surrounding environment image collected by the vehicle-mounted sensor; A receiving module, used for receiving the road section information broadcasted by the roadside equipment of the road section where the vehicle is located; A construction module, used to construct a target map according to the surrounding environment image and the road section information; The display module is used to display the target map on a display interface on the vehicle-mounted device.
9. A vehicle, characterized in that: The vehicle includes an on-board device, the on-board device includes a memory and a processor, the memory stores a computer program, and is characterized in that the processor implements the steps of any one of claims 1 to 6 when executing the computer program.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
Citation Information
Patent Citations
Semantic map construction method based on vehicle infrastructure collaborative perception system
CN114882182A
Emergency navigation method and device, computer equipment, readable storage medium and program product
CN118408567A
Method and apparatus for creating a high-accuracy map
DE102017222564A1
Generation and use of HD maps
US20190323855A1
Map generation method, positioning method, apparatus, device, storage medium, and computer program
WO2021104180A1
