Map generation method and device, training method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a map generation method and device, a training method and device, electronic equipment, a storage medium and a program product, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like, and can be applied to scenes of automatic driving, automatic parking and the like. The map generation method comprises the following steps: acquiring an environment image and an initial map related to a target passing area, wherein the initial map comprises an initial lane line; performing feature fusion on the environment image and the initial map to obtain a target fusion feature; and updating the at least one initial lane line based on the target fusion feature to obtain a target map, the target map including a lane line difference attribute representing a difference between a lane line in the target map and the initial lane line.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, especially to technical fields such as computer vision, deep learning, and large models, and can be applied to scenarios such as autonomous driving and automatic parking. Background Art

[0002] With the rapid development of science and technology, the number of vehicles on the road has shown a rapid growth trend. Vehicles can use road maps to realize automatic driving functions, and drivers can also assist in driving vehicles based on more accurate road maps. The accuracy of road maps has a great impact on improving vehicle driving safety and efficiency. Summary of the invention

[0003] The present disclosure provides a map generation method, a vehicle control method, a training method, a device, an electronic device, a storage medium, and a program product.

[0004] According to one aspect of the present disclosure, a map generation method is provided, comprising: acquiring an environment image and an initial map related to a target passage area, the initial map comprising initial lane lines; performing feature fusion on the environment image and the initial map to obtain a target fusion feature; updating at least one initial lane line based on the target fusion feature to obtain a target map, the target map comprising a lane line difference attribute characterizing the difference between a lane line in the target map and the initial lane line.

[0005] According to another aspect of the present disclosure, a vehicle control method is provided, comprising: controlling vehicle driving according to a target map, wherein the target map is determined according to the map generation method provided by an embodiment of the present disclosure.

[0006] According to another aspect of the present disclosure, a training method for a deep learning model is provided, the deep learning model includes a feature fusion network and a map detection network, the training method includes: obtaining training samples, the training samples include a sample environment image, a sample initial map and a label map related to a sample traffic area, the sample initial map includes a sample initial lane line, and the label map includes a label lane line; using the feature fusion network to perform feature fusion on the sample environment image and the sample initial map to obtain a sample target fusion feature; using the map detection network to update the sample initial map based on the sample target fusion feature to obtain a sample map, the sample map includes a lane line difference attribute characterizing the difference between the sample lane line in the sample map and the sample initial lane line; training the deep learning model according to the sample map and the label map.

[0007] According to another aspect of the present disclosure, a map generation device is provided, including: a first acquisition module, used to acquire an environment image and an initial map related to a target passage area, the initial map including initial lane lines; a fusion module, used to perform feature fusion on the environment image and the initial map to obtain a target fusion feature; a target map acquisition module, used to update at least one initial lane line based on the target fusion feature to obtain a target map, the target map including a lane line difference attribute characterizing the difference between the lane line in the target map and the initial lane line.

[0008] According to another aspect of the present disclosure, a vehicle control device is provided, including: a control module, used to control vehicle driving according to a target map, wherein the target map is determined according to the map generation method provided by an embodiment of the present disclosure.

[0009] According to another aspect of the present disclosure, a training device for a deep learning model is provided, wherein the deep learning model includes a feature fusion network and a map detection network, and the device includes: a sample acquisition module, used to acquire training samples, wherein the training samples include a sample environment image, a sample initial map and a label map related to a sample passage area, wherein the sample initial map includes a sample initial lane line, and the label map includes a label lane line; a sample target fusion feature acquisition module, used to perform feature fusion on the sample environment image and the sample initial map using the feature fusion network to obtain a sample target fusion feature; a sample map acquisition module, used to update the sample initial map based on the sample target fusion feature using the map detection network to obtain a sample map, wherein the sample map includes a lane line difference attribute characterizing the difference between the sample lane line in the sample map and the sample initial lane line; and a training module, used to train the deep learning model according to the sample map and the label map.

[0010] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided by an embodiment of the present disclosure.

[0011] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method provided by an embodiment of the present disclosure.

[0012] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the method provided by the embodiment of the present disclosure when executed by a processor.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0015] Figure 1 An exemplary system architecture to which the map generation method and device according to an embodiment of the present disclosure can be applied is schematically shown;

[0016] Figure 2 A flowchart of a map generation method according to an embodiment of the present disclosure is schematically shown;

[0017] Figure 3 The schematic diagram of the deep learning model according to the embodiment of the present disclosure is shown;

[0018] Figure 4 The schematic diagram shows a principle diagram of a decoder according to an embodiment of the present disclosure;

[0019] Figure 5 A schematic diagram schematically shows a target map according to an embodiment of the present disclosure;

[0020] Figure 6 A flow chart schematically shows a vehicle control method according to an embodiment of the present disclosure;

[0021] Figure 7 A flowchart of a deep learning model training method according to an embodiment of the present disclosure is schematically shown;

[0022] Figure 8 A block diagram of a map generating device according to an embodiment of the present disclosure is schematically shown;

[0023] Fig. 9 A block diagram of a vehicle control device according to an embodiment of the present disclosure is schematically shown;

[0024] Fig.10 A block diagram schematically shows a training device for a deep learning model according to an embodiment of the present disclosure; and

[0025] Fig.11 A block diagram of an electronic device suitable for implementing a map generation method, a vehicle control method, and a deep learning model training method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0026] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0027] In the technical solution disclosed in the present invention, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations, necessary confidentiality measures are taken, and do not violate public order and good morals.

[0028] The inventors found that vehicles need to use road maps to perform autonomous driving functions such as automatic driving and automatic parking. The accuracy of road maps plays a relatively important role in the realization of vehicle autonomous driving functions. However, road maps may change due to reasons such as road construction, which makes it difficult for road maps to accurately represent the real-time road traffic environment.

[0029] The embodiments of the present disclosure provide a map generation method, a training method, a vehicle control method, an apparatus, an electronic device, a storage medium, and a program product. The map generation method includes: obtaining an environment image and an initial map related to a target traffic area, the initial map including initial lane lines; performing feature fusion on the environment image and the initial map to obtain a target fusion feature; updating at least one initial lane line based on the target fusion feature to obtain a target map, the target map including a lane line difference attribute representing the difference between a lane line in the target map and an initial lane line.

[0030] According to an embodiment of the present disclosure, by performing feature fusion on the environment image and the initial map, the semantic attributes of the image lane line in the current environment image and the initial lane line can be fused in the target fusion feature. By updating at least one initial lane line based on the target fusion feature, the image semantic attributes of the current environment image can be used to more accurately update the initial map, so that the target map can more accurately represent the lane lines in the target passage area. At the same time, the image semantics of the environment image can be enhanced based on the initial lane line attributes such as some lane lines in the initial map or the topological relationship between some lane lines, so as to improve the representation accuracy of the target map for the lane lines in the target passage area. By representing the difference between the lane lines in the target map and the initial lane lines through the lane line difference attributes of the target map, the lane line changes of the target map relative to the initial map can be more accurately obtained, thereby further improving the representation accuracy of the target map for the target passage area, so that the vehicle can perform the automatic driving function based on the lane line difference attributes, and the vehicle driver can also adjust the vehicle driving action in time based on the lane line difference attributes, thereby improving driving efficiency and safety.

[0031] Figure 1 An exemplary system architecture to which the map generation method and apparatus according to an embodiment of the present disclosure can be applied is schematically shown.

[0032] It should be noted that Figure 1 The examples shown are only examples of system architectures to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios. For example, in another embodiment, the exemplary system architecture to which the map generation method and apparatus can be applied may include a terminal device, but the terminal device may implement the map generation method and apparatus provided by the embodiments of the present disclosure without interacting with the server.

[0033] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a vehicle 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0034] The user can use the first terminal device 101 and the second terminal device 102 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101 and the second terminal device 102, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc. (only as examples).

[0035] The first terminal device 101 and the second terminal device 102 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0036] The vehicle 103 may be equipped with a vehicle-side sensor, which may include an image sensor, but is not limited thereto. The vehicle-side sensor may also be a laser radar, a millimeter-wave radar, or other types of sensors. The vehicle 103 may be any type of vehicle, such as a car, a truck, etc. The vehicle 103 may also be equipped with a communication device, and the vehicle 103 may transmit information to and from the server 105 through the network 104 based on the communication device.

[0037] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for the content browsed by the user using the first terminal device 101 and the second terminal device 102. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0038] The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server can also be a server of a distributed system, or a server combined with blockchain.

[0039] It should be noted that the map generation method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the map generation device provided in the embodiment of the present disclosure can generally be set in the server 105. The map generation method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the vehicle 103 and / or the server 105. Accordingly, the map generation device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the vehicle 103 and / or the server 105.

[0040] Alternatively, the map generation method provided in the embodiment of the present disclosure may also be executed by the first terminal device 101, the second terminal device 102, or the vehicle 103. Accordingly, the content processing device provided in the embodiment of the present disclosure may also be disposed in the first terminal device 101, the second terminal device 102, or the vehicle 103.

[0041] It should be understood that Figure 1 The number of terminal devices, vehicles, networks and servers in the embodiment is only for illustration. Any number of terminal devices, vehicles, networks and servers may be provided as required.

[0042] Figure 2 The flowchart of the map generating method according to the embodiment of the present disclosure is schematically shown.

[0043] like Figure 2 As shown, the map generating method includes operations S210 to S230.

[0044] In operation S210 , an environment image and an initial map related to a target passage area are acquired.

[0045] In operation S220, feature fusion is performed on the environment image and the initial map to obtain a target fusion feature;

[0046] In operation S230, at least one initial lane line is updated based on the target fusion feature to obtain a target map.

[0047] According to an embodiment of the present disclosure, the target traffic area may include a road traffic area, such as a highway, a national highway, a public traffic area of ​​a city traffic road light. Alternatively, the road traffic area may also include a non-public traffic area such as an industrial park road and a parking lot. The road in the target traffic area may have lane lines, and the lane lines may include lane lines of any lane line type such as a stop line, a double solid line, a parking space area line, etc.

[0048] According to an embodiment of the present disclosure, an environmental image may include an image obtained by capturing an image of a target traffic area. The environmental image may include any type of image, such as a monocular image, a depth image, a surround view image, etc. The environmental image may be captured by an image capture device based on a vehicle traveling in the target traffic area. However, this is not limited to the above, and the environmental image may also be captured based on other equipment such as roadside equipment equipped with an image capture device. The embodiment of the present disclosure does not limit the specific method of capturing the environmental image.

[0049] According to an embodiment of the present disclosure, the initial map may include any map that can represent the target traffic area, such as a high-definition map and a standard-definition map. The initial map includes initial lane lines, which may be represented based on vector lines, and may also have any type of lane line attributes, such as lane line type, lane line shape, and lane line topological relationship.

[0050] According to the embodiments of the present disclosure, feature fusion of the environment image and the initial map may include processing the environment image and the initial map based on any type of deep learning algorithm such as a convolutional neural network algorithm and an attention network algorithm. By performing feature fusion on the environment image and the initial map, the target fusion feature can fuse the semantic attributes of the image lane lines and the initial lane lines in the current environment image, and at the same time, the image semantics of the environment image can be enhanced based on the initial lane line attributes such as some lane lines in the initial map or the topological relationship between some lane lines, thereby improving the representation accuracy of the target fusion feature for the lane lines in the target traffic area.

[0051] According to the embodiments of the present disclosure, by updating at least one initial lane line based on the target fusion feature, the initial map can be updated more accurately by using the image semantic attributes of the current environment image, thereby enabling the target map to improve the representation accuracy of the road environment for the target traffic area. By characterizing the difference between the lane line in the target map and the initial lane line through the lane line difference attribute of the target map, the lane line change of the target map relative to the initial map can be obtained more accurately, thereby further improving the representation accuracy of the target map for the target traffic area.

[0052] According to an embodiment of the present disclosure, the target map includes a lane line difference attribute that characterizes the difference between the lane line in the target map and the initial lane line. The lane line difference attribute can characterize any lane line attribute difference such as lane line shape, lane line type, etc. between the lane line (or target lane line) in the target map and the initial lane line. The lane line difference attribute can be displayed in any manner such as icons, colors, text, etc. in the target map, so that the vehicle driver can adjust the vehicle driving action in time based on the lane line difference attribute, thereby improving driving efficiency and safety. At the same time, the vehicle can also be instructed to update the autonomous driving strategy based on the lane line difference attribute, so that the vehicle can perform the autonomous driving function based on a more accurate target lane line indication.

[0053] In one example, the initial map may be generated in a historical period, and the environmental image may be obtained by collecting images of the target traffic area in a preset period. The preset period is later than the historical period. By determining the target map based on the map generation method provided in the embodiment of the present disclosure, the changes between the road lane lines in the target traffic area in the preset period and the initial lane lines in the initial map can be more accurately identified, and the map of the target traffic area can be accurately and timely updated, so that the intelligent traffic management system and the urban planning system can perform tasks according to the real-time lane line difference attributes, thereby improving the operating efficiency of the system.

[0054] It should be noted that in the embodiment of the present disclosure, the lane lines in the target map can be represented as target lane lines, and the lane lines in the environment image can be represented as image lane lines.

[0055] According to an embodiment of the present disclosure, the environmental image includes a first environmental image acquired by a vehicle sensor. The first environmental image may include any type of image, such as a surround view image and a monocular image. The first environmental image may be related to the vehicle-side perspective of the vehicle. Alternatively, the first environmental image may also be a bird's-eye view image obtained by image fusion of images acquired by a vehicle-side image acquisition device. The first environmental image may be a real-time image acquired by a vehicle traveling in a target traffic area, so that the vehicle may be used to acquire real-time environmental images of the target traffic area, so that the target map can more timely and accurately characterize the roads in the target traffic area.

[0056] According to an embodiment of the present disclosure, the environment image may also include a second environment image collected by a roadside device. The second environment image collected by the roadside device may have a higher image acquisition viewing angle, so that an image that can more clearly represent the target passage area can be collected when there are many vehicles on the road, thereby enabling the target fusion feature to improve the representation accuracy for the target passage area.

[0057] According to an embodiment of the present disclosure, the environmental image may also include a first environmental image, a second environmental image, and other image acquisition devices that acquire images of the target passage area. For example, the environmental image may also include satellite images, aerial images, and the like. For environmental images from different sources, feature extraction may be performed on the environmental images from multiple sources, and coordinate conversion may be performed based on the camera parameters corresponding to each environmental image to obtain an environmental image after coordinate alignment.

[0058] According to an embodiment of the present disclosure, the feature fusion of the environment image and the initial map to obtain the target fusion feature includes: extracting features from the environment image to obtain environment image features; extracting features from the vector points of the initial lane line to obtain initial lane line features; and fusing the environment image features and the initial lane line features based on an attention mechanism to obtain the target fusion feature.

[0059] According to an embodiment of the present disclosure, feature extraction of an environmental image may include processing the environmental image based on any type of neural network algorithm such as a convolutional neural network to obtain environmental image features. In the case where the environmental image is a multi-perspective image, the image features corresponding to each perspective may be processed based on a perspective conversion algorithm such as LSS (Lift Splat Shoot) and a deformable attention algorithm to achieve perspective alignment of multi-perspective environmental image features. For another example, the image features may be subjected to posture conversion or perspective conversion based on the camera parameters corresponding to each perspective image to obtain multiple aligned initial image features. Multiple initial image features may be subjected to feature fusion to obtain environmental image features of a multi-perspective environmental image.

[0060] In one example, the environmental image includes a first environmental image, a second environmental image, and an aerial image from a bird's-eye view. The first initial image feature or the second initial environmental image feature can be converted from a perspective view to a BEV (bird's-eye view) based on a perspective transformation algorithm such as PV2BEV (Perspective View to Bird Eye View) such as an LSS algorithm and a deformable attention algorithm. For aerial images, the BEV perspective features of the aerial image can be extracted based on feature extraction algorithms such as a convolutional neural network algorithm and a Transformer algorithm. The first initial image features, the second initial environmental image features, and the BEV perspective features of the aerial image from images from different sources are transformed in feature pose based on camera parameters to obtain aligned multi-source initial image features. For example, each initial feature can be aligned by translation and rotation. After the aligned multi-source initial image features are spliced, they are input into the convolutional neural network layer for feature weighting to obtain the environmental image feature F of the fused multi-view image. I .

[0061] According to the embodiments of the present disclosure, feature extraction of the vector points of the initial lane line may include processing each vector point of the initial lane line based on a neural network algorithm. However, it is not limited thereto, and the initial lane line may also be processed based on other methods to obtain the initial lane line features. The embodiments of the present disclosure do not limit the specific method of obtaining the initial lane line features, as long as the lane line attributes of the initial lane line can be characterized.

[0062] In one example, multiple vector points of the initial lane line can be processed based on the attention mechanism to achieve feature extraction and fusion of the positional relationship and lane line type relationship between the multiple vector points, so that the obtained initial lane line features can more accurately represent the lane line attribute semantics of the initial lane line, thereby improving the representation accuracy of the initial lane line. In this way, by fusing the initial lane line features and image features, the target fusion features can more accurately represent the relationship and difference between the attribute semantics of the image lane line in the environment image and the semantic attributes of the initial lane line, thereby improving the representation accuracy of the target map for the lane line difference attributes, as well as the representation accuracy of the lane line attributes of the image lane line in the environment image.

[0063] According to an embodiment of the present disclosure, feature extraction is performed on the vector points of the initial lane line to obtain the initial lane line features, including: shape and position feature extraction is performed on the vector points of the initial lane line to obtain vector point shape and position features; lane line type feature extraction is performed on the vector points of the initial lane line to obtain vector point type features; and the shape and position features and vector point type features corresponding to each of the initial lane lines are integrated to obtain the initial lane line features.

[0064] According to an embodiment of the present disclosure, the initial lane line may include a plurality of sequentially arranged vector points. The shape and position features of the vector points of the initial lane line may be extracted by processing the vector points based on a deep learning algorithm such as a convolutional neural network algorithm to obtain the shape and position features of the vector points. However, the shape and position features may also be extracted based on other methods. For example, the vector points may be encoded based on a sine function to obtain the shape and position features of the vector points.

[0065] According to an embodiment of the present disclosure, the lane line type of the initial lane line may be the lane line type of the vector point. The lane line type may include a single solid line, a single dashed line, a double solid line, etc. Extracting lane line type features from the vector points of the initial lane line may include processing the vector points based on a deep learning algorithm such as a convolutional neural network algorithm to obtain vector point type features. However, this is not limited to the above, and lane line type features may also be extracted based on other methods. For example, the initial lane line type of the vector point may be semantically encoded based on a one-hot encoding method to obtain a vector point type feature. The vector point type feature may characterize the lane line type of the vector point.

[0066] According to an embodiment of the present disclosure, fusing the vector point shape position features and vector point type features corresponding to each initial lane line to obtain the initial lane line features may include: fusing the vector point shape position features and the vector point type features to obtain the vector point features corresponding to each vector point. The initial lane line features may be obtained by fusing the vector point features of each vector point of the initial lane line. The vector point shape position features may characterize the position and direction of the vector point, and the vector point shape position features of multiple vector points of the same initial lane line may characterize the position and geometry of the initial lane line.

[0067] In one example, the vector points in the vector line segments represented by the initial lane line can be encoded based on a sine function to obtain the vector point shape position features. Based on the one-hot encoding method, the initial lane line type of the initial lane line is feature extracted to obtain the initial lane line type features. The vector point shape position features and the vector point type features are spliced ​​to obtain the vector point features corresponding to each vector point. The vector point features of each vector point of the initial lane line are spliced ​​according to the sorting position of the vector points to obtain the lane line features to be processed. The lane line features to be processed can be of degree M * (N * d + K), where M is the number of vector line segments represented by the initial lane line, N is the number of vector points contained in the initial lane line, d is the dimension of the vector point shape position features of each vector point, and K is the dimension of the vector point type features. The lane line features to be processed are processed based on the self-attention network, so that the multiple vector point features of each of the multiple initial lane lines are self-attentionally learned to obtain the initial lane line features of each of the multiple initial lane lines of the initial map.

[0068] According to the embodiments of the present disclosure, by extracting features of the shape position semantics and lane line type semantics of the vector points of the initial lane line respectively, and fusing multiple initial lane line vector point shape position features and vector point type features based on the self-attention mechanism, the initial lane line feature can be integrated with the shape semantics, position semantics and lane line type semantics of the vector points in the corresponding initial lane line, and the shape semantics, position semantics and lane line type semantics of the respective vector points of other initial lane lines in the initial map can also be integrated. In this way, the initial lane line feature can be finely granularly integrated with the attribute semantics of the local vector points, and the attribute semantics of the vector points of other initial lane lines in the initial map can be integrated, so that the initial lane line feature can more accurately represent the semantic attributes of the initial lane lines of the initial map. By fusing the initial lane line features and image features to determine the target map, the lane lines in the target map can more accurately represent the road conditions of the target traffic area in the current environment image.

[0069] According to an embodiment of the present disclosure, fusing environmental image features and initial lane line features based on an attention mechanism to obtain a target fusion feature may include: fusing environmental image features and initial lane line features to obtain an intermediate fusion feature; and fusing the intermediate fusion feature and a preset lane line query feature based on an attention mechanism to obtain a target fusion feature.

[0070] According to an embodiment of the present disclosure, the lane line query feature is related to the attributes of the lane line in the target map. The lane line query feature can represent any lane line attribute such as the position, shape, type, etc. of the target lane line.

[0071] According to an embodiment of the present disclosure, fusing the environment image features and the initial lane line features to obtain the intermediate fused features may include: processing the environment image features and the initial lane line features based on a cross attention mechanism to obtain the intermediate fused features. However, this is not limited to this, and the environment image features and the initial lane line features may also be fused based on other methods.

[0072] According to an embodiment of the present disclosure, the intermediate fusion feature and the preset lane line query feature are fused based on the attention mechanism, and a decoder based on the attention network model can be included to fuse the intermediate fusion feature and the preset lane line query feature. The decoder can be constructed based on a decoder of any type of attention network model such as a Transformer model.

[0073] According to an embodiment of the present disclosure, updating at least one initial lane line based on a target fusion feature to obtain a target map may include: performing lane line detection based on the target fusion feature to obtain a target lane line; performing lane difference detection based on the target fusion feature and the initial lane line feature to obtain a lane line difference attribute; and determining the target map based on the target lane line and the lane line difference attribute.

[0074] According to an embodiment of the present disclosure, the initial lane line feature is determined based on the initial map. The initial lane line feature can be determined based on any of the above embodiments, and the embodiments of the present disclosure will not be repeated here.

[0075] According to an embodiment of the present disclosure, the target fusion feature can be processed based on the lane line detection branch of the deep learning model to obtain the target lane line. The lane line detection branch can be constructed based on a convolutional neural network. However, it is not limited to this, and the lane line detection branch can also be constructed based on other types of neural network algorithms such as an attention network algorithm and a multi-layer perceptron algorithm.

[0076] According to an embodiment of the present disclosure, the target lane line may have any target lane line attributes such as a target vector line representing a geometric shape, a target lane line type, etc.

[0077] In one example, the lane line detection branch may include multiple detection heads, such as a vector line detection head, a lane line type detection head, a lane line topology relationship detection head, etc. The multiple detection heads may each output a corresponding lane line attribute.

[0078] According to an embodiment of the present disclosure, lane difference detection based on target fusion features and initial lane line features may include processing the target fusion features and initial lane line features based on a lane difference check branch to obtain lane line difference attributes. The lane difference check branch may be constructed based on any type of deep learning algorithm such as a convolutional neural network algorithm, a multi-layer perceptron algorithm, or the like.

[0079] According to an embodiment of the present disclosure, the lane line difference attribute can represent the difference between the target lane line and the initial lane line, for example, it can represent the difference in lane line type between the target lane line and the initial lane line. However, it is not limited thereto, and the lane line difference attribute can also represent that the target lane line is a newly added lane line, etc.

[0080] In one example, the lane line attribute of the target lane line may be a double solid line type, and the initial lane line may be a double dashed line type. The lane line difference attribute may be rendered in a specified color at the position of the target lane line in the target map to indicate that the lane line attribute of the target lane line has changed.

[0081] It should be noted that the lane line difference attribute can also be represented based on any other manner, for example, the lane line difference attribute can be represented based on any manner such as difference mark, text, etc. The embodiment of the present disclosure does not limit the representation manner of the lane line difference attribute.

[0082] Figure 3 The schematic diagram shows a principle diagram of a deep learning model according to an embodiment of the present disclosure.

[0083] like Figure 3 As shown, the deep learning model 300 may include an image coding layer 311, a map coding layer 312, an encoder 313, a decoder 314, a lane line detection branch 321, and a lane line difference detection branch 322. Among them, the image coding network 311, the map coding network 312, the encoder 313, and the decoder 314 can be used as a feature fusion network, and the lane line detection branch 321 and the lane line difference detection branch 322 can be used as a map detection network.

[0084] The environmental image P301 is input into the image coding layer 311, and the environmental image features are output. The environmental image features may be multi-perspective images, and the environmental image features may be environmental image features from a bird's-eye view perspective obtained after conversion based on a perspective conversion algorithm. The initial map G301 is input into the map coding layer 312, and the initial lane line features are output. The initial lane line features may represent multiple initial lane lines in the initial map G301. The environmental image features and the initial lane line features are input into the encoder 313, and the encoder 313 may fuse the environmental image features and the initial lane line features based on a cross-attention mechanism to obtain intermediate fusion features. The encoder 313 may be constructed based on an attention network algorithm.

[0085] The preset lane line query feature F311 and the intermediate fusion feature can be input into the decoder 314 to output the target fusion feature. The decoder 314 can be constructed based on the attention network algorithm, for example, it can include a self-attention network layer and a cross-attention network layer. The target fusion feature is input into the lane detection branch 321, and the target lane line 301 is output. The target fusion feature and the initial lane line feature are input into the lane line difference detection branch, and the lane line difference attribute 302 is output. By fusing the target lane line 301 and the lane line difference attribute 302, the target map G302 can be obtained.

[0086] According to the embodiments of the present disclosure, the deep learning model can be a large model. The large model can be a model with a large-scale parameter structure, and the parameter magnitude is generally tens of millions or hundreds of millions, and may reach billions or tens of billions. In terms of the network structure of the large model, for example, a network structure such as UFO (Unified Feature Optimization) can be used. By processing the environmental image and the initial map through a large model, the environmental image and the initial map can be fully integrated based on the relatively powerful feature understanding and expression capabilities of the large model, and efficient processing of environmental images with a large data scale can be achieved, thereby improving the detection accuracy and efficiency of the target map.

[0087] According to an embodiment of the present disclosure, the lane line query feature can be obtained by pre-training a deep learning model. The lane line query feature can be used as an intermediate fusion feature to query any semantic attributes such as the shape, position, and lane line type of the image lane line in the environment image, thereby improving the representation accuracy of the target fusion feature.

[0088] According to an embodiment of the present disclosure, fusing intermediate fusion features and preset lane line query features based on an attention mechanism to obtain target fusion features may include: fusing n-1th level query features and intermediate fusion features based on an attention mechanism to obtain nth level query features; and when N=n, determining the target fusion features based on the Nth level query features.

[0089] According to an embodiment of the present disclosure, the first-level query feature may be a preset lane line query feature. It should be noted that N≥n>1, and both N and n are integers. The target fusion feature may be made to more accurately characterize the road lane line in the target traffic area by multi-level attention fusion of the intermediate fusion feature and the lane line query feature.

[0090] Figure 4 The schematic diagram shows a principle diagram of a decoder according to an embodiment of the present disclosure.

[0091] like Figure 4 As shown, the decoder 410 may include a first decoder layer 411 , a second decoder layer 412 , and a third decoder layer 413 .

[0092] When n=2, the lane line query feature F411 and the intermediate fusion feature F401 output by the encoder are input into the first decoder layer 411. The first decoder layer 411 can use the lane line query feature F411 as the first-level query feature, fuse the first-level query feature and the intermediate fusion feature F401 based on the attention mechanism, and output the second-level query feature F412.

[0093] When n=3, the second-level query feature F412 and the intermediate fused feature F401 are input into the second decoder layer 412. The second decoder layer 412 can fuse the second-level query feature and the intermediate fused feature F401 based on the attention mechanism, and output the third-level query feature F413.

[0094] When N=n=4, the third-level query feature and the intermediate fused feature F401 are input to the third decoder layer 413. The third decoder layer 413 can fuse the third-level query feature and the intermediate fused feature F401 based on the attention mechanism and output the fourth-level query feature. The fourth-level query feature can be used as the target fused feature F401'.

[0095] like Figure 4As shown, the second decoder layer 412 may include a second multi-head self-attention sublayer 4121, a second cross-attention sublayer 4122, and a second self-attention sublayer 4123. The second-level query feature F412 is input into the second multi-head self-attention sublayer 4121, and self-attention fusion can be implemented to output the second-level initial query fusion feature. The second-level initial query fusion feature and the intermediate fusion feature F401 can be input into the second cross-attention sublayer 4122, and the second-level intermediate query fusion feature is output. The second-level intermediate query fusion feature is input into the second self-attention sublayer 4123, and the third-level query feature F413 is output.

[0096] It should be understood that the first decoder layer 411 and the third decoder layer 413 may have the same or similar network structure as the second decoder layer 412, and the embodiments of the present disclosure will not be described in detail herein.

[0097] In one embodiment, the decoder may include a plurality of cascaded decoder layers. The decoder layer may be constructed based on a cascaded multi-head self-attention sublayer, a cross-attention sublayer, and a self-attention sublayer. The multi-head self-attention sublayer may be determined based on a multi-head self-attention (MHSA) network algorithm, the cross-attention sublayer may be determined based on a deformable cross-attention (DCA) network algorithm, and the self-attention sublayer may be determined based on an intra-instance self-attention (IISA) network algorithm. The multi-head self-attention sublayer may be used to fully semantically exchange feature vectors representing the semantics of each lane line in the n-th level query feature. The cross-attention sublayer may be used to perform attention interaction between the second-level initial query fusion feature and the intermediate fusion feature in the BEV space to obtain semantic information. The self-attention sublayer may further enhance the information interaction of the query features between the various vector points in the lane line through the self-attention mechanism, and then realize that each level of query features can smoothly and accurately predict the lane line attributes and lane line difference attributes of the target lane line through multi-level attention interaction.

[0098] It should be understood that the decoder layer may also include a normalization (LayerNorm, LN) network sublayer and a feed-forward neural network (Feed-Forward Network, FFN) sublayer, which will not be described in detail in the embodiments of the present disclosure.

[0099] According to an embodiment of the present disclosure, the lane line query feature can be used to explicitly encode features related to lane line attributes in the intermediate fusion feature. The lane line query feature can be related to the vector line of the target lane line. Alternatively, the lane line query feature can also be related to the target vector point in the target lane line.

[0100] In one example, the lane line query feature may include a vector line query feature representing each vector line. The vector line query feature may be expressed as , Indicates the number of target lane lines, Represents the vector line query feature corresponding to the i-th target lane line.

[0101] In one example, the lane line query feature may include a vector point query feature that represents a vector point in a vector line. The vector point query feature may be expressed as , represents the number of vector points in the vector line, Represents the vector point query feature corresponding to the j-th vector point.

[0102] In one example, the lane line query feature may be determined based on the vector line query feature and the vector point query feature. The lane line query feature of the i-th target lane line may be determined by fusing the vector line query feature and the vector point query feature of the i-th target lane line.

[0103] For example, the lane line query feature of the i-th target lane line may be determined based on the following formula (1).

[0104] (1);

[0105] in, Represents the lane line query feature of the i-th target lane line.

[0106] In one example, the lane line query feature can be determined based on the vector line query feature, the vector point query feature, and the foreground query feature. The lane line query feature of the target lane line can be determined by fusing the vector line query feature, the vector point query feature, and the foreground query feature of the target lane line. The foreground query feature can characterize the features of the expression of foreground segmentation for the image lane line in the road image. Determining the lane line query feature based on the foreground query feature can enable the lane line query feature to further accurately and explicitly characterize the target lane line, so that the target fusion feature can explicitly express the image lane line in the environment image by multi-level fusion of the lane line query feature and the intermediate fusion feature, thereby improving the detection accuracy of the target lane line.

[0107] According to an embodiment of the present disclosure, performing lane difference detection based on target fusion features and initial lane line features to obtain lane line difference attributes may include: fusing the target fusion features and the initial lane line features to obtain a target association matrix; and determining the lane line difference attributes based on the association matrix elements.

[0108] According to an embodiment of the present disclosure, the target fusion features and the initial lane line features may be processed based on the lane line difference detection branch to obtain a target association matrix.

[0109] According to an embodiment of the present disclosure, an association matrix element in the target association matrix represents whether the i-th target lane line matches the j-th initial lane line. The association matrix element in the i-th row and j-th column of the target association matrix may be the [i, j]-th matrix element, and the [i, j]-th matrix element may represent whether the i-th target lane line matches the j-th initial lane line.

[0110] In one example, the lane line difference attribute can be characterized as whether the lane line attributes such as shape and position between the target map and the initial map match.

[0111] According to an embodiment of the present disclosure, whether the i-th target lane line matches the j-th initial lane line can be expressed as the i-th target lane line and the j-th initial lane line are associated lane lines. For example, the i-th target lane line and the j-th initial lane line are lane lines with the same position and shape.

[0112] In one example, fusing the target fusion features and the initial lane line features may include inputting a multi-layer perceptron layer into a lane line difference detection branch based on the target fusion features and the initial lane line features to unify the feature space, and then performing an outer product operation on the features output by the multi-layer perceptron layer to obtain a target association matrix.

[0113] In one example, the target fusion feature and the initial lane line feature may be combined, which may include: m and the initial lane feature E m Perform an outer product operation, and input the outer product operation result into the multi-layer perceptron layer of the lane line difference detection branch to obtain the target association matrix A. For example, the target association matrix A can be determined according to the following formula (2)

[0114] (2);

[0115] In formula (2) Represents the outer product operation.

[0116] According to an embodiment of the present disclosure, the target association matrix can represent the association relationship between the target lane line and the initial lane line based on the affinity matrix (also called similarity matrix or affinity matrix). For example, it can be determined whether the lane line type between the i-th target lane line and the j-th initial lane line is the same.

[0117] The association matrix element may, for example, represent the similarity between the i-th target lane line and the j-th initial lane line. When the similarity is greater than or equal to a preset threshold, it may be characterized that the i-th target lane line matches the j-th initial lane line as a matching lane line. When the similarity is less than a preset threshold, it may be characterized that the i-th target lane line matches the j-th initial lane line as a mismatched lane line.

[0118] According to an embodiment of the present disclosure, the lane line difference attribute may include: at least one of a newly added attribute, a deleted attribute, and a type changed attribute.

[0119] According to an embodiment of the present disclosure, a newly added attribute indicates that the initial map does not have the i-th target lane line. For example, when the similarity represented by the [i, j]-th matrix element in the association matrix element is less than a preset threshold and the i-th target lane line exists in the target map, the newly added attribute can be determined.

[0120] According to an embodiment of the present disclosure, the deletion attribute indicates that the jth initial lane line is deleted in the target map. For example, when the similarity represented by the [i, j]th matrix element in the association matrix element is less than a preset threshold, and there is no jth initial lane line matching the ith target lane line in the target map, the deletion attribute can be determined.

[0121] According to an embodiment of the present disclosure, the type change attribute indicates that the i-th target lane line and the j-th initial lane line are lane lines with similar positions, and the i-th target lane line and the j-th initial lane line have different lane line types. For example, the type change attribute can be determined when the similarity represented by the [i, j]-th matrix element in the association matrix element is greater than or equal to a preset threshold, and the target lane line type of the i-th target lane line is different from the lane line type of the j-th initial lane line.

[0122] According to an embodiment of the present disclosure, the lane line difference attribute can also characterize the lane line same attribute. The lane line same attribute can characterize that the i-th target lane line and the j-th initial lane line in the target map are the same lane line. For example, the lane line same attribute can be determined when the similarity represented by the [i, j]-th matrix element in the association matrix element is greater than or equal to a preset threshold, and the target lane line type of the i-th target lane line is the same as the lane line type of the j-th initial lane line.

[0123] Figure 5 A schematic diagram of a target map according to an embodiment of the present disclosure is schematically shown.

[0124] like Figure 5As shown, the red line filling of the lane line difference attribute L511 in the target map G501 represents the newly added attribute, which can indicate that the first target lane line corresponding to the lane line difference attribute L511 is a newly added lane line relative to the initial map. The light blue line filling of the lane line difference attribute L512 represents the type change attribute, which can indicate that the second target lane line corresponding to the lane line difference attribute L512 has the same position as the initial lane line in the initial map, but the lane line type of the second target lane line relative to the initial lane line has changed. The dark blue line filling of the lane line difference attributes L513, L514 and L515 respectively represents the same attribute of the lane line. The third target lane line, the fourth target lane line and the fifth target lane line corresponding to the lane line difference attributes L513, L514 and L515 respectively have the same position and lane line type as the corresponding three initial lane lines in the initial map, and the three initial lane lines have not changed. The red line filling of the lane line difference attribute L515 represents the deletion attribute, which can indicate that the initial lane line corresponding to the position and shape of the lane line difference attribute L515 in the initial map has been deleted.

[0125] Figure 6 The flowchart of the vehicle control method according to the embodiment of the present disclosure is schematically shown.

[0126] like Figure 6 As shown, the vehicle control method includes operation S610.

[0127] In operation S610, the vehicle is controlled to travel according to the target map.

[0128] According to an embodiment of the present disclosure, the target map is determined according to the map generation method provided by the above embodiment.

[0129] According to an embodiment of the present disclosure, the vehicle can re-plan the driving path based on the target lane line and lane line difference attributes indicated by the target map to avoid illegal driving actions of the vehicle due to changes in lane line type or lane line shape, thereby improving driving safety.

[0130] Figure 7 A flowchart of a deep learning model training method according to an embodiment of the present disclosure is schematically shown.

[0131] like Figure 7 As shown, the training method of the deep learning model includes operations S710~S740.

[0132] In operation S710, a training sample is acquired.

[0133] In operation S720, a feature fusion network is used to perform feature fusion on the sample environment image and the sample initial map to obtain a sample target fusion feature.

[0134] In operation S730, the map detection network is used to update the sample initial map based on the sample target fusion feature to obtain a sample map.

[0135] In operation S740, a deep learning model is trained based on the sample map and the label map.

[0136] According to an embodiment of the present disclosure, the deep learning model includes a feature fusion network and a map detection network.

[0137] According to an embodiment of the present disclosure, the training sample includes a sample environment image related to the sample traffic area, a sample initial map and a label map, the sample initial map includes a sample initial lane line, and the label map includes a label lane line.

[0138] According to an embodiment of the present disclosure, the sample map includes a lane line difference attribute characterizing the difference between a sample lane line in the sample map and a sample initial lane line.

[0139] According to an embodiment of the present disclosure, the label map may be map information that accurately represents the lane lines of the sample image in the current sample environment image. The label map may be used as a label (or true value) for training a deep learning model. The deep learning model may be trained based on the sample map and the label map, and a target loss value may be determined based on the sample map and the label map, and the deep learning model may be trained based on the target loss value to obtain a trained deep learning model.

[0140] According to an embodiment of the present disclosure, a deep learning model is trained based on a sample map and a label map, and the trained deep learning model obtained can be used to execute the map generation method provided in the above embodiment. For example, an environmental image and an initial map can be processed based on the trained deep learning model to obtain a target map.

[0141] It should be noted that the technical terms involved in the training method provided in the embodiment of the present disclosure, including but not limited to sample environment images, sample initial maps, deep learning models, etc., have the same or corresponding meanings as the technical terms involved in the map generation method provided in the embodiment of the present disclosure, and the embodiment of the present disclosure will not be repeated here.

[0142] According to an embodiment of the present disclosure, the sample map includes sample lane lines and sample lane line types. The label map may also include label lane lines and label lane line types correspondingly.

[0143] According to an embodiment of the present disclosure, training a deep learning model based on a sample map and a label map may include: processing sample lane lines and label lane lines according to a first loss function to obtain a lane line position loss value; processing the sample lane line type of the sample lane line according to a second loss function to obtain a lane line type loss value; and training a deep learning model based on the lane line type loss value and the lane line position loss value.

[0144] According to an embodiment of the present disclosure, the first loss function and the second loss function may be the same or different. For example, the first loss function and the second loss function may both be cross entropy loss functions. For another example, the first loss function may be an exponential loss function, and the second loss function may be a cross entropy loss function.

[0145] According to an embodiment of the present disclosure, training a deep learning model based on a lane line type loss value and a lane line position loss value may include determining a target loss value based on the lane line type loss value and the lane line position loss value, and adjusting model parameters of the deep learning model based on the target loss value until the target loss value converges to obtain a trained deep learning model.

[0146] In one example, the map detection network of the deep learning model may include a lane detection branch, a lane type detection branch, and a foreground segmentation branch. The lane detection branch, the lane type detection branch, and the foreground segmentation branch may output sample lanes, sample lane types, and sample segmentation semantic features, respectively. The sample lanes, sample lane types, and sample segmentation semantic features may respectively characterize the geometry of the image lanes in the sample environment image, the lane type, and the foreground mask.

[0147] In one example, the lane detection branch output dimension is A vector representing the sample lane line The normalized coordinates of the vector points. The output dimension of the lane line type detection branch is A vector indicating that the sample lane line belongs to The output dimension of the probability foreground segmentation branch of the sample lane line type can be a tensor of N×N. The value of each tensor element in the tensor represents the probability of the sample lane line at the current position. In the training stage, the Hungarian matching mechanism can be used to process the sample prediction value. With label value . Among them, the sample prediction value may include the sample lane line, the sample lane line type and the sample segmentation semantic feature. The label value may include the label lane line, the label lane line type and the label segmentation semantic feature of the label map. The sample lane line type and the label lane line type are processed based on the Focal Loss loss function to obtain the lane line type loss value. The sample lane line and the label lane line are processed based on L1 Loss (or Mean Absolute Error) to obtain the lane line loss value. The label segmentation semantic feature and the sample segmentation semantic feature can also be processed based on L1 Loss or other types of loss functions to obtain the segmentation semantic loss value. The target loss value is determined based on the cumulative sum of the lane line loss value, the lane line type loss value and the segmentation semantic loss value.

[0148] In one example, the lane line loss value may include a vector point position loss value L corresponding to the sample vector point. reg , vector point direction loss value L dir , the sample vector point can be used to construct the sample lane line. The vector point position loss value L reg It can be obtained by processing the position of the sample vector point and the position of the label vector point based on the loss function. The vector point direction loss value L dir It can be obtained by processing the sample vector point direction and the label vector point direction based on the loss function. The sample vector point direction can indicate the direction of the connecting edge of two adjacent sample vector points in the sample lane line, and the label vector point direction can indicate the direction of the connecting edge of two adjacent label vector points in the label lane line.

[0149] According to the embodiments of the present disclosure, by determining the vector point position loss value and the vector point direction loss value, it is possible to achieve a fine-grained characterization of the difference between the position and geometry of the sample lane line and the label lane line, so that the target loss value determined according to the vector point position loss value and the vector point direction loss value can fine-grainedly characterize the difference between the sample lane line predicted by the deep learning model and the true value. In this way, using the target loss value to train the deep learning model can improve the detection accuracy of the deep learning model for the target vector point in the target lane line, so as to improve the detection accuracy of the deep learning model.

[0150] In one example, the target loss value can be based on the vector point position loss value L reg , vector point direction loss value L dir , Lane line type loss value L cls and segmentation semantic loss value L seg to be sure.

[0151] For example, the target loss value can be determined based on the following formula (3):

[0152] (3).

[0153] In formula (3), ω, λ, ρ and μ represent the lane line type loss value L, cls , vector point position loss value L reg , vector point direction loss value L dir and segmentation semantic loss value L seg The respective weight coefficients.

[0154] According to an embodiment of the present disclosure, processing the sample lane line type of the sample lane line according to the second loss function to obtain the lane line type loss value may include: processing the sample lane line type based on the loss function and dynamic label data determined based on the lane line distance to obtain the lane line type loss value.

[0155] According to an embodiment of the present disclosure, the lane line distance represents the distance between the sample lane line and the label lane line. For example, the lane line distance can be determined based on the similarity between the sample lane line and the label lane line. However, it is not limited to this, and the lane line loss value can also be determined based on the distance between the sample vector point position and the label lane line position. For example, the distance between the i-th sample vector point position and the i-th label vector point position can be determined, and the lane line distance can be determined based on the weighted average of the distances between the N sample vector point positions and the N label vector points.

[0156] According to an embodiment of the present disclosure, the dynamic label data may be determined based on the lane line distance, for example, the dynamic label data may be obtained by normalizing the lane line distance. However, it is not limited thereto, and the dynamic label data may also be determined based on other methods. For example, the dynamic label data may also be determined based on a preset label value and the normalized lane line distance. By introducing dynamic label data determined based on the lane line distance, the training method can update the dynamic label data of the current round based on the detection result of the current sample lane line position after determining the sample lane line in each round. In this way, the dynamic label data used to calculate the lane line type loss value can be adjusted according to the prediction accuracy of the sample lane line position, thereby reducing the difference between the lane line type loss value and the lane line loss value during the training process, achieving training consistency between the lane line position regression task and the lane line type regression task, avoiding the situation where the detection tasks of the sample lane line and the sample lane line type have an imbalance in accuracy, and improving the robustness of the deep learning model.

[0157] In one example, the lane line type loss value may be determined based on the following formula (4).

[0158] (4);

[0159] In formula (4), p i represents the classification probability of the sample lane line type of the i-th sample lane line, d i represents the lane line distance between the i-th sample lane line and the j-th label lane line that matches the i-th sample lane line, N pos and N neg Respectively represent the number of foreground sample lane lines and the number of background sample lane lines, γ represents a hyperparameter, and BCE() represents a binary cross entropy loss function. Among them, the foreground sample lane lines have matching label lane lines in the label map, and the background lane lines do not have matching label lane lines in the label map.

[0160] According to an embodiment of the present disclosure, training a deep learning model based on a sample map and a label map may also include: processing sample target fusion features and sample initial lane line features based on a lane difference detection branch of a map detection network to obtain a sample association matrix; and using the acquired label difference attributes as label data to train a deep learning model based on sample matrix elements and label difference attributes.

[0161] According to an embodiment of the present disclosure, the sample matrix elements in the sample association matrix represent the similarity between the i-th sample lane line and the j-th sample initial lane line.

[0162] According to an embodiment of the present disclosure, the label difference attribute characterizes the difference between the i-th label lane line and the j-th sample initial lane line in the label map. For example, the label difference attribute can characterize whether the i-th label lane line and the j-th sample initial lane line are the same lane line. When the i-th label lane line and the j-th sample initial lane line are at the same position, the [i, j]-th label difference attribute can be characterized as 1, and when the i-th label lane line and the j-th sample initial lane line are not the same, the [i, j]-th label difference attribute can be characterized as 0. The label difference attribute can characterize whether the label lane line matches the sample lane line. The label difference attribute can determine the label difference attribute matrix based on the same arrangement as the sample association matrix. The loss function is used to process the label difference attribute matrix and the sample association matrix to obtain the difference attribute loss value. The deep learning model is trained based on the difference attribute loss value to obtain the trained deep learning model.

[0163] In one example, a target loss value can be determined based on the difference attribute loss value and the lane line loss value and the lane line type loss value, and a deep learning model can be trained based on the target loss value.

[0164] In one example, a deep learning model can be trained based on a lane line loss value and a lane line type loss value to obtain an intermediate deep learning model. An intermediate deep learning model can be trained based on a difference attribute loss value to obtain a trained deep learning model. In this way, the deep learning model can be trained in the first stage based on the lane line loss value and the lane line type loss value to improve the prediction accuracy of the sample association matrix, so that training the intermediate deep learning model based on the difference attribute loss value can accelerate the convergence speed and improve the training efficiency of the deep learning model.

[0165] In one example, the loss function L f () Process the [i,j]th label difference attribute And the [i,j]th sample matrix element in the sample association matrix, get the difference attribute loss value L a .

[0166] ; (5)

[0167] According to the embodiment of the present disclosure, in order to improve the reliability of the sample association matrix, the sample matrix elements with low confidence in the sample association matrix can be processed based on the Hungarian matching strategy, so that the sample matrix elements in the processed target sample association matrix represent that each sample lane line only matches one label lane line, so that the target association matrix can be constructed. . Where m represents the number of sample lane lines, and n represents the number of label lane lines. Then the target sample matrix element Equal to 1 indicates that the i-th sample lane line matches the j-th label lane line, and the target sample matrix element is equal to 0, indicating that the i-th sample lane line does not match the j-th label lane line. Therefore, the target sample association matrix The target sample matrix elements in can represent the following lane line difference attributes.

[0168] Based on the target sample matrix elements in the target sample association matrix M, the following four sample lane line difference attributes can be represented.

[0169] (1) Sample lane lines have the same attributes: Mij=1 and the sample lane line type of the i-th sample lane line is the same as that of the j-th labeled lane line.

[0170] (2) Sample type change attribute: Mij=1 and the sample lane line types of the i-th sample lane line and the j-th labeled lane line are inconsistent.

[0171] (3) New attribute of the sample: Mi=0, indicating that the i-th sample lane line does not match any labeled lane line.

[0172] (4) Sample deletion attribute: Mj=0, indicating that the jth labeled lane line does not match any sample lane line.

[0173] Training a deep learning model based on sample matrix elements and label difference attributes may include deleting the target sample matrix elements corresponding to at least one of the sample newly added attributes and the sample deleted attributes in the target sample association matrix M to obtain a training association matrix. Based on the matrix elements of the training association matrix, the corresponding sample training matrix elements are screened out from the sample association matrix A. Based on the sample training matrix elements screened out from the sample association matrix A, the difference attribute loss value is determined with the label difference attribute, and the deep learning model is trained according to the difference attribute loss value. In this way, the sample matrix elements in the sample association matrix that are all predicted to be 0 by the deep learning model can be deleted to prevent them from participating in the deep learning model training, thereby reducing the loss calculation overhead during the training process and improving the robustness of the deep learning model after training.

[0174] In one example, the difference attribute loss value may be determined based on the following formula (6).

[0175] (6).

[0176] in Represents the elements of the sample training matrix after screening Matching tag difference attributes, It can represent the cross entropy loss function.

[0177] It should be noted that the process of obtaining, collecting or processing information involved in the embodiments of the present disclosure is performed under the condition of obtaining authorization from relevant users or organizations, and the relevant users or organizations are clearly informed that the purpose of the method provided by the embodiments of the present disclosure is to improve the driving safety and driving efficiency of the vehicle. The method provided by the embodiments of the present disclosure adopts necessary encryption or desensitization measures for the acquired information, including but not limited to the information to be tested, the lane attributes of the associated area, etc., to avoid information leakage.

[0178] Figure 8 The block diagram of a map generating device according to an embodiment of the present disclosure is schematically shown.

[0179] like Figure 8 As shown, the map generating device 800 includes: a first acquisition module 810 , a fusion module 820 and a target map obtaining module 830 .

[0180] The first acquisition module 810 is used to acquire an environment image and an initial map related to a target passage area, where the initial map includes initial lane lines.

[0181] The fusion module 820 is used to perform feature fusion on the environment image and the initial map to obtain target fusion features.

[0182] The target map acquisition module 830 is used to update at least one initial lane line based on the target fusion feature to obtain a target map, wherein the target map includes a lane line difference attribute representing the difference between the lane line in the target map and the initial lane line.

[0183] According to an embodiment of the present disclosure, the target map acquisition module 830 includes: a target lane line detection submodule, a lane line difference attribute acquisition submodule and a target map determination submodule.

[0184] The target lane line detection submodule is used to perform lane line detection based on the target fusion features to obtain the target lane line.

[0185] The lane line difference attribute acquisition submodule is used to perform lane difference detection based on the target fusion feature and the initial lane line feature to obtain the lane line difference attribute, wherein the initial lane line feature is determined based on the initial map.

[0186] The target map determination submodule is used to determine the target map based on the target lane line and the lane line difference attribute.

[0187] According to an embodiment of the present disclosure, the lane line difference attribute acquisition submodule includes: a target association matrix acquisition unit and a lane line difference attribute determination unit.

[0188] The target association matrix obtaining unit is used to fuse the target fusion features and the initial lane line features to obtain the target association matrix. The association matrix elements in the target association matrix represent whether the i-th target lane line matches the j-th initial lane line.

[0189] The lane line difference attribute determination unit is used to determine the lane line difference attribute based on the association matrix elements.

[0190] According to an embodiment of the present disclosure, the lane line difference attribute includes at least one of the following: a newly added attribute, a deleted attribute, and a type changed attribute.

[0191] The newly added attribute indicates that the initial map does not have the i-th target lane line.

[0192] The deletion attribute represents that the jth initial lane line is deleted in the target map.

[0193] The type change attribute indicates that the i-th target lane line and the j-th initial lane line are lane lines with similar positions, and the i-th target lane line and the j-th initial lane line have different lane line types.

[0194] According to an embodiment of the present disclosure, the fusion module includes: a first feature extraction submodule, a second feature extraction submodule and a fusion submodule.

[0195] The first feature extraction submodule is used to extract features from the environment image to obtain environment image features.

[0196] The second feature extraction submodule is used to extract features from vector points of the initial lane line to obtain initial lane line features.

[0197] The fusion submodule is used to fuse the environment image features and the initial lane line features based on the attention mechanism to obtain the target fusion features.

[0198] According to an embodiment of the present disclosure, the fusion submodule includes: an intermediate fusion feature obtaining unit and a target fusion feature obtaining unit.

[0199] The intermediate fusion feature acquisition unit is used to fuse the environment image features and the initial lane line features to obtain the intermediate fusion features.

[0200] The target fusion feature acquisition unit is used to fuse the intermediate fusion feature and the preset lane line query feature based on the attention mechanism to obtain the target fusion feature, and the lane line query feature is related to the attribute of the lane line in the target map.

[0201] According to an embodiment of the present disclosure, the target fusion feature obtaining unit includes: a first fusion subunit and a target fusion feature determining subunit.

[0202] The first fusion subunit is used to fuse the n-1th level query feature and the intermediate fusion feature based on the attention mechanism to obtain the nth level query feature, where the first level query feature is a preset lane line query feature.

[0203] The target fusion feature determination subunit is used to determine the target fusion feature based on the Nth level query feature when N=n, wherein N≥n>1, and N and n are both integers.

[0204] According to an embodiment of the present disclosure, the second feature extraction submodule includes: a vector point shape position feature acquisition unit, a vector point type feature acquisition unit and an initial lane line feature acquisition unit.

[0205] The vector point shape position feature acquisition unit is used to extract the shape position features of the vector points of the initial lane line to obtain the vector point shape position features.

[0206] The vector point type feature acquisition unit is used to extract lane line type features from the vector points of the initial lane line to obtain vector point type features.

[0207] The initial lane line feature acquisition unit is used to fuse the vector point shape position features and vector point type features corresponding to each initial lane line to obtain the initial lane line features.

[0208] According to an embodiment of the present disclosure, the environment image includes at least one of the following: a first environment image collected by a vehicle sensor; a second environment image collected by a roadside device.

[0209] Fig. 9 A block diagram of a vehicle control device according to an embodiment of the present disclosure is schematically shown.

[0210] like Fig. 9 As shown, the vehicle control device 900 includes a control module 910 .

[0211] The control module 910 is used to control the vehicle to travel according to the target map, wherein the target map is determined according to the map generation method provided in the above embodiment.

[0212] Fig.10 A block diagram of a deep learning model training device according to an embodiment of the present disclosure is schematically shown.

[0213] like Fig.10 As shown, the training device 1000 of the deep learning model includes: a sample acquisition module 1010, a sample target fusion feature 1020, a sample map acquisition module 1030 and a training module 1040.

[0214] The sample acquisition module acquisition module 1010 is used to acquire training samples, the training samples include sample environment images related to the sample passage area, sample initial maps and label maps, the sample initial map includes sample initial lane lines, and the label map includes label lane lines.

[0215] The sample target fusion feature acquisition module 1020 is used to use a feature fusion network to perform feature fusion on the sample environment image and the sample initial map to obtain the sample target fusion feature.

[0216] The sample map acquisition module 1030 is used to use the map detection network to update the sample initial map based on the sample target fusion feature to obtain the sample map, wherein the sample map includes a lane line difference attribute representing the difference between the sample lane line in the sample map and the sample initial lane line.

[0217] The training module 1040 is used to train the deep learning model according to the sample map and the label map.

[0218] According to an embodiment of the present disclosure, a sample map includes sample lane lines and sample lane line types.

[0219] According to an embodiment of the present disclosure, the training module includes: a lane line position loss value obtaining submodule, a lane line type loss value obtaining submodule and a first training submodule.

[0220] The lane line position loss value obtaining submodule is used to process the sample lane line and the label lane line according to the first loss function to obtain the lane line position loss value.

[0221] The lane line type loss value obtaining submodule is used to process the sample lane line type of the sample lane line according to the second loss function to obtain the lane line type loss value. The first loss function and the second loss function are the same or different.

[0222] The first training submodule is used to train a deep learning model based on a lane line type loss value and a lane line position loss value.

[0223] According to an embodiment of the present disclosure, the lane line type loss value obtaining submodule includes a lane line type loss value obtaining unit.

[0224] The lane line type loss value obtaining unit is used to process the sample lane line type and the dynamic label data determined based on the lane line distance based on the loss function to obtain the lane line type loss value, wherein the lane line distance represents the distance between the sample lane line and the label lane line.

[0225] According to an embodiment of the present disclosure, the training module includes: a sample association matrix acquisition submodule and a second training submodule.

[0226] The sample association matrix acquisition submodule is used to process the sample target fusion features and the sample initial lane line features based on the lane difference detection branch of the map detection network to obtain the sample association matrix. The sample matrix elements in the sample association matrix represent the similarity between the i-th sample lane line and the j-th sample initial lane line.

[0227] The second training submodule is used to use the acquired label difference attributes as label data, and train the deep learning model based on the sample matrix elements and the label difference attributes, wherein the label difference attributes represent the difference between the i-th label lane line in the label map and the j-th sample initial lane line.

[0228] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0229] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided according to the embodiment of the present disclosure.

[0230] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute a method provided according to an embodiment of the present disclosure.

[0231] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the method provided according to the embodiment of the present disclosure is implemented.

[0232] Fig.11 A block diagram of an electronic device suitable for implementing a map generation method, a vehicle control method, and a training method for a deep learning model according to an embodiment of the present disclosure is schematically shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0233] like Fig.11 As shown, the device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 to a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the device 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0234] A number of components in the device 1100 are connected to the I / O interface 1105, including: an input unit 1106, such as a keyboard, a mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a disk, an optical disk, etc.; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1109 allows the device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0235] The computing unit 1101 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1101 performs the various methods and processes described above, such as a map generation method, a vehicle control method, and a training method for a deep learning model. For example, in some embodiments, the map generation method, the vehicle control method, and the training method for a deep learning model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the map generation method, the vehicle control method, and the training method for a deep learning model described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to execute a map generation method, a vehicle control method, or a deep learning model training method in any other appropriate manner (e.g., by means of firmware).

[0236] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0237] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0238] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0239] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0240] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0241] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0242] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0243] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A map generation method, comprising: Acquire an environment image and an initial map related to a target traffic area, wherein the initial map includes initial lane lines; Performing feature fusion on the environment image and the initial map to obtain target fusion features; At least one of the initial lane lines is updated based on the target fusion feature to obtain a target map, wherein the target map includes a lane line difference attribute representing a difference between a lane line in the target map and the initial lane line.

2. The method according to claim 1, wherein: The updating of at least one of the initial lane lines based on the target fusion feature to obtain the target map comprises: Perform lane line detection based on the target fusion feature to obtain a target lane line; Performing lane difference detection based on the target fusion feature and the initial lane line feature to obtain the lane line difference attribute, wherein the initial lane line feature is determined based on the initial map; and The target map is determined based on the target lane line and the lane line difference attribute.

3. The method according to claim 2, wherein: The lane difference detection is performed based on the target fusion feature and the initial lane line feature to obtain the lane line difference attribute, including: Fusion of the target fusion features and the initial lane line features to obtain a target association matrix, wherein an association matrix element in the target association matrix represents whether the i-th target lane line matches the j-th initial lane line; and The lane line difference attribute is determined based on the association matrix element.

4. The method according to any one of claims 1 to 3, wherein: The lane line difference attribute includes at least one of the following: A new attribute is added to indicate that the initial map does not have the i-th target lane line; A deletion attribute, indicating that the jth initial lane line is deleted from the target map; The type change attribute indicates that the i-th target lane line and the j-th initial lane line are lane lines with similar positions, and the i-th target lane line and the j-th initial lane line have different lane line types.

5. The method according to claim 1, wherein: The performing feature fusion on the environment image and the initial map to obtain target fusion features comprises: Extracting features from the environment image to obtain environment image features; Extracting features from the vector points of the initial lane line to obtain initial lane line features; and The environment image features and the initial lane line features are fused based on the attention mechanism to obtain the target fusion features.

6. The method according to claim 5, wherein: The step of fusing the environment image features and the initial lane line features based on the attention mechanism to obtain the target fusion features includes: Fusing the environment image features and the initial lane line features to obtain intermediate fusion features; and The target fusion feature is obtained by fusing the intermediate fusion feature and the preset lane line query feature based on the attention mechanism, and the lane line query feature is related to the attribute of the lane line in the target map.

7. The method according to claim 6, wherein: The intermediate fusion feature and the preset lane line query feature are fused based on the attention mechanism to obtain the target fusion feature, including: fusing the n-1th level query feature and the intermediate fusion feature based on the attention mechanism to obtain the nth level query feature, where the first level query feature is the preset lane line query feature; and In the case of N=n, the target fusion feature is determined based on the Nth level query feature, wherein N≥n>1, and N and n are both integers.

8. The method according to claim 5, wherein: The extracting features of the vector points of the initial lane line to obtain the initial lane line features comprises: Extracting shape and position features of the vector points of the initial lane line to obtain shape and position features of the vector points; Extracting lane line type features from the vector points of the initial lane line to obtain vector point type features; and The vector point shape position features and vector point type features corresponding to each of the initial lane lines are integrated to obtain the initial lane line features.

9. The method according to claim 1, wherein: The environment image includes at least one of the following: A first environment image acquired by a vehicle sensor; The second environment image collected by the roadside equipment.

10. A vehicle control method, comprising: The vehicle driving is controlled according to a target map, wherein the target map is determined according to the method according to any one of claims 1 to 9.

11. A method for training a deep learning model, wherein the deep learning model includes a feature fusion network and a map detection network, the method comprising: Acquire a training sample, wherein the training sample includes a sample environment image related to a sample traffic area, a sample initial map, and a label map, wherein the sample initial map includes a sample initial lane line, and the label map includes a label lane line; Using the feature fusion network to perform feature fusion on the sample environment image and the sample initial map to obtain sample target fusion features; Using the map detection network, updating the sample initial map based on the sample target fusion feature to obtain a sample map, wherein the sample map includes a lane line difference attribute representing a difference between a sample lane line in the sample map and the sample initial lane line; The deep learning model is trained according to the sample map and the label map.

12. The method according to claim 11, wherein: The sample map includes sample lane lines and sample lane line types; Wherein, training the deep learning model according to the sample map and the label map includes: Processing the sample lane line and the label lane line according to a first loss function to obtain a lane line position loss value; Processing the sample lane line type of the sample lane line according to a second loss function to obtain a lane line type loss value, wherein the first loss function and the second loss function are the same or different; The deep learning model is trained based on the lane line type loss value and the lane line position loss value.

13. The method according to claim 12, wherein: Processing the sample lane line type of the sample lane line according to the second loss function to obtain the lane line type loss value includes: The sample lane line type and the dynamic label data determined based on the lane line distance are processed based on the loss function to obtain the lane line type loss value, wherein the lane line distance represents the distance between the sample lane line and the label lane line.

14. The method according to claim 11 or 12, wherein: The training of the deep learning model according to the sample map and the label map comprises: Processing the sample target fusion features and the sample initial lane line features based on the lane difference detection branch of the map detection network to obtain a sample association matrix, wherein the sample matrix elements in the sample association matrix represent the similarity between the i-th sample lane line and the j-th sample initial lane line; and The acquired label difference attribute is used as label data, and the deep learning model is trained based on the sample matrix elements and the label difference attribute, wherein the label difference attribute represents the difference between the i-th label lane line and the j-th sample initial lane line in the label map.

15. A map generating device, comprising: A first acquisition module, used to acquire an environment image and an initial map related to a target passage area, wherein the initial map includes initial lane lines; A fusion module, used for performing feature fusion on the environment image and the initial map to obtain target fusion features; A target map acquisition module is used to update at least one of the initial lane lines based on the target fusion feature to obtain a target map, wherein the target map includes a lane line difference attribute representing the difference between the lane line in the target map and the initial lane line.

16. The device according to claim 15, wherein: The target map acquisition module includes: A target lane line detection submodule is used to perform lane line detection based on the target fusion feature to obtain a target lane line; a lane line difference attribute acquisition submodule, configured to perform lane difference detection based on the target fusion feature and the initial lane line feature to obtain the lane line difference attribute, wherein the initial lane line feature is determined based on the initial map; and The target map determination submodule is used to determine the target map based on the target lane line and the lane line difference attribute.

17. The device according to claim 16, wherein: The lane line difference attribute acquisition submodule includes: a target association matrix obtaining unit, used to fuse the target fusion feature and the initial lane line feature to obtain a target association matrix, wherein the association matrix elements in the target association matrix represent whether the i-th target lane line matches the j-th initial lane line; and A lane line difference attribute determination unit is used to determine the lane line difference attribute based on the association matrix element.

18. The device according to any one of claims 15 to 17, wherein: The lane line difference attribute includes at least one of the following: A new attribute is added to indicate that the initial map does not have the i-th target lane line; A deletion attribute, indicating that the jth initial lane line is deleted from the target map; The type change attribute indicates that the i-th target lane line and the j-th initial lane line are lane lines with similar positions, and the i-th target lane line and the j-th initial lane line have different lane line types.

19. The device according to claim 15, wherein: The fusion module includes: A first feature extraction submodule is used to extract features from the environment image to obtain environment image features; A second feature extraction submodule is used to extract features from the vector points of the initial lane line to obtain initial lane line features; and The fusion submodule is used to fuse the environment image features and the initial lane line features based on the attention mechanism to obtain the target fusion features.

20. The device according to claim 19, wherein The fusion submodule includes: an intermediate fusion feature obtaining unit, used to fuse the environment image feature and the initial lane line feature to obtain an intermediate fusion feature; and The target fusion feature acquisition unit is used to fuse the intermediate fusion feature and the preset lane line query feature based on the attention mechanism to obtain the target fusion feature, and the lane line query feature is related to the attribute of the lane line in the target map.

21. The device according to claim 20, wherein: The target fusion feature obtaining unit comprises: A first fusion subunit is used to fuse the n-1th level query feature and the intermediate fusion feature based on an attention mechanism to obtain an nth level query feature, where the first level query feature is the preset lane line query feature; and The target fusion feature determination subunit is used to determine the target fusion feature based on the Nth level query feature when N=n, wherein N≥n>1, and N and n are both integers.

22. The method according to claim 19, wherein: The second feature extraction submodule comprises: A vector point shape position feature acquisition unit, used to extract shape position features of the vector points of the initial lane line to obtain vector point shape position features; A vector point type feature acquisition unit is used to extract lane line type features from the vector points of the initial lane line to obtain vector point type features; and The initial lane line feature acquisition unit is used to fuse the vector point shape position features and the vector point type features corresponding to each of the initial lane lines to obtain the initial lane line features.

23. The device according to claim 15, wherein: The environment image includes at least one of the following: A first environment image acquired by a vehicle sensor; The second environment image collected by the roadside equipment.

24. A vehicle control device, comprising: A control module, used for controlling vehicle driving according to a target map, wherein the target map is determined according to a method according to any one of claims 1 to 9.

25. A training device for a deep learning model, the deep learning model comprising a feature fusion network and a map detection network, the device comprising: A sample acquisition module, used to acquire training samples, wherein the training samples include a sample environment image related to a sample traffic area, a sample initial map and a label map, wherein the sample initial map includes a sample initial lane line, and the label map includes a label lane line; A sample target fusion feature acquisition module is used to use the feature fusion network to perform feature fusion on the sample environment image and the sample initial map to obtain a sample target fusion feature; A sample map acquisition module, configured to use the map detection network to update the sample initial map based on the sample target fusion feature to obtain a sample map, wherein the sample map includes a lane line difference attribute representing the difference between a sample lane line in the sample map and the sample initial lane line; A training module is used to train the deep learning model according to the sample map and the label map.

26. The device according to claim 25, wherein The sample map includes sample lane lines and sample lane line types; Wherein, the training module includes: A lane line position loss value obtaining submodule, used for processing the sample lane line and the label lane line according to a first loss function to obtain a lane line position loss value; A lane line type loss value obtaining submodule, used for processing the sample lane line type of the sample lane line according to a second loss function to obtain a lane line type loss value, wherein the first loss function and the second loss function are the same or different; The first training submodule is used to train the deep learning model based on the lane line type loss value and the lane line position loss value.

27. The device according to claim 26, wherein: The lane line type loss value obtaining submodule includes: A lane line type loss value obtaining unit is used to process the sample lane line type and the dynamic label data determined based on the lane line distance based on the loss function to obtain the lane line type loss value, wherein the lane line distance represents the distance between the sample lane line and the label lane line.

28. The device according to claim 25 or 26, wherein: The training module includes: a sample association matrix acquisition submodule, for processing the sample target fusion features and the sample initial lane line features based on the lane difference detection branch of the map detection network to obtain a sample association matrix, wherein the sample matrix elements in the sample association matrix represent the similarity between the i-th sample lane line and the j-th sample initial lane line; and The second training submodule is used to use the acquired label difference attribute as label data, and train the deep learning model based on the sample matrix elements and the label difference attribute, wherein the label difference attribute represents the difference between the i-th label lane line and the j-th sample initial lane line in the label map.

29. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 14.

30. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 14.

31. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 14.

Images