Map updating method and device, model training method and device, electronic equipment and medium

Abstract

The present disclosure provides a map updating method and device, a model training method and device, an electronic device and a medium, relates to the technical field of artificial intelligence, and in particular to the fields of autonomous driving, intelligent transportation, computer vision, image processing and the like. The specific implementation scheme is: for the same position information, determining associated historical map data and M acquisition image data based on time sequence, M is an integer greater than or equal to 1; encoding the historical map data and the M acquisition image data respectively to obtain historical map features and M acquisition image features; determining the position information and the category information of the target instance according to the historical map features and the M acquisition image features, the target instance representing lane information; and updating the map data according to the position information and the category information of the target instance.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to fields such as autonomous driving, intelligent transportation, computer vision, and image processing. More specifically, this disclosure provides a map updating method, a map updating model training method, a map updating device, a map updating model training device, an electronic device, a storage medium, and a computer program product. Background Technology

[0002] Autonomous driving maps can provide autonomous vehicles with rich road element data and lane-level path planning, thus ensuring safer and more efficient autonomous driving. However, the coverage and update timeliness of autonomous driving maps have become key issues restricting the development of this field. Summary of the Invention

[0003] This disclosure provides a map updating method, a map updating model training method, a map updating device, a map updating model training device, an electronic device, a storage medium, and a computer program product.

[0004] According to one aspect of this disclosure, a map updating method is provided, comprising: for the same location information, determining associated historical map data and M time-series acquired image data, where M is an integer greater than or equal to 1; encoding the historical map data and the M acquired image data respectively to obtain historical map features and M acquired image features; determining the location information and category information of a target instance based on the historical map features and the M acquired image features, where the target instance represents lane information; and updating the map data based on the location information and category information of the target instance.

[0005] According to another aspect of this disclosure, a method for training a map update model is provided, comprising: determining training samples, the training samples including: a map prior, N time-series-based sample images, and labels; the map prior and the N sample images are all associated with the same location information, the labels characterize whether there is a real instance in the N sample images and characterize the real location information and real category of the real instance, the real instance characterizes lane information, and N is an integer greater than or equal to 1; encoding the map prior and the N sample images respectively to obtain map features and N sample image features; determining the output location information and output category of at least one output instance based on the map features and the N sample image features; and training the map update model based on the real location information, output location information, real category, and output category.

[0006] According to another aspect of this disclosure, a map updating apparatus is provided, comprising: a first data determining module, a first encoding module, a target instance determining module, and an updating module. The first data determining module is used to determine associated historical map data and M time-series acquired image data for the same location information, where M is an integer greater than or equal to 1. The first encoding module is used to encode the historical map data and the M acquired image data respectively to obtain historical map features and M acquired image features. The target instance determining module is used to determine the location information and category information of a target instance based on the historical map features and the M acquired image features, where the target instance represents lane information. The updating module is used to update the map data based on the location information and category information of the target instance.

[0007] According to another aspect of this disclosure, a training apparatus for a map update model is provided, comprising: a sample determination module, a second encoding module, a second data determination module, and a training module. The sample determination module determines training samples, which include: a map prior, N time-series-based sample images, and labels; the map prior and the N sample images are all associated with the same location information; the labels characterize whether a real instance exists in the N sample images and characterize the real location information and real category of the real instance; the real instance characterizes lane information; and N is an integer greater than or equal to 1. The second encoding module encodes the map prior and the N sample images respectively to obtain map features and N sample image features. The second data determination module determines the output location information and output category of at least one output instance based on the map features and the N sample image features. The training module trains the map update model based on the real location information, output location information, real category, and output category.

[0008] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the methods provided in this disclosure.

[0009] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods provided in this disclosure.

[0010] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods provided in this disclosure.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0013] Figure 1 This is a schematic diagram illustrating an application scenario of the training method, map updating method, and apparatus for the map updating model according to embodiments of this disclosure;

[0014] Figure 2 This is a schematic flowchart of a training method for a map update model according to an embodiment of the present disclosure;

[0015] Figure 3A This is a schematic diagram of a lane line representation method according to an embodiment of the present disclosure;

[0016] Figure 3B This is a schematic diagram of a lane group representation method according to an embodiment of the present disclosure;

[0017] Figure 4 This is a schematic diagram illustrating the training method of a map update model according to an embodiment of the present disclosure;

[0018] Figure 5 This is a schematic diagram illustrating the principle of a map updating method according to an embodiment of the present disclosure;

[0019] Figure 6 This is a schematic flowchart of a map update method according to an embodiment of the present disclosure;

[0020] Figure 7 This is a schematic structural block diagram of a training device for a map update model according to an embodiment of the present disclosure;

[0021] Figure 8 This is a schematic structural block diagram of a map updating apparatus according to embodiments of the present disclosure; and

[0022] Figure 9 This is a structural block diagram of an electronic device used to implement the training method and / or map update method of the map update model according to the embodiments of this disclosure. Detailed Implementation

[0023] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0024] In some embodiments, a labor-intensive model can be used to update the map. In this model, the entire process is divided into stages and elements, and trained operators discover and modify the changed map elements.

[0025] The aforementioned labor-intensive model has high costs, low efficiency, and uncertainties in production quality due to the skill level of the workers.

[0026] In other embodiments, maps can be updated based on a combination of general image recognition algorithms and human interactive interpretation. This generally includes two approaches. The first approach involves the model directly providing the modified result, typically a multi-stage generation model based on image segmentation combined with post-processing. A segmentation model is used to obtain the pixel positions of elements such as lines and polygons in the image, and then a post-processing strategy is used to extract vectorized line and polygon information, which is then provided to operators to assist in interactive map generation. The second approach involves a change detection model directly providing a determination of whether points of interest have changed, which is then provided to operators to assist in modification.

[0027] The first approach, combining the aforementioned general image recognition algorithms with human interactive interpretation, suffers from several limitations. First, it is constrained by multi-stage error accumulation, resulting in a funnel effect in its automation capabilities. For instance, after the segmentation model outputs a mask, a series of operations such as fitting, line extraction, and thinning are required to extract lane lines, leading to low accuracy and recall in the recognition results. Second, the change detection model provides coarse information, insufficient for achieving a highly automated, end-to-end model covering all elements.

[0028] This disclosure aims to propose a training method for a map update model and a map update method. In some embodiments, the method uses an existing map as a priori and is trained based on image data acquired by a camera. By constructing an end-to-end generation framework and training paradigm, it achieves vectorized modeling and generation of map elements, thereby avoiding the funnel effect. Furthermore, this method fully integrates existing maps and images, enabling high-fidelity map reconstruction and convenient capture of changes in map elements, achieving end-to-end map updates.

[0029] The technical solutions provided in this disclosure will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0030] Figure 1This is a schematic diagram illustrating an application scenario of the training method, map updating method, and apparatus for the map updating model according to embodiments of this disclosure.

[0031] like Figure 1 As shown, the application scenario 100 of this embodiment may include an electronic device 110, which may be any electronic device with processing capabilities, including but not limited to smartphones, tablets, laptops, desktop computers, and servers.

[0032] The electronic device 110 can, for example, process the input acquired image 140 and historical map data 150 to obtain a target instance 170, thereby using the target instance 170 to update the historical map data 150.

[0033] According to embodiments of this disclosure, such as Figure 1 As shown, the application scenario 100 may also include a server 120. The electronic device 110 can communicate with the server 120 via a network, which may include a wireless or wired communication link.

[0034] For example, server 120 can be used to train map update model 160, and in response to a model acquisition request sent by electronic device 110, it sends the trained map update model 160 to electronic device 110, so that electronic device 110 can perform map update operations. In one embodiment, electronic device 110 can also send the acquired image 140 to server 120 via a network, and the server updates the map according to the trained map update model 160.

[0035] According to embodiments of this disclosure, such as Figure 1 As shown, the application scenario 100 may also include a database 130, which can maintain a massive number of training samples. Each training sample may include a sample image, map priors, and labels. The server 120 can access the database 130 and extract a portion of the training samples from the database 130 to train the map update model 160.

[0036] When training the map update model 160, the loss of the map update model can be determined by using a loss function based on the output instances and labels, and the model training can be completed by minimizing the model loss.

[0037] It should be noted that the training method for the map update model provided in this disclosure can be executed by server 120, and the map update method provided in this disclosure can be executed by electronic device 110 or server 120. Accordingly, the training device for the map update model provided in this disclosure can be located in server 120, and the map update device provided in this disclosure can be located in electronic device 110 or server 120.

[0038] It should be understood that Figure 1 The number and types of electronic devices, servers, and databases shown are merely illustrative. Depending on implementation needs, any number and type of electronic devices, servers, and databases can be included.

[0039] Figure 2 This is a schematic flowchart of a training method for a map update model according to an embodiment of the present disclosure.

[0040] like Figure 2 As shown, the training method 200 for the map update model may include operations S210 to S240.

[0041] In operation S210, training samples are determined, which include: map prior, N time-series sample images and labels, where N is an integer greater than or equal to 1.

[0042] In operation S220, the map prior and N sample images are encoded respectively to obtain map features and N sample image features.

[0043] In operation S230, based on map features and N sample image features, the output location information and output category of at least one output instance are determined.

[0044] When operating S240, a map update model is trained based on the real location information, output location information, real category, and output category.

[0045] For example, multiple training samples can be pre-constructed, which can be based on manual annotation. Each training sample may include a map prior and at least one sample image. The map prior represents map information for a certain geographic area, and the at least one sample image is captured using a camera located in that geographic area. Thus, the map prior and the N sample images are all associated with the same location information. The sample images may include at least one real instance or may not include a real instance. The real instance represents lane information, which may include line features and polygon features. Line features may include lane lines, such as single solid lines and double yellow lines. Polygon features may include lane groups and zebra crossings. For example, if a sample image includes two lane lines, then the sample image includes two real instances.

[0046] Each training sample can also include a label, which indicates whether a real instance exists in the N images. Furthermore, if a real instance exists in the sample image, the label also represents the real instance's location information and real category. A real instance can be represented by multiple points; therefore, the real location information can include the coordinates of multiple points, with each point's coordinates corresponding to a real sub-location. The real category can include a single solid white line, a dashed white line, a double yellow line, etc.

[0047] For example, a map prior and N sample images are input into a map update model to be trained. The map update model performs encoding, feature fusion, and decoding operations, and then outputs a prediction result. The prediction result can include the output location information and output category of the output instance. The output location information is similar to the real location information, and can include the coordinates of only one point or multiple points. The output category can be referenced from the real category.

[0048] Next, based on the differences between the real and output location information, and the differences between the real and output categories, a predetermined loss function is used to calculate the total loss function value. The network weights in the map update model are then adjusted using the backpropagation algorithm to complete the training of the map update model. The predetermined loss function may include cross-entropy loss functions, etc., and this disclosure does not limit its application.

[0049] This disclosure employs a map update model trained based on sample images and map priors, enabling end-to-end vectorized map feature generation. This method alters the conventional automated map update paradigm by encoding and learning map priors to ensure the fidelity of map reconstruction. Furthermore, it directly uses sequential sample images as input, reducing process steps and improving update efficiency.

[0050] Before training the map update model, training samples need to be built in advance. This embodiment adopts a scheme of directly converting the results of manual operations into the training data structure. The process of automatically building training samples will be described in detail below.

[0051] First, we will introduce how to obtain sample images in the training samples. For example, sample images, location information, and equipment parameters can be obtained based on a self-collecting vehicle. Where N represents the total number of time-series sample images. Pos represents the i-th sample image based on time sequence. i This indicates the camera's position information when acquiring sample images. i This represents the parameters of the camera used to acquire sample images. It should be noted that a single sample image can be a panoramic image, in which case a single sample image can include K images from different perspectives at the same location. For example, acquiring 6 images from different perspectives at the first moment constitutes the first frame of sample images, and then acquiring 6 images from different perspectives at the second moment constitutes the second frame of sample images. In other embodiments, a single sample image may also include only one image from a single perspective. The above describes the methods for obtaining sample images from the training samples.

[0052] Then, based on the location information Pos mentioned above iExtract the location information Pos from the pre-built database. i The associated map data in the database consists of pre-marked data (e.g., lane lines, lane groups) by production workers. It's important to note that the map data retrieved from the database includes both new and old versions. The new version matches the sample images, while the old version may not. For example, if a sample image has four lane lines, the new version of the map data must also include those four lane lines, while the old version may only include three. Training samples include sample images, map priors, and labels. The sample images are captured by a camera. The new version of the map data is used to determine the labels, and the old version is used to determine the map priors in the training samples.

[0053] Next, the process of determining labels based on the new version of map data will be explained.

[0054] For the new version of map data, the manual work on lane lines can be extracted first based on geometric (e.g., the coordinates of multiple points on a lane line) and style (e.g., straight line, curve, white, yellow) categories. This process yields some raw vectorized point sets and the corresponding style and color information for each set. Each point set represents the location information of an instance, and the style and color information corresponding to that set represents the instance's category. Thus, the raw vectorized point sets can be obtained. and categories Where, N ori This represents the number of real instances in a single frame of a sample image. represents the actual class of a single instance, and c is the total number of actual classes.

[0055] Obtain the original vectorized point set After that, we can also perform analysis on each point set P. i ∈P ori Fitting and interpolation are performed to obtain a uniform set of fixed numbers of points. Where N p This represents the number of points. If the number of instance pixels is less than N... p Then, the actual number of pixels can be taken. Considering the consistency of model building, this embodiment fixes the number of instances per sample image to N. ins The number N ins This is a preset value, which can be greater than the number of instances of a regular image, for example, N. ins Take 50. For instances with fewer than N... ins The image can be padded to complete the point set P. ori The filled part is represented as This yields the point set representation of a single image. and the corresponding categories Each instance's label includes the location information represented by the point set and the true category corresponding to the point set. Therefore, the label (Ground True, GT) of each instance can be represented as Y. i =(P i L i It should be noted that the above point set representation... In this process, part of the point set is obtained through padding, while the other part of the point set already exists before padding. The label needs to indicate whether the point set is obtained through padding so that the loss function value can be accurately calculated later.

[0056] The above describes how to obtain labels from training samples. Next, we will explain the process of determining map priors based on older versions of map data.

[0057] For older versions of map data, a similar approach to processing newer versions is used to obtain the point sets and categories from the older data. Next, the point sets from the older map data need to be visually represented and processed into a modality easily accepted by the map update model. One approach is to map the point sets to masks. The point set is mapped to a mask with a predetermined pixel value as the background, according to its category and width h. The value range is 0, 1, ..., c, for example, 1 represents a single solid line, 2 represents a double yellow line, and H... gt and W gt These represent the height and width of the map prior, respectively, and the predetermined pixel value can be 0. For example, a sample image includes 3 lane lines, each lane line is a real instance, each lane line corresponds to a category, and each lane line is represented by a set of 50 points. For each real instance, the 50 points of that real instance are mapped onto a mask with a black background, and the width h of each point is a preset value. This results in 3 lane lines on the mask with a black background, from which the map prior can be obtained.

[0058] The above section explained how to obtain the sample images, map priors, and labels in the training samples. Next, we will explain how to represent lane information.

[0059] In this embodiment, lane information includes line features and area features. Line features may include lane lines, and area features may include lane groups. Each individual lane information can be represented using multiple points.

[0060] Figure 3A This is a schematic diagram of a lane line representation method according to an embodiment of the present disclosure. Figure 3BThis is a schematic diagram of a lane group representation method according to an embodiment of the present disclosure. In the diagram, the dots with black backgrounds represent starting points, and the arrows represent directions.

[0061] like Figure 3A As shown, the set of points corresponding to the lane lines includes multiple points arranged in a linear fashion, and since the lane lines do not need to distinguish directions, the consistency between the start and end points does not need to be emphasized. Figure 3A Both equivalent permutations 311 and 312 can represent lane lines.

[0062] like Figure 3B As shown, the point set corresponding to a lane group includes multiple points, and the order of these points does not affect the instance representation. Therefore, a lane group consisting of k points can be represented by 2×k equivalent permutations. For example... Figure 3B A lane group is represented by 3 points. There are 6 ways to represent a lane group (321-326). Any one of the representation methods can be used to represent a lane group.

[0063] The structure and training algorithm of the map update model will be introduced next.

[0064] In this embodiment, the map update model may include an image encoder E. bev Map Prior Encoder E map The system includes a confidence graph network and a decoder. Multiple task modules can also be connected to the output of the decoder. The image encoder can be a BEV encoder.

[0065] First, obtain pre-built training samples, including map priors. and N sample images based on time series Map Priority Input map prior encoder E map Map Prior Encoder E map Mapping it to the feature space yields the map feature F. map Input the sample image into the image encoder E. bev Image encoder E bev Mapping it to the feature space yields the sample image features.

[0066] Next, based on the N sample image features, N second confidence maps can be determined, each corresponding to one of the N sample image features. For example, the N sample image features... Input the confidence graph network to obtain N second confidence graphs. Sample image features and second confidence level Figure 1Each sample image feature is corresponding to and of the same size. Each sample image feature includes multiple sub-features. Each sub-feature can represent a pixel or multiple points in a region of the sample image. Correspondingly, each second confidence map includes multiple confidences. The multiple confidences in the second confidence map represent the weights of multiple sub-features in the corresponding sample image feature. For example, the first confidence represents the weight of the first sub-feature.

[0067] Next, based on the features of N sample images, N second confidence maps, and map features, the output location information and output category of at least one output instance can be determined.

[0068] In one embodiment, N sample image features and N second confidence maps can be fused to obtain a second intermediate fused feature F. seguence The fusion process may include: performing dot product on the corresponding sample image features and the second confidence map, and then summing the dot product frame by frame to obtain the fused second intermediate fusion feature F. seuence For example, a sample image is acquired at multiple time points, and each sample image corresponds to a sample image feature. A confidence map network learns the importance of each sub-feature within each sample image feature. For instance, points near lane lines in a sample image are considered more important; if a point is highly important, the confidence map network outputs a larger weight for that point. Then, a second confidence map is output based on the confidence map network, and multiple sample image features are fused into a second intermediate fused feature F based on this second confidence map. seuence .

[0069] Then, based on the second intermediate fusion feature F sequence and map features F map Determine the second target fusion feature F all =Cat(F sequence F map For example, the second intermediate fusion feature F can be used. sequence and map features F map By stitching along the channel dimension, the second target fusion feature F is obtained. all =Cat(F sequence F map For example, the intermediate fusion feature F sequence The dimensions are C1*H*W, and the map feature is F. map The dimensions are C2*H*W, and the second target fusion feature F after splicing is... all =Cat(F sequence F map The dimensions of the array are (C1+C2)*H*W, where C represents the number of channels, H represents the height, and W represents the width.

[0070] Then the second target feature F can be fused.all =Cat(F sequence F map Decoding yields the output location information and output category of at least one output instance. For example, the second target fusion feature F can be used. all =Cat(F sequence F map The input is fed into the decoder to obtain the output location information and output class confidence of each output instance. The decoder can include multiple network layers (transformer layers) that learn instance-level and point-level contextual information, combining a multi-task learning framework of semantic segmentation and instance segmentation to obtain the output results. in This indicates the output location information of the output instance. Indicates multiple output categories, This represents the confidence level of each output category. The output category with the highest confidence level can be used as the output category determined by the model.

[0071] After obtaining the output location information, output category, and confidence level of each output instance, the total loss function value can be calculated based on the differences between the output location information and the true location information, as well as the differences between the output category and the true category. The parameters of the map update model can be adjusted by minimizing the total loss function value.

[0072] This embodiment considers the consistency of multi-frame, multi-angle sample images and learns point-level confidence maps, enabling adaptive fusion of multi-frame images to improve generation quality. Furthermore, improving visual feature quality through temporal information effectively mitigates issues such as occlusion and irregular truncation. By learning intuitive visual representations and manually defined rule representations, multi-stage processes involving multiple steps and elements are avoided. This enhances end-to-end automation capabilities across all elements and addresses the funnel effect.

[0073] Furthermore, fusing multiple sample images based on the second confidence map can make the sub-features with larger weights have a greater impact on the second intermediate fused features, thereby improving the representation accuracy of the second target fused features and thus improving the model training effect.

[0074] It should be noted that in other embodiments, the confidence graph network can be omitted, and multiple sample image features can be directly fused. For example, corresponding sub-features can be fused to obtain fused features. Then, the fused features are fused with map features again, and then input into the decoder together.

[0075] It should be noted that during the training process, the output of the map update model includes multiple output instances. Therefore, the model output information and labels can be matched to obtain matching relationships. Based on the matching relationships, the total loss function value can be accurately calculated to ensure the model iteration effect.

[0076] In one example, the matching relationship includes a first matching relationship, which represents the matching relationship between the real instance and the output instance, that is, which real instance the output instance matches.

[0077] The first matching relationship can be determined as follows: At least one real instance is permuted to obtain a sequence of real instances. At least one output instance is also permuted and combined to obtain multiple candidate instance sequences. Then, for each candidate output instance sequence, a first difference evaluation value is determined based on the output position information and output category of each output instance in the candidate output instance sequence, the real position information and real category of each real instance in the real instance sequence, the order of the real instance sequence, and the order of each candidate instance sequence. Thus, multiple candidate instance sequences correspond to multiple first difference evaluation values. The first matching relationship is then determined based on the minimum first difference evaluation value.

[0078] For example, using Let represent a set of candidate instance sequences. This set includes multiple candidate instance sequences, each containing multiple output instances arranged in a specific order. The order of the instances in the multiple candidate instance sequences is distinct from each other. An instance pair is formed by combining the i-th real instance in the real instance sequence and the i-th output instance in the candidate instance sequence. The sum of the differences between these instance pairs is then used to determine the first difference evaluation value. The candidate instance matching sequence corresponding to the minimum first difference evaluation value is then defined. It can be represented as:

[0079]

[0080] Among them, cost ins The loss term representing the matching between instances includes at least one of a class loss function and a geometric loss function, as shown in the following formula with cost. ins Examples include categorical loss functions and geometric loss functions.

[0081]

[0082] Among them, cost cls To represent the category loss function, focal loss can be used. geoThis represents the geometric loss function, which measures the geometric correlation between the point set of output instances and the point set of real instances. The Hungarian matching algorithm can be used as the geometric loss function; this embodiment does not limit the choice between the category loss function and the geometric loss function.

[0083] This embodiment establishes a first matching relationship, which allows the total loss function value to be calculated based on the first matching relationship, thereby improving the model training effect and ensuring the model inference accuracy.

[0084] In another example, the real location information of the same real instance includes multiple real sub-location information. Similarly, the output location information of the same output instance includes multiple output sub-location information. It can be understood that both the real instance and the output instance are represented by multiple points, each real sub-location information representing the coordinates of a point in the real instance, and each output sub-location information representing the coordinates of a point in the output instance. The matching relationship includes a second matching relationship, which characterizes the matching relationship between the real sub-location information and the output sub-location information, i.e., which point in the real instance matches a point in the output instance.

[0085] The second matching relationship can be determined as follows: For output instances and real instances that satisfy the first matching relationship, the multiple real sub-location information of the real instances is arranged to obtain a real point sequence. The multiple output sub-location information of the output instances is also arranged and combined to obtain multiple candidate point sequences. Then, based on the multiple real sub-location information, the multiple output sub-location information, the order of the real point sequences, and the order of the candidate point sequences, a second difference evaluation value is determined for each candidate point sequence, thus multiple candidate point sequences correspond to multiple second difference evaluation values. Finally, the second matching relationship is determined based on the minimum second difference evaluation value.

[0086] For example, using This represents a set of candidate point sequences, comprising multiple candidate point sequences, each containing multiple points from an output instance. The order of the points in the candidate point sequences is distinct from each other. A point pair is formed by combining the i-th point from the true point sequence and the i-th point from the candidate point sequence. The sum of the differences between these point pairs is then used to determine the second difference evaluation value. The candidate point sequence corresponding to the minimum second difference evaluation value is... It can be represented as:

[0087]

[0088] Among them, cost point This represents the matching loss term between points in the output instance and points in the real instance, that is, the matching loss between the output sub-location information and the real sub-location information. The Manhattan distance can be used as the function of the matching loss. This embodiment does not limit the function of the matching loss.

[0089] This embodiment determines the second matching relationship, which allows the total loss function value to be calculated based on the second matching relationship, thereby improving the model training effect and ensuring the model inference accuracy.

[0090] The preceding text introduced the first matching relationship at the instance level and the second matching relationship at the point level within an instance. In practical applications, the matching relationship includes at least one of the first and second matching relationships. After obtaining the matching relationships, the total loss function value can be determined based on the matching relationships, ground truth location information, output location information, ground truth class, and output class. Then, the map update model is trained based on the total loss function value.

[0091] It should be noted that in other embodiments, other methods can be used to determine the above-mentioned matching relationship. For example, the first matching relationship can be determined based on the relative positional relationship of each output instance and the relative positional relationship of each real instance. Alternatively, based on the output position information of the output instances and the real position information of the real instances, multiple output instances and multiple real instances can be numbered respectively in a top-down and left-to-right manner, and output instances and real instances with the same number can be determined to satisfy the first matching relationship.

[0092] The loss function used in the model training process will be explained below.

[0093] In this embodiment, multiple predetermined loss function values ​​can be calculated, and then the total loss function value can be calculated based on the multiple predetermined loss functions. The multiple predetermined loss function values ​​may include point-level geometric loss function values, instance-level geometric loss function values, category loss function values, binary classification loss function values, semantic segmentation loss function values, and instance segmentation loss function values. The following describes each predetermined loss function.

[0094] For the point-level geometric loss function, the output sub-position information and the true sub-position information that satisfy the second matching relationship can be combined into a point pair, and the sum of the distances between each point pair can be calculated to determine the point-level geometric loss function value.

[0095] For example, the point-level geometric loss function is as follows:

[0096]

[0097] Where dist represents the distance between points, which can be the Manhattan distance. N ins and N p L represents the number of instances and the number of points within a single instance, respectively. i Indicates the instance category.

[0098] For instance-level geometric loss function values, the values ​​can be determined based on the differences between the first and second line segments in a line segment pair. The first line segment has two output sub-position information as its endpoints, and the second line segment has two real sub-position information as its endpoints. In the same line segment pair, the two endpoints of the first line segment and the two endpoints of the second line segment satisfy a second matching relationship.

[0099] For example, the instance-level geometric loss function is as follows:

[0100]

[0101] Where cosine represents similarity, for example, cosine similarity can be used, and link represents the line segment connecting two points. The line segment can be the line connecting two adjacent points. For example, if a real instance includes 50 real sub-location information, then 49 line segments can be determined. The output instance is similar, and will not be elaborated here. N ins and N p L represents the number of instances and the number of points within a single instance, respectively. i Indicates the instance category.

[0102] For the category loss function, the function value can be calculated as follows: For output instances and real instances that satisfy the first matching relationship, the category loss function value is determined based on the difference between the output category of the output instance and the real category of the real instance.

[0103] For example, the category loss function is as follows:

[0104]

[0105] in To represent loss, for example, using focal loss. and L i These represent the output class and the true class, respectively. The class probability can be used as the confidence level; for classes with confidence levels above a threshold, these are used as the output class. Furthermore, this embodiment incorporates the positional relationships of instance geometry into the class constraints for task alignment, thus allowing the confidence level to simultaneously characterize both geometry and class.

[0106] Furthermore, due to the complexity of road conditions in natural scenes, such as ground wear, occlusion, and sudden changes in lighting, map update models are prone to erroneously outputting redundant and inconsistent feature vector results. Therefore, to mitigate this issue from a model perspective, this embodiment introduces a binary classification loss function as a constraint.

[0107] For the binary classification loss function, the function value can be calculated as follows: For each output instance, determine the binary classification loss function value based on whether there is a real instance that satisfies the first matching relationship with the output instance. That is, classify the predicted instances that do not match a real instance as invalid, and classify the predicted instances that match a real instance as valid. The valid and invalid classes correspond to different binary classification loss function values.

[0108] For example, the binary classification loss function is as follows:

[0109]

[0110]

[0111] in, L represents the binary cross-entropy loss function. bce(i) Represents the truth value, which is either 0 or 1, where L represents the output instance that matches the true instance. bce(i) The value is 1, indicating that the output instance L did not match a real instance. bce(i) It is 0.

[0112] This embodiment also introduces semantic segmentation and instance segmentation tasks. Both tasks achieve feature-level constraints by determining the segmentation mask of map elements, thereby improving the model's prediction accuracy and recall.

[0113] For the semantic segmentation loss function, the output position information of at least one output instance and the real position information of at least one real instance can be processed based on semantic segmentation to obtain a first output mask and a first real mask respectively; the semantic segmentation loss function value is determined according to the difference between the first output mask and the first real mask.

[0114] For instance segmentation loss function, the output position information of at least one output instance and the real position information of at least one real instance can be processed based on instance segmentation to obtain a second output mask and a second real mask respectively; the instance segmentation loss function value is determined based on the difference between the second output mask and the second real mask.

[0115] For example, the semantic segmentation loss function and the instance segmentation loss function are as follows:

[0116]

[0117]

[0118] in M and represent the output mask and the ground mask, respectively. The subscript 'sem' indicates semantic segmentation, and the subscript 'ins' indicates instance segmentation. This represents the binary cross-entropy loss. This represents the damage function (Dice loss).

[0119] After obtaining the point-level geometric loss function value, instance-level geometric loss function value, category loss function value, binary classification loss function value, semantic segmentation loss function value, and instance segmentation loss function value, the weighted sum of the above predetermined loss function values ​​can be calculated as the total loss function.

[0120] For example, the total loss function is as follows:

[0121]

[0122] Where λ represents the weighting coefficient.

[0123] This embodiment uses the above six predetermined loss functions to calculate the total loss function, thereby constraining the training of the map update model from multiple dimensions, so as to make the output results of the map update model more accurate and more consistent.

[0124] It should be noted that the above text introduced six predetermined loss functions. In practical applications, one to six predetermined loss functions can be selected arbitrarily, and the weighted sum of the selected predetermined loss function values ​​can be calculated as the total loss function value.

[0125] It should be noted that when calculating the above six predetermined loss functions, functions such as Manhattan distance, cosine similarity, focus loss, binary cross-entropy loss function, and damage function are used. In practical applications, these functions can be adjusted according to actual needs. This embodiment does not limit the functions used.

[0126] It's important to note that the difference between semantic segmentation and instance segmentation is as follows: For example, consider three lane lines, and the first and second lane lines belong to the same category. In semantic segmentation, the first and second lane lines are within the same mask. In instance segmentation, however, the first and second lane lines are within two different masks.

[0127] Next, combine Figure 4 This paper introduces the structure and training algorithm of the map update model.

[0128] Figure 4 This is a schematic diagram illustrating the training method of a map update model according to an embodiment of the present disclosure.

[0129] like Figure 4As shown, the map update model may include an image encoder 407, a map prior encoder 409, a confidence graph network 408, and a decoder 410. Multiple task modules may also be added to the output of the decoder 410. These multiple task modules may include, for example, a regression module 411, a classification module 412, and a segmentation module 413. The BEV encoder may be used as the image encoder 407.

[0130] First, pre-constructed training samples are obtained, including a map prior 405 and N time-series sample images 401. The map prior 405 is input into a map prior encoder 409, which outputs map features 406. The N sample images 401 are input into an image encoder 407, which outputs N sample image features 402.

[0131] Next, the N sample image features 402 are input into the confidence map network 408 to obtain N second confidence maps 403. The sample image features 402 and the second confidence maps 403 are in one-to-one correspondence and have the same size. Then, the corresponding sample image features 402 and second confidence maps 403 can be multiplied, and the multiplication is performed frame by frame to obtain the fused second intermediate fused features 404.

[0132] Then, the second intermediate fusion feature 404 and the map feature 406 are stitched together along the channel dimension to obtain the second target fusion feature.

[0133] The second target fusion feature is then input into the decoder 410 to obtain the output location information 407 of each output instance and the output category and the confidence of each output category.

[0134] The regression module 411 can determine the point set corresponding to each output instance based on the output location information 407, in order to calculate the point-level geometric loss function value and the instance-level geometric loss function value. The classification module 412 is used to perform category classification in order to calculate the category loss function value. The classification module 412 is also used to perform valid and invalid binary classification in order to calculate the binary classification loss function value. The segmentation module 413 is used to perform semantic segmentation processing and instance segmentation processing in order to calculate the semantic segmentation loss function value and the instance segmentation loss function value. Using the above multiple predetermined loss function values, the total loss function value can be calculated, and then the parameters of the map update model can be updated. For example, the parameters of the image encoder 407, the map prior encoder 409, the confidence graph network 408, and the decoder 410 can be updated.

[0135] Figure 5 This is a schematic diagram illustrating the principle of a map update method according to an embodiment of the present disclosure.

[0136] like Figure 5As shown, the map update method includes the following operations: For the same location information, it determines the associated historical map data 502 and M time-series acquired image data 501, where M is an integer greater than or equal to 1, and M can be the same as or different from N mentioned above. Then, it inputs the historical map data 502 and the M acquired image data 501 into the trained map update model 503 to obtain the location information and category of the target instance. Then, it updates the historical map data according to the location information and category of the target instance, thereby obtaining the updated map data 504.

[0137] In this embodiment, the map update model 503 described above can be trained using the training method introduced above. The data processing procedure of the map update model 503 may include: encoding the historical map data 502 and the M acquired image data 501 respectively to obtain historical map features and M acquired image features, and then determining the location information and category information of the target instance based on the historical map features and the M acquired image features.

[0138] The following combination Figure 6 This section provides a detailed explanation of the map update method.

[0139] Figure 6 This is a schematic flowchart of a map update method according to an embodiment of the present disclosure.

[0140] like Figure 6 As shown, the map update method 600 may include operations S610 to S640.

[0141] In operation S610, for the same location information, the associated historical map data and M time-series acquired image data are determined, where M is an integer greater than or equal to 1.

[0142] In operation S620, historical map data and M acquired image data are encoded respectively to obtain historical map features and M acquired image features.

[0143] During operation of S630, the location and category information of the target instance are determined based on historical map features and M acquired image features. The target instance represents lane information.

[0144] When operating the S640, map data is updated based on the location and category information of the target instance.

[0145] For example, a trained map update model includes an image encoder, a map prior encoder, and a decoder. Historical map data and M acquired image data can be input into the image encoder and the map prior encoder, respectively, to obtain historical map features and M acquired image features. Then, some fusion processing can be performed on the historical map features and the M acquired image features, and then input into the decoder to obtain the location and category information of the target instance.

[0146] It should be noted that the number of target instances can be one or more. The location information of each target instance includes the coordinates of multiple points, and the category information can include single solid lines, double yellow lines, etc. The location information can be mapped to historical map data, and each point can be displayed in the form of a category indicator for that target instance. Multiple points can also be fitted into a line to achieve the effect of updating the map.

[0147] This disclosure uses existing historical map data as prior art and acquired images as input to achieve end-to-end geographic feature generation based on a map update model. These features include, but are not limited to, lines (lane lines), polygons (zebra crossings), and manually defined rules (lane groups). Using existing historical map data as prior art enables high-fidelity map reconstruction. By directly generating vectorized target results from acquired images as input, the visual representation of complex geographic scenes can be fully learned, and a series of preprocessing and post-processing operations can be avoided, thereby improving the final recognition effect.

[0148] According to another embodiment of this disclosure, the map update model further includes a confidence map network. Accordingly, the process of determining the location and category information of the target instance based on historical map features and M acquired image features may include: determining M first confidence maps corresponding to the M acquired image features. For example, the M acquired image features are input into the confidence map network, and the network outputs M first confidence maps. It should be noted that, similar to the training process, each acquired image feature includes multiple sub-features, and each first confidence map includes multiple confidence levels. The multiple confidence levels in the first confidence map represent the weights of the multiple sub-features in the corresponding acquired image features. Subsequently, the location and category information of the target instance can be determined based on the M acquired image features, the M first confidence maps, and historical map features.

[0149] In one example, M acquired image features and M first confidence maps can be fused to obtain a first intermediate fused feature. For example, the corresponding image features and second confidence maps can be multiplied, and then the multiplied features are added frame by frame to obtain the fused first intermediate fused feature. Next, a first target fused feature can be determined based on the first intermediate fused feature and historical map features. For example, the first intermediate fused feature and historical map features can be concatenated along the channel dimension to obtain the first target fused feature. Then, the first target fused feature is decoded to obtain the location information and category information of the target instance. For example, the first target fused feature can be input into the decoder to obtain the output location information and output category confidence of each target instance.

[0150] This embodiment determines the weights of sub-features through a confidence graph network and fuses multiple sample images based on the second confidence graph. This allows sub-features with larger weights to have a greater impact on the first intermediate fused feature, thereby improving the representation accuracy of the first target fused feature and thus improving the model training effect.

[0151] Figure 7 This is a schematic structural block diagram of a training apparatus for a map update model according to an embodiment of the present disclosure.

[0152] like Figure 7 As shown, the map updating device 700 may include a first data determination module 710, a first encoding module 720, a target instance determination module 730, and an updating module 740.

[0153] The first data determination module 710 is used to determine the associated historical map data and M time-series acquired image data for the same location information, where M is an integer greater than or equal to 1.

[0154] The first encoding module 720 is used to encode the historical map data and the M acquired image data respectively to obtain the historical map features and the M acquired image features.

[0155] The target instance determination module 730 is used to determine the location and category information of the target instance based on historical map features and M acquired image features. The target instance represents lane information.

[0156] The update module 740 is used to update map data based on the location and category information of the target instance.

[0157] In this embodiment, the target instance determination module includes a first determination submodule and a second determination submodule. The first determination submodule is used to determine M first confidence maps corresponding to each of the M acquired image features, based on M acquired image features. Each acquired image feature includes multiple sub-features, and each first confidence map includes multiple confidence levels. The multiple confidence levels in the first confidence map represent the weights of the multiple sub-features in the corresponding acquired image feature. The second determination submodule is used to determine the location information and category information of the target instance based on the M acquired image features, the M first confidence maps, and historical map features.

[0158] In this embodiment, the second determining submodule includes: a first fusion unit, a first determining unit, and a first decoding unit. The first fusion unit is used to fuse M acquired image features and M first confidence maps to obtain first intermediate fusion features. The first determining unit is used to determine first target fusion features based on the first intermediate fusion features and historical map features. The first decoding unit is used to decode the first target fusion features to obtain the location information and category information of the target instance.

[0159] Figure 8 This is a schematic structural block diagram of a map updating apparatus according to an embodiment of the present disclosure.

[0160] like Figure 8 As shown, the training device 800 for the map update model may include a sample determination module 810, a second encoding module 820, a second data determination module 830, and a training module 840.

[0161] The sample determination module 810 is used to determine training samples, which include: map prior, N time-series sample images and labels; the map prior and the N sample images are all associated with the same location information, the labels represent whether there are real instances in the N sample images and represent the real location information and real category of the real instances, the real instances represent lane information, and N is an integer greater than or equal to 1.

[0162] The second encoding module 820 is used to encode the map prior and N sample images respectively to obtain map features and N sample image features.

[0163] The second data determination module 830 is used to determine the output location information and output category of at least one output instance based on map features and N sample image features.

[0164] The training module 840 is used to train the map update model based on the real location information, output location information, real category, and output category.

[0165] In this embodiment, the second data determination module includes a third determination submodule and a fourth determination submodule. The third determination submodule is used to determine N second confidence maps corresponding to each of the N sample image features, based on the N sample image features. Each sample image feature includes multiple sub-features, and each second confidence map includes multiple confidence levels. The multiple confidence levels in the second confidence map represent the weights of the multiple sub-features in the corresponding sample image feature. The fourth determination submodule is used to determine the output location information and output category of at least one output instance based on the N sample image features, the N second confidence maps, and map features.

[0166] In this embodiment, the fourth determination submodule includes: a second fusion unit, a second determination unit, and a second decoding unit. The second fusion unit is used to fuse N sample image features and N second confidence maps to obtain second intermediate fusion features. The second determination unit is used to determine second target fusion features based on the second intermediate fusion features and map features. The second decoding unit is used to decode the second target fusion features to obtain output location information and output category of at least one output instance.

[0167] In this embodiment, the training module includes a matching submodule, a total loss determination submodule, and a training submodule. The matching submodule determines matching relationships, which include at least one of a first matching relationship and a second matching relationship. The first matching relationship represents the matching relationship between real instances and output instances. The real location information of the same real instance includes multiple real sub-location information, and the output location information of the same output instance includes multiple output sub-location information. The second matching relationship represents the matching relationship between real sub-location information and output sub-location information. The total loss determination submodule determines the total loss function value based on the matching relationships, real location information, output location information, real category, and output category. The training submodule trains the map update model based on the total loss function value.

[0168] In this embodiment, the matching submodule includes: a first arrangement unit, a second arrangement unit, a first difference determination unit, and a first relationship determination unit. The first arrangement unit arranges at least one real instance to obtain a sequence of real instances. The second arrangement unit arranges and combines at least one output instance to obtain multiple candidate instance sequences. The first difference determination unit, for each candidate output instance sequence, determines a first difference evaluation value based on the output position information and output category of each output instance in the candidate output instance sequence, the real position information and real category of each real instance in the real instance sequence, and the order of the real instance sequence and the order of each candidate instance sequence. The first relationship determination unit determines a first matching relationship based on the minimum first difference evaluation value among the multiple first difference evaluation values ​​for the multiple candidate instance sequences.

[0169] In this embodiment, the matching submodule includes a second difference determination unit and a second relationship determination unit. The second difference determination unit is used to, for output instances and real instances that satisfy a first matching relationship, arrange multiple real sub-location information of the real instances to obtain a real point sequence; arrange and combine multiple output sub-location information of the output instances to obtain multiple candidate point sequences; and determine multiple second difference evaluation values ​​for the multiple candidate point sequences based on the multiple real sub-location information, the multiple output sub-location information, the order of the real point sequences, and the order of the candidate point sequences. The second relationship determination unit is used to determine a second matching relationship based on the minimum second difference evaluation value among the multiple second difference evaluation values.

[0170] In this embodiment, the total loss determination submodule includes: a total loss determination unit, used to determine a total loss function value based on a predetermined loss function value; wherein the process of determining the predetermined loss function value includes at least one of the following: determining a point-level geometric loss function value based on the distance between output sub-position information and real sub-position information that satisfy a second matching relationship; determining an instance-level geometric loss function value based on the difference between a first line segment and a second line segment in a line segment pair; wherein the first line segment has two output sub-position information as endpoints, and the second line segment has two real sub-position information as endpoints; in the same line segment pair, the two endpoints of the first line segment and the two endpoints of the second line segment satisfy a second matching relationship; for output instances and real instances that satisfy the first matching relationship, based on the output of the output instance... The difference between the category and the true category of the real instance is used to determine the category loss function value; for each output instance, the binary classification loss function value is determined based on whether there is a real instance that satisfies a first matching relationship with the output instance; based on semantic segmentation, the output position information of at least one output instance and the real position information of at least one real instance are processed to obtain a first output mask and a first real mask, respectively; based on the difference between the first output mask and the first real mask, the semantic segmentation loss function value is determined; and based on instance segmentation, the output position information of at least one output instance and the real position information of at least one real instance are processed to obtain a second output mask and a second real mask, respectively; based on the difference between the second output mask and the second real mask, the instance segmentation loss function value is determined.

[0171] According to embodiments of this disclosure, this disclosure also provides an electronic device, including at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described method.

[0172] According to embodiments of this disclosure, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the above-described method.

[0173] According to embodiments of this disclosure, this disclosure also provides a computer program product, including a computer program that implements the above-described method when executed by a processor.

[0174] Figure 9 This is a structural block diagram of an electronic device used to implement the training method and / or map update method of the map update model according to embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0175] like Figure 9 As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 902 or a computer program loaded into random access memory (RAM) 903 from storage unit 908. RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0176] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0177] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as the methods described above. For example, in some embodiments, the methods described above can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, the computing unit 901 can be configured to perform the methods described above by any other suitable means (e.g., by means of firmware).

[0178] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0179] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0180] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0181] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0182] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0183] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0184] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0185] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0186] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0187] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A map updating method, comprising: For the same location information, identify the associated historical map data and M time-series collected image data, where M is an integer greater than or equal to 1; The historical map data and the M acquired image data are encoded respectively to obtain historical map features and M acquired image features; For each acquired image feature, the importance of each sub-feature in the acquired image feature is learned through a confidence graph network, the weight of each sub-feature is determined, and a first confidence graph of the acquired image feature is generated based on the weight of each sub-feature. Based on the M first confidence maps corresponding to the M acquired image features, the M acquired image features are fused to obtain the first intermediate fused features; Based on the historical map features and the first intermediate fusion features, the location information and category information of the target instance are determined, and the target instance represents lane information; as well as The map data is updated based on the location and category information of the target instance.

2. The method of claim 1, wherein, The step of determining the location and category information of the target instance based on the historical map features and the first intermediate fusion features includes: Based on the first intermediate fusion feature and the historical map feature, a first target fusion feature is determined; and The first target fusion feature is decoded to obtain the location information and category information of the target instance.

3. A method for training a map update model, comprising: The training samples are determined, including: a map prior, N time-series sample images, and labels; the map prior and the N sample images are all associated with the same location information, the labels represent whether there are real instances in the N sample images and represent the real location information and real category of the real instances, the real instances represent lane information, and N is an integer greater than or equal to 1; The map prior and the N sample images are encoded respectively to obtain map features and N sample image features; For each sample image feature, the importance of each sub-feature in the sample image feature is learned through a confidence graph network, the weight of the sub-feature is determined, and a second confidence graph of the sample image feature is generated based on the weight of each sub-feature. Based on the N second confidence maps corresponding to the N sample image features respectively, the N sample image features are fused to obtain the second intermediate fused features; Based on the map features and the second intermediate fusion features, determine the output location information and output category of at least one output instance; and The map update model is trained based on the real location information, the output location information, the real category, and the output category.

4. The method of claim 3, wherein, The step of determining the output location information and output category of the at least one output instance based on the map features and the second intermediate fusion features includes: Based on the second intermediate fusion feature and the map feature, determine the second target fusion feature; and The second target fusion feature is decoded to obtain the output location information and output category of the at least one output instance.

5. The method of claim 3, wherein, Training the map update model based on the real location information, the output location information, the real category, and the output category includes: A matching relationship is determined, wherein the matching relationship includes at least one of a first matching relationship and a second matching relationship; the first matching relationship represents the matching relationship between the real instance and the output instance; the real location information of the same real instance includes multiple real sub-location information, the output location information of the same output instance includes multiple output sub-location information, and the second matching relationship represents the matching relationship between the real sub-location information and the output sub-location information; The total loss function value is determined based on the matching relationship, the true location information, the output location information, the true category, and the output category; and The map update model is trained based on the total loss function value.

6. The method of claim 5, wherein, The determination of the matching relationship includes: Arrange the at least one real instance to obtain a sequence of real instances; The at least one output instance is permuted and combined to obtain multiple candidate instance sequences; For each candidate output instance sequence, based on the output position information and output category of each output instance in the candidate output instance sequence, the real position information and real category of each real instance in the real instance sequence, and the order of the real instance sequence and the order of each candidate instance sequence, a first difference evaluation value is determined for the candidate instance sequence; and The first matching relationship is determined based on the smallest first difference evaluation value among multiple first difference evaluation values ​​for the multiple candidate instance sequences.

7. The method of claim 5, wherein, The determination of the matching relationship includes: For the output instance and the real instance that satisfy the first matching relationship, Arrange the multiple real sub-location information of the real instance to obtain a real point sequence; The multiple output sub-position information of the output instance are arranged and combined to obtain multiple candidate point sequences; Based on the multiple real sub-location information, multiple output sub-location information, the order of the real point sequence, and the order of the candidate point sequence, multiple second difference evaluation values ​​are determined for the multiple candidate point sequences; and The second matching relationship is determined based on the smallest second difference evaluation value among the plurality of second difference evaluation values.

8. The method of claim 5, wherein, The step of determining the total loss function value based on the matching relationship, the true location information, the output location information, the true category, and the output category includes: The total loss function value is determined based on a predetermined loss function value; wherein the process of determining the predetermined loss function value includes at least one of the following: The point-level geometric loss function value is determined based on the distance between the output sub-location information and the real sub-location information that satisfy the second matching relationship; The instance-level geometric loss function value is determined based on the difference between the first and second line segments in the line segment pair; wherein the first line segment has two output sub-position information as endpoints, and the second line segment has two real sub-position information as endpoints; in the same line segment pair, the two endpoints of the first line segment and the two endpoints of the second line segment satisfy the second matching relationship; For output instances and real instances that satisfy the first matching relationship, the category loss function value is determined based on the difference between the output category of the output instance and the real category of the real instance; For each output instance, determine the value of the binary classification loss function based on whether there is a real instance that satisfies the first matching relationship with the output instance; Based on the output position information of the at least one output instance and the real position information of the at least one real instance, a first output mask and a first real mask are obtained respectively; the semantic segmentation loss function value is determined according to the difference between the first output mask and the first real mask; and Based on the output position information of the at least one output instance and the real position information of the at least one real instance, a second output mask and a second real mask are obtained respectively; the instance segmentation loss function value is determined according to the difference between the second output mask and the second real mask.

9. A map updating device, comprising: The first data determination module is used to determine the associated historical map data and M time-series collected image data for the same location information, where M is an integer greater than or equal to 1; The first encoding module is used to encode the historical map data and the M acquired image data respectively to obtain historical map features and M acquired image features; The first confidence map generation module is used to learn the importance of each sub-feature in the acquired image feature through a confidence map network for each acquired image feature, determine the weight of the sub-feature, and generate a first confidence map of the acquired image feature based on the weight of each sub-feature. The first fusion module is used to perform fusion processing on the M acquired image features based on the M first confidence maps corresponding to the M acquired image features respectively, to obtain the first intermediate fusion features; The target instance determination module is used to determine the location information and category information of the target instance based on the historical map features and the first intermediate fusion features, wherein the target instance represents lane information; as well as The update module is used to update the map data based on the location and category information of the target instance.

10. The apparatus of claim 9, wherein, The target instance determination module includes: The first determining unit is configured to determine the first target fusion feature based on the first intermediate fusion feature and the historical map feature; and The first decoding unit is used to decode the first target fusion feature to obtain the location information and category information of the target instance.

11. A training device for a map update model, comprising: A sample determination module is used to determine training samples, which include: map prior, N time-series sample images, and labels; the map prior and the N sample images are all associated with the same location information, the labels represent whether there are real instances in the N sample images and represent the real location information and real category of the real instances, the real instances represent lane information, and N is an integer greater than or equal to 1; The second encoding module is used to encode the map prior and the N sample images respectively to obtain map features and N sample image features; The second confidence map generation module is used to learn the importance of each sub-feature in the sample image feature through a confidence map network for each sample image feature, determine the weight of the sub-feature, and generate a second confidence map of the sample image feature based on the weight of each sub-feature. The second fusion module is used to perform fusion processing on the N sample image features based on the N second confidence maps corresponding to the N sample image features respectively, to obtain the second intermediate fusion features; The second data determination module is used to determine the output location information and output category of at least one output instance based on the map features and the second intermediate fusion features; and The training module is used to train the map update model based on the real location information, the output location information, the real category, and the output category.

12. The apparatus of claim 11, wherein, The second data determination module includes: The second determining unit is configured to determine the second target fusion feature based on the second intermediate fusion feature and the map feature; and The second decoding unit is used to decode the second target fusion feature to obtain the output location information and output category of the at least one output instance.

13. The apparatus of claim 11, wherein, The training module includes: A matching submodule is used to determine a matching relationship, which includes at least one of a first matching relationship and a second matching relationship; the first matching relationship represents the matching relationship between the real instance and the output instance; the real location information of the same real instance includes multiple real sub-location information, the output location information of the same output instance includes multiple output sub-location information, and the second matching relationship represents the matching relationship between the real sub-location information and the output sub-location information; The total loss determination submodule is used to determine the total loss function value based on the matching relationship, the true location information, the output location information, the true category, and the output category; and The training submodule is used to train the map update model based on the total loss function value.

14. The apparatus of claim 13, wherein, The matching submodule includes: The first arrangement unit is used to arrange the at least one real instance to obtain a sequence of real instances; The second permutation unit is used to permutate and combine the at least one output instance to obtain a sequence of multiple candidate instances; The first difference determination unit is configured to, for each candidate output instance sequence, determine a first difference evaluation value for the candidate instance sequence based on the output position information and output category of each output instance in the candidate output instance sequence, the real position information and real category of each real instance in the real instance sequence, the order of the real instance sequence, and the order of each candidate instance sequence; and The first relationship determination unit is used to determine the first matching relationship based on the minimum first difference evaluation value among multiple first difference evaluation values ​​for the multiple candidate instance sequences.

15. The apparatus of claim 13, wherein, The matching submodule includes: The second difference determination unit is used to determine the output instance and the real instance that satisfy the first matching relationship. Arrange the multiple real sub-location information of the real instance to obtain a real point sequence; The multiple output sub-position information of the output instance are arranged and combined to obtain multiple candidate point sequences; Based on the multiple real sub-location information, the multiple output sub-location information, the order of the real point sequence, and the order of the candidate point sequence, determine multiple second difference evaluation values ​​for the multiple candidate point sequences; The second relationship determination unit is used to determine the second matching relationship based on the smallest second difference evaluation value among the plurality of second difference evaluation values.

16. The apparatus of claim 13, wherein, The total loss determination submodule includes: The total loss determination unit is used to determine the total loss function value based on a predetermined loss function value; wherein the process of determining the predetermined loss function value includes at least two of the following: The point-level geometric loss function value is determined based on the distance between the output sub-location information and the real sub-location information that satisfy the second matching relationship; The instance-level geometric loss function value is determined based on the difference between the first and second line segments in the line segment pair; wherein the first line segment has two output sub-position information as endpoints, and the second line segment has two real sub-position information as endpoints; in the same line segment pair, the two endpoints of the first line segment and the two endpoints of the second line segment satisfy the second matching relationship; For output instances and real instances that satisfy the first matching relationship, the category loss function value is determined based on the difference between the output category of the output instance and the real category of the real instance; For each output instance, determine the value of the binary classification loss function based on whether there is a real instance that satisfies the first matching relationship with the output instance; Based on the output position information of the at least one output instance and the real position information of the at least one real instance, a first output mask and a first real mask are obtained respectively; the semantic segmentation loss function value is determined according to the difference between the first output mask and the first real mask; and Based on the output position information of the at least one output instance and the real position information of the at least one real instance, a second output mask and a second real mask are obtained respectively; the instance segmentation loss function value is determined according to the difference between the second output mask and the second real mask.

17. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 8.

18. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 8.

19. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images