Method, device, medium and program product for extracting road boundaries

Abstract

The embodiments of the present disclosure disclose a method, device, medium and computer program product for extracting road boundary, wherein the method comprises: inputting an image containing a road to be processed into a pre-trained boundary extraction model; using the boundary extraction model to: extract image features of the image to be processed; based on the image features, predict key points located on the boundary of the road to obtain a key point set; predict a direction angle of the key points, the direction angle representing an included angle between a line connecting two adjacent key points and a preset direction; and based on the direction angle, connect the key points to obtain the boundary of the road. The direction angle represents the topological connection relationship between the key points, and the connection process of the key points is constrained, so that the accuracy of the road boundary extraction can be improved.

Description

Technical Field

[0001] This disclosure relates to a method, electronic device, storage medium, and computer program product for extracting road boundaries. Background Technology

[0002] Currently, in the field of high-precision mapping, there are increasingly higher requirements for the accuracy and speed of map region division. Map region division requires more accurate road segmentation results. Therefore, how to quickly and accurately extract road boundaries is a popular research direction.

[0003] In related technologies, methods for extracting road boundaries include the following: One method is to extract the road network from satellite images, then combine it with prior knowledge of the context to determine the connectivity between roads, and use manual design to optimize complex targets. Another method is to automatically extract road boundaries from images based on pixel-level classification or segmentation algorithms. Summary of the Invention

[0004] This disclosure provides a method, electronic device, storage medium, and computer program product for extracting road boundaries to improve the accuracy of road boundary extraction.

[0005] One aspect of this disclosure provides a method for extracting road boundaries, comprising: inputting a to-be-processed image containing a road into a pre-trained boundary extraction model; using the boundary extraction model to: extract image features of the to-be-processed image; based on the image features, predicting key points located on the road boundary to obtain a set of key points; predicting the orientation angles of the key points, wherein the orientation angles represent the angle between the line connecting two adjacent key points and a preset direction; and based on the orientation angles, connecting the key points to obtain the road boundary.

[0006] In some embodiments, connecting key points based on azimuth angles to obtain the road boundary includes: determining whether a key point has a downstream key point to be connected based on its position; if a key point does not have a downstream key point and its azimuth angle is not empty, then searching for other key points along the direction indicated by the azimuth angle, and determining the key point closest to the key point as its downstream key point; if a key point does not have a downstream key point and its azimuth angle is empty, then determining the key point as the endpoint of the boundary; connecting the key point with its downstream key points to obtain the boundary.

[0007] In some embodiments, the boundary of a road in the image to be processed is obtained by connecting key points based on the direction angle, including: selecting a current key point from the key point set, and performing the following connection steps based on the current key point: if the direction angle of the current key point is empty, the current key point is determined as the endpoint of the boundary; the current key point is deleted from the key point set to obtain an updated key point set; a new current key point is selected from the updated key point set, and the connection steps are performed again; if the direction angle of the current key point is not empty, the direction indicated by the direction angle is determined as the search direction; other key points are searched along the search direction, and the key point closest to the current key point is determined as the target key point; the current key point and the target key point are connected, and the current key point is deleted from the key point set; if the key point set is not empty, the target key point is used as the current key point, and the connection steps are performed again; if the key point set is empty, the connection is terminated, and the graphic formed by connecting the key points is determined as the boundary.

[0008] In some embodiments, the boundary extraction model includes an encoder, a convolutional layer, a boundary segmentation branch, and a direction prediction branch; extracting image features of the image to be processed includes: using the encoder to upsample the image to be processed, extracting similar features of pixels in the image to be processed, and obtaining coarse-grained features of the image to be processed; using the convolutional layer to extract image features from the coarse-grained features; predicting key points located on the boundary of the road includes: using the boundary segmentation branch, determining the semantic labels of pixels in the image to be identified based on the image features; predicting key points based on the semantic labels; the boundary extraction model includes a direction prediction branch; predicting the orientation angle of the key points includes: using the orientation prediction branch, predicting the orientation angle of the key points based on the image features.

[0009] In some embodiments, the boundary extraction model is trained through the following steps: acquiring a sample set, which includes at least one sample image with a labeled sample tag, the sample tag including sample keypoints and sample orientation angles of the sample keypoints; inputting the sample images into a pre-constructed initial boundary extraction model to predict the keypoints in the sample images and the azimuth angles of the keypoints in the sample images; determining the boundary segmentation loss based on the sample keypoints and the keypoints in the sample images; determining the orientation prediction loss based on the sample azimuth angles and the azimuth angles of the keypoints in the sample images; determining the loss function value based on the boundary segmentation loss and the orientation prediction loss; and adjusting the parameters of the boundary extraction model based on the loss function value to obtain the boundary extraction model.

[0010] In some embodiments, the sample set is obtained through the following steps: acquiring base map data containing sample roads and corresponding layer data; constructing a road network map based on the base map data; determining the minimum contour of the sample road segmentation region in the road network map; taking the points located on the boundary of the minimum contour as contour points and determining the orientation angle of the contour points; mapping the contour points to the layer data to obtain the mapping points of the contour points in the layer data; marking the mapping points as sample key points and marking the orientation angle of the contour points as the sample orientation angle of the sample key points to obtain the marked layer data; and extracting an image of a preset size from the marked layer data as a sample image to obtain the sample set.

[0011] In some embodiments, inputting sample images from a sample set into a pre-constructed initial boundary extraction model to predict key points and azimuth angles of key points in the sample images includes: adjusting the resolution of the sample images to obtain multiple sample images at preset resolutions; inputting the multiple sample images at preset resolutions into the boundary extraction model respectively to predict key points and azimuth angles in each sample image at a preset resolution; determining the boundary segmentation loss based on the key points in the sample images and the sample key points, including: determining the boundary segmentation loss value corresponding to the preset resolution based on the sample key points and the key points in each sample image at the preset resolution; determining the boundary segmentation loss by summing the boundary segmentation loss values ​​corresponding to the multiple preset resolutions; and determining the direction prediction loss based on the azimuth angles of key points in the sample images and the sample azimuth angles, including: determining the direction prediction loss value corresponding to the preset resolution based on the sample direction angles and the direction angles in each sample image at the preset resolution; and determining the direction prediction loss by summing the direction prediction loss values ​​corresponding to the multiple preset resolutions.

[0012] In another aspect of this disclosure, an electronic device is provided, comprising: a memory for storing a computer program; and a processor for executing the computer program stored in the memory, wherein when the computer program is executed, it implements the method for extracting road boundaries in any of the above embodiments.

[0013] In another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method for extracting road boundaries in any of the above embodiments.

[0014] In another aspect of this disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the method for extracting road boundaries in any of the above embodiments.

[0015] The method for extracting road boundaries provided in this disclosure involves inputting a road-containing image to be processed into a pre-trained boundary extraction model. The model then extracts image features from the image and determines key points located on the road boundary based on these features, resulting in a set of key points. Next, the orientation angles of the key points are predicted, and based on these orientation angles, the key points are connected to obtain the road boundary. The orientation angle represents the angle between the line connecting two adjacent key points and a preset direction, thus representing the topological connection relationship between key points. Constraining the connection process of key points by using the orientation angle can improve the accuracy of road boundary extraction.

[0016] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0017] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0018] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0019] Figure 1 This is a schematic diagram illustrating an application scenario of the method disclosed in this paper for extracting road boundaries;

[0020] Figure 2 This is a flowchart of one embodiment of the present disclosure for extracting road boundaries;

[0021] Figure 3 This is a flowchart illustrating the generation of road boundaries in one embodiment of the present disclosure for extracting road boundaries;

[0022] Figure 4 This is a flowchart illustrating the generation of road boundaries in yet another embodiment of the present disclosure for extracting road boundaries;

[0023] Figure 5 This is a schematic diagram of the architecture of a boundary extraction model in one embodiment of the method for extracting road boundaries disclosed herein;

[0024] Figure 6 for Figure 5 The flowchart shown in the embodiment illustrates the training process of the boundary extraction model.

[0025] Figure 7 This is a flowchart illustrating the acquisition of a sample set in one embodiment of the present disclosure for extracting road boundaries;

[0026] Figure 8 This is a flowchart illustrating the determination of a loss function value in one embodiment of the present disclosure for extracting road boundaries;

[0027] Figure 9This is a schematic diagram of one embodiment of the apparatus for extracting road boundaries disclosed herein;

[0028] Figure 10 This is a schematic diagram of the structure of an application embodiment of the electronic device disclosed herein. Detailed Implementation

[0029] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0030] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0031] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0032] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0033] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0034] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0035] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0036] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0037] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0038] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0039] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0040] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0041] The following is combined Figure 1 For example, in Figure 1 In the scenario shown, terminal device 100 can be the execution entity of the method for extracting road boundaries disclosed herein. The image to be processed 110 can be a city satellite image or a real-world image. The execution entity can acquire the image to be processed 110 via a network, and then input the image to be processed 110 into a pre-trained boundary extraction model 120. The boundary extraction model can be a deep neural network for image recognition or instance segmentation, such as a convolutional neural network or a generative adversarial network. The boundary extraction model 120 can extract image features 130 from the image to be processed 110. Image features 130 can include, for example, feature information of corner points in the image, road edge information, and road flow direction information. Then, based on the image features 130, key points located on the road boundary in the image to be processed 110 are identified, resulting in a key point set 140, and the azimuth angles of the key points are predicted, resulting in a feature image 150. Finally, the connection process of the key points is constrained by the azimuth angles, connecting the key points to form the contour of the road, resulting in an image 160 representing the road boundary.

[0042] By characterizing the topological connections between key points using azimuth angles and constraining the connection process of key points, the accuracy of road boundary extraction can be improved.

[0043] Exemplary methods

[0044] The following is combined Figure 2 The method for extracting road boundaries disclosed herein is illustrated by way of example. Figure 2 A flowchart of one embodiment of the method for extracting road boundaries disclosed herein is shown below. Figure 2 As shown, the process includes the following steps.

[0045] Step 210: Input the image to be processed, which contains roads, into the pre-trained boundary extraction model.

[0046] In this embodiment, the image to be processed can be an aerial photograph or remote sensing image containing roads, and the image format can include types such as JPG and PNP. The boundary extraction model can be a deep learning network used for image recognition or instance segmentation, such as a convolutional neural network or a generative adversarial network.

[0047] Then, steps 220 to 250 are performed using the boundary extraction model.

[0048] Step 220: Extract image features from the image to be processed.

[0049] In this embodiment, the executing entity can use the boundary extraction model to perform operations such as sampling, convolution, and pooling on the image to be processed, thereby extracting feature information and obtaining image features of the image to be processed. The feature information may include corner information, road edge information, and road flow direction information.

[0050] Step 230: Based on image features, predict the key points located on the road boundary to obtain a set of key points.

[0051] As an example, the executing entity can use a boundary extraction model to identify pixels in the image to be processed based on the image features extracted in step 220, and determine the semantic labels of the pixels. The semantic labels can represent the instance of the pixel, such as a road boundary, a road area, or a non-road area. Then, based on the semantic labels, key points located on the road boundary are extracted from the image to be processed to obtain a set of key points.

[0052] Step 240: Predict the orientation angle of the key points.

[0053] The direction angle represents the angle between the line connecting two adjacent key points and the preset direction.

[0054] In this embodiment, the orientation angle can represent the topological connection relationship between adjacent key points.

[0055] As an example, the preset direction can be the horizontal coordinate axis direction of the image, and the direction angle can be the angle between the line connecting two adjacent key points and the horizontal coordinate axis direction.

[0056] Step 250: Based on the direction angle, connect the key points to obtain the road boundary.

[0057] In practice, optical factors such as occlusion and shadows can cause the image to be processed to not fully represent the entire road, and only a part of the road can be represented in some areas. This results in breakpoints in the key point set obtained in step 220, that is, there are no downstream key points to be connected in the neighborhood of the key point.

[0058] In this embodiment, for breakpoints, the road connectivity direction can be estimated by the direction angle, thereby determining the adjacent keypoints connected to this type of keypoint.

[0059] The method for extracting road boundaries provided in this embodiment inputs an image containing roads into a pre-trained boundary extraction model. The model then extracts image features from the image and determines key points located on the road boundaries based on these features, resulting in a set of key points. Next, the orientation angles of the key points are predicted, and based on these orientation angles, the key points are connected to obtain the road boundaries. The orientation angle represents the angle between the line connecting two adjacent key points and a preset direction, thus representing the topological connection relationship between key points. Constraining the connection process of key points by using orientation angles can improve the accuracy of road boundary extraction.

[0060] Next, refer to Figure 3 ,exist Figure 3 In the illustrated embodiment, step 250 may further include the following steps.

[0061] Step 310: Based on the location of the key point, determine whether there are downstream key points to be connected.

[0062] In this embodiment, the executing entity can retrieve the neighborhood of the location of the key point and determine the upstream and downstream key points of the key point based on the continuity of the coordinates.

[0063] As an example, the coordinates of key point a are (0, 1). There are key points b (0, 2) and c (0, 0) in its neighborhood. The other side of c is already connected to other key points. Therefore, c is the upstream key point of a, and b is the downstream key point of a. The topological connection relationship of these three key points is that c is connected to a, and a is connected to b.

[0064] If there are no downstream key points for a key point, continue to determine whether the azimuth angle of the key point is empty. If the azimuth angle is not empty, proceed to steps 320 and 340; if the azimuth angle is empty, proceed to step 330.

[0065] If a critical point has a downstream critical point, proceed directly to step 340.

[0066] Step 320: Search for other key points along the direction indicated by the azimuth angle, and determine the key point closest to the key point as the downstream key point of the key point.

[0067] In this embodiment, when a key point has no downstream key points, the azimuth angle can be used to determine whether the key point is the end point of the road boundary. If the azimuth angle of the key point is not empty, it means that the boundary extraction model predicts that the key point is not the end point of the road boundary, and the real-world location corresponding to its downstream key point may be occluded, making it impossible to directly identify from the image to be processed. In this case, the road connectivity direction can be predicted by the direction angle, and the executing entity can perform a search according to the direction indicated by the direction angle. The first key point retrieved is the downstream key point.

[0068] Step 330: Determine this key point as the endpoint of the boundary.

[0069] In this embodiment, if there are no downstream key points and the direction angle is empty, it means that the boundary extraction model predicts that the real area where the key point is located is the end of the road. In this case, the key point can be identified as the end of the road boundary.

[0070] Step 340: Connect the key point to the downstream key point to obtain the boundary.

[0071] In this embodiment, for key points in a continuous region, their downstream key points can be directly determined by their relative positions. For key points in an occluded region, their downstream key points can be estimated using their orientation angles. Then, by connecting the key points and their downstream key points, the complete road boundary can be obtained. This avoids the adverse effects of occlusion on the accuracy of road connectivity and helps improve the accuracy of road boundaries.

[0072] Next, refer to Figure 4 , Figure 4 A flowchart illustrating the generation of road boundaries in yet another embodiment of this disclosure for extracting road boundaries is shown, as follows: Figure 4 As shown, step 250 above may also include the following steps.

[0073] Step 410: Select the current key point from the set of key points.

[0074] For example, the executing entity can randomly select a key point from the set of key points as the current key point.

[0075] In some optional implementations of this embodiment, the executing entity can select the current key point based on coordinates, for example, selecting a key point with extreme coordinate values ​​as the current key point. This reduces the probability of using intermediate key points as the current key point and improves boundary extraction efficiency.

[0076] Then, a connection step is performed based on the current key point. The connection step includes the following steps 420 to 490.

[0077] If the direction angle of the current key point is empty, proceed to steps 420 to 440; if the direction angle of the current key point is not empty, proceed to steps 450 to 490.

[0078] Step 420: Determine the current key point as the endpoint of the boundary.

[0079] In this embodiment, if the direction angle is empty, there is no downstream key point for the current key point, indicating that the current key point is located at the end of the road in the real scene. At this time, executing the main graph can determine the current key point as the end of the boundary.

[0080] Step 430: Delete the current key point from the key point set to obtain the updated key point set.

[0081] Step 440: Select a new current key point from the updated set of key points.

[0082] In this embodiment, the executing entity can select a new current key point from the updated set of key points, determine again whether the azimuth angle of the new current key point is empty, and execute subsequent steps based on the determination result.

[0083] Step 450: If the direction angle of the current key point is not empty, determine the direction indicated by the direction angle as the search direction.

[0084] Step 460: Search for other key points along the search direction, and determine the key point that is closest to the current key point as the target key point.

[0085] Step 470: Connect the current key point with the target key point, and delete the current key point from the key point set.

[0086] Then, the executing entity can determine whether the key point set is empty. If the key point set is not empty, then execute step 480; if the key point set is empty, then execute step 490.

[0087] Step 480: If the set of key points is not empty, take the target key point as the current key point and execute the connection step again.

[0088] In this embodiment, a non-empty key point set indicates that there are unconnected key points, meaning that the connection step has not yet been completed. In this case, the target key point can be used as the current key point, and the connection step described above can be executed again.

[0089] Step 490: If the set of key points is empty, terminate the connection and define the shape formed by connecting the key points as the boundary.

[0090] In this embodiment, an empty keypoint set indicates that all keypoints have been connected, and the shape formed by connecting the keypoints is the boundary of the road.

[0091] from Figure 4 It can be seen that, Figure 4 The flowchart shown illustrates the steps of determining the topological connectivity of all key points based on the direction angle, which can further enhance the constraint of the direction angle on the road boundary extraction process and help to further improve the accuracy of the road boundary.

[0092] Next, refer to Figure 5 , Figure 5 This is a schematic diagram of the boundary extraction model architecture in one embodiment of the method for extracting road boundaries disclosed herein, as shown below. Figure 5 As shown, the boundary extraction model may include an encoder 510, a convolutional layer 520, a boundary segmentation branch 530, and a direction prediction branch 540.

[0093] exist Figure 5 Based on the embodiment shown, the aforementioned step 220 can be implemented as follows: using encoder 520, upsampling the image to be processed 550, extracting similar features of pixels in the image to be processed, and obtaining coarse-grained features of the image to be processed; using convolutional layer 530, extracting image features from the coarse-grained features.

[0094] exist Figure 5 Based on the embodiment shown, the aforementioned step 230 can be implemented as follows: using the boundary segmentation branch 530, based on image features, determine the semantic labels of pixels in the image to be identified 550; based on the semantic labels, predict key points.

[0095] exist Figure 5 Based on the embodiment shown, the aforementioned step 240 can be implemented as follows: using the direction prediction branch 540, the direction angle of the key point is predicted based on image features.

[0096] from Figure 5As can be seen, the boundary extraction model in this embodiment can extract coarse-grained features from the image to be processed through the encoder, and then extract image features from the coarse-grained features, which can improve the accuracy of the image features in characterizing the feature information in the image to be processed. Furthermore, by determining the key points and their orientation angles based on the image features through the boundary segmentation branch and the orientation prediction branch, the accuracy of the boundary extraction model can be further improved.

[0097] Next reference Figure 6 , Figure 6 The training process of a boundary extraction model in one embodiment of the present disclosure for extracting road boundaries is shown, the process including the following steps.

[0098] Step 610: Obtain the sample set.

[0099] The sample set includes at least one sample image with a labeled sample tag. The sample tag includes a sample key point and the sample orientation angle of the sample key point.

[0100] As an example, the sample image could be a labeled aerial photograph or remote sensing image.

[0101] Step 620: Input the sample image into the pre-built initial boundary extraction model to obtain the key points in the sample image and the azimuth angles of the key points in the sample image.

[0102] Step 630: Determine the boundary segmentation loss based on the key points of the sample and the key points in the sample image.

[0103] In a specific example, as shown in formula (1), the execution entity can first determine the intersection-union ratio (IU) between the output of the initial boundary extraction model and the sample labels, and then substitute the IU into the SoftIoU function to obtain the boundary segmentation loss.

[0104]

[0105] In the formula, Let f represent the boundary loss, X represent the sample image, y represent the sample keypoints, and f' represent the boundary loss. seg (*) indicates the prediction function for boundary segmentation.

[0106] Step 640: Determine the direction prediction loss based on the sample azimuth angle and the azimuth angle of key points in the sample image.

[0107] In a specific example, the executing entity can use formula (2) to determine the direction prediction loss.

[0108]

[0109] In the formula, f represents the direction prediction loss.seg (*) represents the prediction function for direction prediction.

[0110] Step 650: Determine the loss function value based on boundary segmentation loss and direction prediction loss.

[0111] As an example, the executing entity can directly determine the loss function value as the sum of the boundary segmentation loss and the direction prediction loss.

[0112] Step 660: Adjust the parameters of the initial boundary extraction model based on the loss function value to obtain the boundary extraction model.

[0113] In this embodiment, the parameters of the initial boundary extraction model can be adjusted based on the direction propagation characteristics of the deep neural network and the loss function value until the loss function converges, thus completing the training process and obtaining the trained boundary extraction model.

[0114] from Figure 6 It can be seen that, Figure 6 The process shown illustrates how the loss function value is determined based on boundary segmentation loss and orientation prediction loss, thereby constraining the training process of the boundary extraction model. Through co-training, the boundary extraction model can simultaneously learn boundary segmentation strategies and orientation prediction strategies, which helps improve the performance of the boundary extraction model.

[0115] Next, refer to Figure 7 , Figure 7 The flowchart illustrating the process of obtaining a sample set in one embodiment of this disclosure for extracting road boundaries is shown, such as... Figure 7 As shown, the process includes the following steps.

[0116] Step 710: Obtain the base map data containing the sample roads and the corresponding layer data.

[0117] Typically, base map data refers to the fundamental data that makes up a map, such as basic roads and waterways. Layers are used to store the information needed for a map. A map can consist of one or more layers, which may include the geographic coordinate system used by the map, the initial extent of the map, and references to layer definitions. As an example, layer data can be satellite layers or two-dimensional layers.

[0118] In a specific example, the executing entity can obtain base map data and corresponding layer data from OSM (OpenStreetMap) via the network. The layer data can be, for example, a satellite layer or a two-dimensional layer.

[0119] Step 720: Construct a road network map based on the base map data.

[0120] As an example, the implementing entity can first extract road intersections from map data as nodes in the road network diagram, then use the set of points corresponding to the roads adjacent to the nodes as edges of the road network, and connect each node with its corresponding edge to form the road network diagram.

[0121] In some optional implementations of this embodiment, the road network map can also be preprocessed, for example, by deleting nodes with a degree of no more than 2 to remove redundant data.

[0122] Step 730: Determine the minimum contour of the sample road segmentation region in the road network map.

[0123] In this embodiment, the minimum contour means that it does not contain any other contours.

[0124] In a specific example, the implementing entity can first consider the road network. Figure 2 The data is valued, and then the contours of multiple regions divided by the sample roads in the road network map are determined by extracting the connected components. Then, the minimum contour is extracted from the minimum contour of the sample road segmentation region.

[0125] Step 740: Take the points located on the boundary of the minimum contour as contour points and determine the orientation angle of the contour points.

[0126] As an example, the executing entity can connect two adjacent contour points according to the flow direction of the preset minimum contour, and then calculate the angle between the connecting line and the horizontal coordinate axis of the road network map to obtain the direction angle of the contour point.

[0127] Step 750: Map the contour points to the layer data to obtain the mapping points of the contour points in the layer data.

[0128] In this embodiment, the executing entity can first adjust the latitude, longitude and scale of the road network map and the layer data to be consistent, and then stack them to determine the one-to-one mapping relationship between pixels and contour points in the layer data, thereby obtaining multiple mapping points.

[0129] Step 760: Mark the mapping points as sample key points, and mark the direction angles of the contour points as sample direction angles of the sample key points to obtain the marked layer data.

[0130] Step 770: Extract an image of a preset size from the marked layer data as a sample image to obtain a sample set.

[0131] In this embodiment, the executing entity can extract multiple sample images from the marked layer data based on the image input size preset by the boundary extraction model to obtain a sample set.

[0132] from Figure 7 It can be seen that, Figure 7The process shown illustrates the steps for automatically labeling layer data to construct sample images, which can improve the efficiency of constructing sample images.

[0133] Next, refer to Figure 8 , Figure 8 A process for determining a loss function value in one embodiment of the present disclosure for extracting road boundaries is shown, the process including the following steps.

[0134] Step 810: Adjust the resolution of the sample images to obtain multiple sample images with preset resolutions.

[0135] Step 820: Input multiple sample images of preset resolution into the boundary extraction model to predict the key points and azimuth angles in each sample image of preset resolution.

[0136] Step 830: Based on the key points of the sample and the key points in the sample image at each preset resolution, determine the boundary segmentation loss value corresponding to the preset resolution.

[0137] Step 840: The sum of the boundary segmentation loss values ​​corresponding to each of the multiple preset resolutions is determined as the boundary segmentation loss.

[0138] Step 850: Based on the sample orientation angle and the orientation angle in the sample image at each preset resolution, determine the orientation prediction loss value corresponding to the preset resolution.

[0139] Step 860: The sum of the direction prediction loss values ​​corresponding to each of the multiple preset resolutions is determined as the direction prediction loss.

[0140] In a specific example, the preset resolutions may include the following three: 200*300, 300*400, and 400*500. The execution entity can adjust the resolution of the sample image to the above three preset resolutions in sequence, and then input it into the boundary extraction model again to obtain three output results corresponding to the above three preset resolutions. Each output result includes the predicted key points and orientation angles. Then, based on the sample key points and the predicted key points, the execution entity can determine the segmentation loss value corresponding to each output result according to formula (1), and determine the sum of the three segmentation loss values ​​as the segmentation loss. At the same time, based on the sample orientation angle and the predicted orientation angle, the execution entity can determine the orientation prediction loss value corresponding to each output result according to formula (2), and determine the three orientation prediction loss values ​​as the orientation prediction loss. Finally, the execution entity can combine the segmentation loss and the orientation prediction loss to determine the loss function value.

[0141] The model can minimize the loss function at various resolutions, thereby improving the accuracy of the model's output road network at all resolutions.

[0142] from Figure 8 It can be seen that, Figure 8 The process shown illustrates the steps of determining the loss function value based on the segmentation loss value and orientation prediction loss value of the boundary extraction model at multiple preset resolutions. The obtained loss function value can characterize the comprehensive performance of the boundary extraction model at different resolutions, thereby constraining the training process of the boundary extraction model and enabling the boundary extraction model to have high accuracy when facing images of different resolutions.

[0143] Exemplary device

[0144] Next, refer to Figure 9 , Figure 9 A schematic diagram of one embodiment of the apparatus for extracting road boundaries disclosed herein is shown, as follows: Figure 9 As shown, the device includes: an image input unit 910 configured to input a pre-trained boundary extraction model with an image to be processed containing a road; and a boundary extraction unit 920 configured to use the boundary extraction model to: extract image features of the image to be processed; predict key points located on the boundary of the road based on the image features, thereby obtaining a set of key points; predict the orientation angle of the key points, wherein the orientation angle represents the angle between the line connecting two adjacent key points and a preset direction; and connect the key points based on the orientation angle to obtain the boundary of the road.

[0145] In this embodiment, the boundary extraction unit 920 may further include: a judgment module configured to determine whether a key point has a downstream key point to be connected based on the position of the key point; a first retrieval module configured to retrieve other key points along the direction indicated by the azimuth angle if the key point does not have a downstream key point and the azimuth angle of the key point is not empty, and determine the key point closest to the key point as the downstream key point of the key point; a first endpoint module configured to determine the key point as the endpoint of the boundary if the key point does not have a downstream key point and the azimuth angle of the key point is empty; and a first connection module configured to connect the key point with the downstream key point of the key point to obtain the boundary.

[0146] In this embodiment, the boundary extraction module 920 may further include: a selection module configured to select a current keypoint from the keypoint set and perform a connection step based on the current keypoint; a second endpoint module configured to determine the current keypoint as the endpoint of the boundary if the direction angle of the current keypoint is empty; an update module configured to delete the current keypoint from the keypoint set to obtain an updated keypoint set; the selection module is further configured to select a new current keypoint from the updated keypoint set and perform the connection step again; a direction module configured to determine the direction indicated by the direction angle as the search direction if the direction angle of the current keypoint is not empty; a second search module configured to search for other keypoints along the search direction and determine the keypoint closest to the current keypoint as the target keypoint; and a second connection module configured to connect the current keypoint and the target keypoint and delete the current keypoint from the keypoint set. The traversal module is configured to use the target keypoint as the current keypoint and perform the connection step again; the termination module is configured to terminate the connection if the set of keypoints is empty and define the shape formed by connecting the keypoints as the boundary.

[0147] In this embodiment, the boundary extraction model includes an encoder, a convolutional layer, a boundary segmentation branch, and a direction prediction branch; the boundary extraction unit 920 may further include an encoder module configured to upsample the image to be processed using the encoder, extract similar features of pixels in the image to be processed, and obtain coarse-grained features of the image to be processed; a convolutional module configured to extract image features from the coarse-grained features using the convolutional layer; a segmentation module configured to determine the semantic labels of pixels in the image to be identified based on the image features using the boundary segmentation branch; a determination module configured to predict key points based on the semantic labels; and a prediction module configured to predict the orientation angle of the key points, including: predicting the orientation angle of the key points based on the image features using the direction prediction branch.

[0148] In this embodiment, the device may further include a model training unit, which includes: a sample acquisition module configured to acquire a sample set, the sample set including at least one sample image with labeled sample tags, the sample tags including sample keypoints and sample orientation angles of the sample keypoints; a sample input module configured to input the sample images into a pre-constructed initial boundary extraction model to predict the keypoints in the sample images and the azimuth angles of the keypoints in the sample images; a boundary loss module configured to determine the boundary segmentation loss based on the sample keypoints and the keypoints in the sample images; a direction loss module configured to determine the direction prediction loss based on the sample azimuth angles and the azimuth angles of the keypoints in the sample images; a loss function module configured to determine the loss function value based on the boundary segmentation loss and the direction prediction loss; and a parameter optimization module configured to adjust the parameters of the boundary extraction model based on the loss function value to obtain the boundary extraction model.

[0149] In this embodiment, the device further includes a sample construction unit, which comprises: a data acquisition module configured to acquire base map data containing sample roads and corresponding layer data; a road network construction module configured to construct a road network map based on the base map data; a contour tracking module configured to determine the minimum contour of the sample road segmentation region in the road network map; a direction angle module configured to take points located on the boundary of the minimum contour as contour points and determine the direction angle of the contour points; a mapping module configured to map the contour points to the layer data to obtain the mapping points of the contour points in the layer data; a marking module configured to mark the mapping points as sample key points and mark the direction angle of the contour points as the sample direction angle of the sample key points to obtain the marked layer data; and an extraction module configured to extract an image of a preset size from the marked layer data as a sample image to obtain a sample set.

[0150] In this embodiment, the sample input module further includes: an adjustment submodule configured to adjust the resolution of the sample image to obtain multiple sample images with preset resolutions; and an input submodule configured to input the multiple sample images with preset resolutions into the boundary extraction model respectively to predict the key points and azimuth angles in each sample image with preset resolutions.

[0151] The boundary loss module further includes: a boundary component module, configured to determine the boundary segmentation loss value corresponding to the preset resolution based on the sample key points and the key points in the sample image at each preset resolution; and a boundary summation module, configured to determine the boundary segmentation loss by summing the boundary segmentation loss values ​​corresponding to multiple preset resolutions.

[0152] The orientation loss module further includes: an orientation component module configured to determine the orientation prediction loss value corresponding to the preset resolution based on the sample orientation angle and the orientation angle in the sample image at each preset resolution; and an orientation summation module configured to determine the orientation prediction loss by summing the orientation prediction loss values ​​corresponding to multiple preset resolutions.

[0153] In addition, this disclosure also provides an electronic device, including:

[0154] Memory, used to store computer programs;

[0155] A processor is configured to execute a computer program stored in the memory, wherein, when the computer program is executed, it implements the method for extracting road boundaries as described in any of the above embodiments of the present disclosure.

[0156] Figure 10 This is a schematic diagram illustrating the structure of an application embodiment of the electronic device disclosed herein. Below, reference is made to… Figure 10This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0157] like Figure 10 As shown, the electronic device includes one or more processors and memory.

[0158] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0159] The memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and a processor may execute the program instructions to implement the methods for extracting road boundaries and / or other desired functions described in the various embodiments of this disclosure above.

[0160] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0161] In addition, the input device may include, for example, a keyboard, a mouse, etc.

[0162] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0163] Of course, for the sake of simplicity, Figure 10 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0164] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the methods for extracting road boundaries according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0165] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0166] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the methods for extracting road boundaries according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0167] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0168] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0169] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0170] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0171] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0172] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0173] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0174] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0175] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for extracting a road boundary, characterized by, The method comprises the following steps: inputting an image to be processed containing a road into a pre-trained boundary extraction model; using the boundary extraction model to extract image features of the image to be processed; based on the image features, predicting key points located on the boundary of the road to obtain a key point set; predicting a direction angle of the key points, the direction angle representing an included angle between a line connecting two adjacent key points and a preset direction; and connecting the key points based on the direction angle to obtain the boundary of the road; wherein, based on the direction angle, connecting the key points to obtain the boundary of the road comprises: determining whether the key point has a downstream key point to be connected based on the position of the key point; if the key point has no downstream key point and the orientation angle of the key point is not empty, searching for other key points in the direction indicated by the orientation angle and determining the key point closest to the key point as the downstream key point of the key point; if the key point has no downstream key point and the orientation angle of the key point is empty, determining the key point as the end point of the boundary; connecting the key point and the downstream key point of the key point to obtain the boundary.

2. The method of claim 1, wherein, Based on the direction angle, connecting the key points to obtain the boundary of the road in the image to be processed comprises: selecting a current key point from the key point set and performing the following connection steps based on the current key point: if the direction angle of the current key point is empty, determining the current key point as the end point of the boundary; deleting the current key point from the key point set to obtain an updated key point set; selecting a new current key point from the updated key point set and performing the connection step again; if the direction angle of the current key point is not empty, determining the direction indicated by the direction angle as a search direction; searching for other key points in the search direction and determining the key point closest to the current key point as a target key point; connecting the current key point and the target key point and deleting the current key point from the key point set; if the key point set is not empty, taking the target key point as the current key point and performing the connection step again; if the key point set is empty, terminating the connection and determining the graph formed by the key points as the boundary.

3. The method according to any of claims 1 or 2, characterized in that, The boundary extraction model comprises an encoder, a convolutional layer, a boundary segmentation branch and a direction prediction branch; the extraction of the image features of the image to be processed comprises: using the encoder to upsample the image to be processed and extract similar features of the pixel points in the image to be processed to obtain coarse-grained features of the image to be processed; and using the convolutional layer to extract the image features from the coarse-grained features; the prediction of the key points located on the boundary of the road comprises: using the boundary segmentation branch to determine semantic labels of the pixel points in the image to be recognized based on the image features; and predicting the key points based on the semantic labels. The boundary extraction model comprises a direction prediction branch; and the direction angle of the key point is predicted based on the image feature by using the direction prediction branch.

4. The method according to any of claims 1 or 2, characterized in that, The boundary extraction model is trained through the following steps: Obtain a sample set comprising at least one sample image with a labeled sample label, wherein the sample label comprises a sample key point and a sample direction angle of the sample key point; input the sample image into a pre-constructed initial boundary extraction model to predict the key point in the sample image and the direction angle of the key point in the sample image; determine a boundary segmentation loss based on the sample key point and the key point in the sample image; determine a direction prediction loss based on the sample direction angle and the direction angle of the key point in the sample image; determine a loss function value based on the boundary segmentation loss and the direction prediction loss; adjust the parameters of the initial boundary extraction model based on the loss function value to obtain the boundary extraction model.

5. The method of claim 4, wherein, The sample set is obtained through the following steps: obtain base map data comprising a sample road and layer data corresponding to the base map data; construct a road network graph based on the base map data; determine the minimum contour of the sample road segmentation area in the road network graph; determine the direction angle of the contour point; map the contour point to the layer data to obtain a mapping point of the contour point in the layer data; label the mapping point as a sample key point and label the direction angle of the contour point as a sample direction angle of the sample key point to obtain labeled layer data; cut an image of a preset size from the labeled layer data as a sample image to obtain the sample set.

6. The method of claim 4, wherein, input the sample image in the sample set into a pre-constructed initial boundary extraction model to predict the key point in the sample image and the direction angle of the key point in the sample image, comprising: adjusting the resolution of the sample image to obtain a plurality of sample images of a plurality of preset resolutions; inputting the plurality of sample images of the plurality of preset resolutions into the boundary extraction model to predict the key point and the direction angle in each sample image of a preset resolution; determine a boundary segmentation loss based on the key point in the sample image and the sample key point, comprising: determining a boundary segmentation loss value corresponding to each preset resolution based on the sample key point and the key point in the sample image of each preset resolution; and determining the sum of the boundary segmentation loss values corresponding to the plurality of preset resolutions as the boundary segmentation loss; determine a direction prediction loss based on the direction angle of the key point in the sample image and the sample direction angle, comprising: determining a direction prediction loss value corresponding to each preset resolution based on the sample direction angle and the direction angle in the sample image of each preset resolution; and determining the sum of the direction prediction loss values corresponding to the plurality of preset resolutions as the direction prediction loss.

7. An electronic device, comprising: comprise: a memory for storing a computer program; A processor for executing a computer program stored in the memory, and the computer program, when executed, implements the method of any one of claims 1-6.

8. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the method of any one of claims 1-6.

9. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions, when executed by the processor, implement the method of any one of claims 1-6.

Images