Road image generation method and apparatus, and computer device, storage medium and computer program product

Abstract

A road image generation method, which is executed by a computer device. The method comprises: performing instance segmentation on road imagery to obtain an instance segmentation result, and on the basis of the instance segmentation result, generating a road surface image (S202); performing hole filling on a hole area in the road surface image, and instantiating the road surface image which has undergone the hole filling, so as to obtain an instantiated image (S204); on the basis of the edges of road instances in the instantiated image, determining a common element shared by each road instance (S206); on the basis of each road instance, performing semantic segmentation on the common element to obtain semantic information of the common element (S208); and on the basis of the semantic information of the common element, performing image editing on each road instance in the instantiated image to generate a road image (S210).

Description

Road image generation method, device, computer equipment, storage medium and computer program product

[0001] Related applications

[0002] This application claims priority to Chinese patent application number 2023112658548, filed on September 28, 2023, entitled “Road Image Reconstruction Method, Device, Computer Equipment and Storage Medium,” the entire text of which is hereby incorporated by reference. Technical Field

[0003] The present application relates to the field of computer technology, and in particular to a road image generation method, apparatus, computer equipment, storage medium, and computer program product. Background Art

[0004] Road images contain visual and semantic information of the road area and can be widely used in navigation, autonomous driving, traffic management and other fields to support accurate route planning and path navigation.

[0005] Currently, road images are usually generated based on road images obtained from satellites, cameras or other sensors. The quality of the sensors, weather conditions, occlusions or other factors may result in poor quality of the road images, which in turn affects the accuracy of the road images.

[0006] Summary of the Invention

[0007] The present application provides a road image generation method, apparatus, computer equipment, storage medium and computer program product.

[0008] In a first aspect, the present application provides a method for generating a road image, which is executed by a computer device, and the method comprises:

[0009] Performing instance segmentation on the road image to obtain instance segmentation results, and generating a road surface image based on the instance segmentation results;

[0010] Filling holes in the road surface image, and instantiating the hole-filled road surface image to obtain an instantiated image;

[0011] determining common elements shared by the road instances based on edges of the road instances in the instantiated image;

[0012] Performing semantic division on the common elements according to each of the road instances to obtain semantic information of the common elements; and

[0013] Image editing is performed on each road instance in the instantiated image based on the semantic information of the common element to generate a road image.

[0014] In a second aspect, the present application further provides a road image generation device. The device comprises:

[0015] A road surface image generation module is used to perform instance segmentation on the road image, obtain instance segmentation results, and generate a road surface image based on the instance segmentation results;

[0016] An instantiated image generation module is used to fill the hole areas in the road surface image and instantiate the hole-filled road surface image to obtain an instantiated image;

[0017] a common element determining module, configured to determine common elements shared by all the road instances based on edges of the road instances in the instantiated image;

[0018] a semantic determination module, configured to perform semantic division on the common elements according to each of the road instances to obtain semantic information of the common elements; and

[0019] The image processing module is used to perform image editing on each road instance in the instantiated image based on the semantic information of the common elements to generate a road image.

[0020] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a road image generation method when executing the computer program.

[0021] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a road image generation method.

[0022] In a fifth aspect, the present application further provides a computer program product, which includes a computer program that implements the steps of a road image generation method when executed by a processor.

[0023] The details of one or more embodiments of the present application are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the present application will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the disclosed drawings without any creative work.

[0025] FIG1 is a diagram showing an application environment of a road image generation method according to an embodiment;

[0026] FIG2 is a schematic flow chart of a method for generating a road image in one embodiment;

[0027] FIG3 is a schematic diagram of a road image in one embodiment;

[0028] FIG4 is a schematic diagram of a road surface image in one embodiment;

[0029] FIG5 is a schematic diagram of an instantiated image in one embodiment;

[0030] FIG6 is a schematic diagram of a road image in one embodiment;

[0031] FIG7 is a schematic flow chart of a method for generating a road image in another embodiment;

[0032] FIG8 is a schematic diagram of processing results of each step of a road image generation method according to an embodiment;

[0033] FIG9 is a schematic diagram of an intersection image in one embodiment;

[0034] FIG10 is a schematic diagram of a conventional solution processing result in one embodiment;

[0035] FIG11 is a schematic diagram of a road image in another embodiment;

[0036] FIG12 is a schematic diagram of semantic information of a road image according to one embodiment;

[0037] FIG13 is a schematic diagram of a reconstruction result of a large intersection map in one embodiment;

[0038] FIG14 is a schematic diagram showing the result of generating a road image in one embodiment;

[0039] FIG15 is a structural block diagram of a road image generating device according to an embodiment;

[0040] FIG16 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0042] The road image generation method provided in the embodiment of the present application can be applied in the application environment shown in Figure 1. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other servers. The road image generation method can be executed by the terminal 102 or the server 104 alone, or by the terminal 102 and the server 104 in collaboration. Taking the execution of the terminal 102 alone as an example, the terminal 102 performs instance segmentation on the road image to obtain an instance segmentation result, and generates a road surface image based on the instance segmentation result; fills the hole areas in the road surface image, and instantiates the road surface image after the hole filling to obtain an instantiated image; determines the common elements shared by each road instance based on the edge of the road instance in the instantiated image; performs semantic division on the common elements according to each road instance to obtain the semantic information of the common elements; and performs image editing on each road instance in the instantiated image based on the semantic information of the common elements to generate a road image.

[0043] Among them, the terminal 102 can be, but is not limited to, various desktop computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 102 and the server 104 can be directly or indirectly connected through wired or wireless communication, and this application does not impose any restrictions on this.

[0044] In one embodiment, as shown in FIG2 , a road image generation method is provided. The method is described by taking the computer device in FIG1 as an example, and includes the following steps:

[0045] S202 , performing instance segmentation on the road image to obtain instance segmentation results, and generating a road surface image based on the instance segmentation results.

[0046] Among them, road image refers to the image that captures or records roads, streets or roads, which can be obtained by satellites, cameras, drones or other equipment. Road images contain road information, which can also be called road features. Road features include point road features, line road features and surface road features. Point road features refer to specific points or positions on the road, such as the center point of an intersection, the location of a road sign, and a specific location on the road. Point road features can be used to determine intersections, mark special positions on routes, etc.; line road features refer to linear features on the road, such as lane lines, edge lines, road center lines, etc. Line road features can be used to define the geometry and layout of roads, as well as the paths of vehicles; surface road features refer to closed areas of road areas, usually representing the road surface or lanes of roads. Surface road features can be used to describe the overall shape and surface of roads. Road elements are very important for many transportation and geographic information system applications, such as map making, navigation, traffic monitoring, and environmental perception of self-driving cars.

[0047] Instance segmentation is used to assign each pixel in an image to the object or instance to which it belongs. Specifically, the goal of instance segmentation is to detect and segment different object instances in an image, even if they belong to the same category. For example, in an image, instance segmentation can be used to identify different cars, pedestrians, and bicycles and assign a unique identifier to each car, each pedestrian, and each bicycle. In an embodiment of the present application, instance segmentation can use deep learning methods, such as convolutional neural networks (CNNs), to detect and segment object instances in an image.

[0048] Instance segmentation of road images refers to assigning each pixel in the road image to the road instance to which it belongs. Road instance is the actual representation of each road area or road object in the road image. Each road instance has a unique identifier or label, which represents a specific part of the image, which can be a section of road, an intersection, a lane, a median strip or other road-related objects.

[0049] A road surface image is an image used to represent the shape and location of the drivable area of ​​a road, namely the road surface. Specifically, it can be a binary image, where road areas can be represented by white or other distinct colors, while non-road areas are represented by black or other background colors. Figure 3 shows a road image in one embodiment, and Figure 4 shows a road surface image obtained by performing an example segmentation on the road image. This road surface image contains some holes.

[0050] Specifically, after obtaining the road image, the computer device may preprocess the road image to obtain a preprocessed road image, perform instance segmentation on the preprocessed road image to obtain an instance segmentation result, and generate a road surface image based on the instance segmentation result.

[0051] Among them, preprocessing can be image denoising, resizing, contrast enhancement, etc. Image denoising is used to remove noise or interference in the image to improve image quality. Resizing is used to adjust the size of the image to make it suitable for the input requirements of the model or algorithm. Contrast enhancement is used to enhance the contrast of the image to make the details in the image more clearly visible.

[0052] S204 , performing hole filling on the hole areas in the road surface image, and instantiating the hole-filled road surface image to obtain an instantiated image.

[0053] Among them, the hole area refers to the area inside the road area in the road surface image that is marked as a non-road area. The hole area may be caused by noise, occlusion, incomplete data or other factors in image processing. They are represented in the road surface image as an area not covered by the road area, usually appearing in black or other background colors.

[0054] Hole filling refers to eliminating missing road areas in road surface images to ensure the integrity and accuracy of the road areas.

[0055] Instantiation is the operation of forming an independent object, and the formed independent object is called an instance. An independent object refers to an object that can be operated independently. In the present application, instantiation specifically refers to subdividing the road area in the road surface image after hole filling into different road instances in order to better understand and process the road elements in the image, that is, for each sub-area of ​​the road area in the road surface image after hole filling, it is assigned to a specific road instance to obtain an instantiated image, and the instantiated image can more finely identify different parts of the road. When instantiating the road surface image after hole filling, at least one road area in the road surface image can be specifically identified, and different road areas represent different roads; based on the identified road areas, the position of each road area is located in the road surface image after hole filling to obtain a road instance, and finally an instantiated image containing the obtained road instance is obtained.

[0056] Specifically, after obtaining the road surface image, the computer device performs hole detection on the road surface image to obtain the hole area in the road surface image, and fills the hole area in the road surface image to obtain the road surface image after hole filling. The original road area and the hole filling area in the road surface image after hole filling are respectively instantiated to obtain an instantiated image.

[0057] As shown in Figure 5, an instantiated image is shown, in which each different filled area represents a different instance object, such as the road surface marking an intersection. The area filled with diagonal lines in the center of Figure 5 represents the road surface at the intersection, and the area filled with oblique square grids in the lower half of Figure 5 represents the isolation zone.

[0058] S206 : Determine common elements shared by all road instances based on the edges of the road instances in the instantiated image.

[0059] Edges are the dividing lines between instances and other instances or non-road areas, marking the boundaries or borders between different road instances or between roads and non-road areas. These edges visually represent the outlines or boundaries of road instances in the image.

[0060] Common elements refer to at least one of the connection points and connection edges shared between different road instances. Shared connection points can also be called key points, and shared connection edges can also be called key edges. Key points and key edges represent the association and adjacency relationship between different road instances and are used to connect different instances together to indicate their topological relationship in the image.

[0061] Specifically, after obtaining the instantiated image, the computer device can extract the edges of each road instance in the instantiated image, and determine the connection relationship between the road instances in the instantiated image based on the edges of each road instance, and determine the common elements shared by each road instance based on the determined connection relationship between the road instances.

[0062] S208 , performing semantic division on the common elements according to each road instance to obtain semantic information of the common elements.

[0063] Among them, semantic segmentation refers to the process of associating each common element with specific semantic information or semantic categories. In other words, semantic segmentation is the process of classifying each common element (such as key points and key edges) into different semantic categories or labels to indicate their meaning and role in the image.

[0064] Common element semantic information refers to the semantic classification or labeling of common elements. These semantic classifications or labels are used to indicate the meaning and role of each key element in the image, facilitating better understanding and processing of road images. Common element semantic information includes at least one of edge semantic information and point semantic information. Edge semantic information refers to the semantic information of key edges, while point semantic information refers to the semantic information of key points.

[0065] Specifically, after obtaining the common elements in the instantiated image, the computer device determines the road instance to which any common element belongs, performs semantic segmentation on the common element based on the road instance to which it belongs, and obtains semantic information of the common element.

[0066] S210 , performing image editing on each road instance in the instantiated image based on semantic information of the common elements to generate a road image.

[0067] Image editing can specifically involve semantically based visualization processing, which can enrich the image's information content and improve its readability. Semantic-based visualization processing can specifically include color marking, line width, and line type marking. Color marking refers to semantically applying different colors to different road instances and different parts of road instances to distinguish different road features. Line width and line type marking refers to semantically applying lines to different parts of the edges of different road instances to distinguish different types of road markings.

[0068] Specifically, after obtaining the semantic information of each common element, the computer device performs semantic optimization on the corresponding road instance based on the semantic information of each common element to obtain the semantically optimized road instance, and generates a road image based on the semantically optimized road instance.

[0069] As shown in Figure 6, a road image in one embodiment, different filled areas in Figure 6 represent different road instances, and the same filling pattern represents the same semantics, that is, the category of the road instances is the same. For example, the area filled with an oblique square grid in the upper half of Figure 6 represents an isolation belt, and the area filled with an oblique square grid in the lower half of Figure 6 represents an isolation belt.

[0070] The above-mentioned road image generation method obtains a road surface image by performing instance segmentation on the road image, obtains an instanced image by filling the hole areas in the road surface image, and instantiates the road surface image after filling the holes. The common elements shared by each road instance are determined based on the edges of the road instance in the instanced image; the common elements are semantically divided according to each road instance to obtain the semantic information of the common elements; and the road instances in the instanced image are image edited based on the semantic information of the common elements to generate the road image. The hole filling and instantiation processing can avoid the occurrence of holes in the reconstructed image due to occlusion problems of the road image, ensure the integrity and continuity of each road instance (such as the road surface of the section and the road surface of the intersection) in the final generated road image, and improve the accuracy of the road image. The common elements can be determined by determining the relationship between adjacent instances and performing semantic division. Further combined with image editing based on semantic information, the final generated road image can contain more accurate and rich semantic information, thereby improving the information richness and accuracy of the road image.

[0071] In one embodiment, a computer device performs instance segmentation on a road image to obtain instance segmentation results, and generates a road surface image based on the instance segmentation results. The process includes the following steps: segmenting different road instances from the road image; performing regional fusion on each road instance to obtain a road surface area that is fused with at least one road instance; and generating a road surface image with the road surface area that is fused with at least one road instance as the foreground.

[0072] Among them, regional fusion refers to merging the road surface areas corresponding to adjacent or overlapping road instances into a larger continuous area.

[0073] Specifically, after obtaining the road image, the computer device uses a pre-trained segmentation model to extract features from the road image to obtain road features, and performs instance segmentation processing based on the road features to obtain instance segmentation results. The instance segmentation results include different segmented road instances, semantic categories corresponding to each road instance, and instance identifiers corresponding to each road instance. Each road instance corresponds to a sub-area of ​​the road area. After obtaining each road instance, the sub-areas corresponding to each road instance are merged in sequence according to their respective positions in the road image until all road instances are merged to obtain a road surface area. The fused road surface area is used as the foreground, and the background is set to a background color, such as black or other background colors, to generate a road surface image.

[0074] For example, the road image is segmented to obtain a semantic mask of each road instance, and each semantic mask is fused to obtain a merged mask, which is then used as the foreground semantics to generate a road surface image.

[0075] In the above embodiment, the computer device segments different road instances from the road image; performs regional fusion on each road instance to obtain a road surface area that is fused with at least one road instance; and generates a road surface image with the road surface area that is fused with at least one road instance as the foreground, thereby obtaining a road surface image containing the basic road surface area, providing a basis for the subsequent production of road images.

[0076] In one embodiment, the process of a computer device filling a hole area in a road surface image includes the following steps: performing hole detection on the road surface image to obtain candidate hole areas; determining the hole areas in the candidate hole areas that meet the filling conditions; and performing dilation processing on the road surface areas adjacent to the hole areas in the road surface image to obtain the road surface image after the holes are filled.

[0077] Among them, hole detection refers to the identification and positioning of hole areas in road surface images. Hole areas refer to areas in road surface images that are marked as non-road areas within the road area. Hole areas may be caused by noise, occlusion, incomplete data or other factors in image processing. They are represented in road surface images as areas not covered by the road area, usually appearing in black or other background colors.

[0078] Candidate hole regions refer to regions that are identified as potential holes or need to be filled. These regions are identified as possible candidates in the hole detection stage, but further verification is required to determine whether they actually need to be filled.

[0079] The filling condition refers to the condition used to determine whether the candidate hole area needs to be filled, which may specifically include at least one of the hole size condition, hole shape condition, hole position condition, and context information condition. The hole size condition is used to determine the minimum and maximum allowable sizes of the hole. It can be understood that a hole that is too small may be regarded as noise and not filled, while a hole that is too large may not be a real hole and therefore does not need to be filled; the hole shape condition is used to determine the shape of the hole that needs to be filled, for example, the filled hole can be limited to a rectangular, circular or other regular shape; the hole position condition is used to determine the location of the hole that needs to be filled, for example, if the hole is located in a non-core area of ​​the road or on a non-main path for vehicles, it does not need to be filled; the context information condition is used to limit the relationship between the filled hole and the surrounding road area, for example, the hole is only filled when there is sufficient context information to indicate that the hole is reasonable.

[0080] Dilation processing is used to expand the road area to the surrounding pixels to fill the hole area.

[0081] Specifically, after obtaining the road surface image, the computer device can perform a connected domain analysis on the road surface image to obtain a connected domain analysis result, determine candidate hole areas based on the connected domain analysis result, and determine whether each candidate hole area meets the filling condition, so as to screen out the hole areas that meet the filling condition from each candidate hole area for hole filling. When performing hole filling, the adjacent road surface area corresponding to the hole area can be determined, and the expansion parameter can be determined based on the positional relationship and size relationship between the determined adjacent road surface area and the hole area, and the adjacent road surface area can be expanded based on the expansion parameter, so that the pixels of the road surface area facing the street are expanded to the hole area, thereby filling the hole area and obtaining the road surface image after hole filling.

[0082] For example, after the merged mask is used as foreground semantics to generate a road surface image, the merged mask can be expanded to fill the hole image in the road surface image and obtain a hole-filled road surface image.

[0083] In the above embodiment, the computer device performs hole detection on the road surface image to obtain candidate hole areas; determines the hole areas in the candidate hole areas that meet the filling conditions; and performs expansion processing on the road surface areas adjacent to the hole areas in the road surface image to obtain the road surface image after the holes are filled, thereby improving the accuracy of the shape and position information of the road area in the road surface image, and further making the road image generated based on the road surface image after the holes are filled more consistent with the actual road conditions, thereby improving the accuracy of the road image.

[0084] In one embodiment, the road surface image after hole filling includes the hole-filled area and other road surface areas. The computer device instantiates the road surface image after hole filling, and the process of obtaining the instantiated image includes the following steps: instantiating the other road surface areas to obtain an initial instantiated image; instantiating the hole-filled area in the initial instantiated image to obtain an instantiated image.

[0085] The hole filling area refers to the road surface area obtained after filling the hole area; the other road surface areas refer to the road surface areas obtained by regional fusion of the original road instances in the road surface image.

[0086] Specifically, after obtaining the road surface image after hole filling, the computer device can instantiate other road surface areas in the road surface image based on the instance recognition results of the original road image to obtain an initial instantiated image, and instantiate the hole-filled areas in the initial instantiated image according to the size characteristics of the hole-filled areas to obtain an instantiated image.

[0087] In one embodiment, each hole-filling region can be divided into a first hole-filling region and a second hole-filling region according to its size characteristics, wherein the first hole-filling region refers to a hole-filling region whose hole size reaches a size threshold, and the second hole-filling region refers to a hole-filling region whose hole size does not reach the size threshold. For example, a hole-filling region whose hole area reaches an area threshold is determined as the first hole-filling region, and a hole-filling region whose hole area does not reach the area threshold is determined as the second hole-filling region; the first hole-filling region can be instantiated according to the first instantiation method, and the second hole-filling region can be instantiated according to the second instantiation method.

[0088] In the above embodiment, the computer device instantiates other road surface areas to obtain an initial instantiated image; and instantiates the hole-filled area in the initial instantiated image to obtain an instantiated image, so that different sub-areas of the road surface area can be divided into different road instances, so that the road elements in the road image can be better modeled and processed, thereby improving the accuracy of the road image.

[0089] In one embodiment, a computer device instantiates other road surface areas to obtain an initial instantiated image, including the following steps: obtaining an instance segmentation result obtained by instance segmenting a road image; the instance segmentation result includes at least two road instances and corresponding confidence levels; for a sub-area of ​​each road surface area in the road surface image after hole filling, determining at least one candidate road instance in at least two road instances that matches the shape and position of the sub-area; when the number of candidate road instances is at least two, selecting a target road instance whose confidence level meets the confidence level condition from the candidate road instances; and filling the sub-area based on the target road instance to obtain an initial instantiated image.

[0090] Among them, confidence is used to represent the degree of credibility of the recognition of the corresponding road instance. High confidence indicates that the recognition result is very reliable, while low confidence indicates that there is a certain degree of uncertainty in the recognition result.

[0091] Specifically, the computer device determines the matching degree between the shape and position of each road instance and the sub-region in each road surface area in the road surface image after the hole is filled, and selects a candidate road instance that matches the sub-region from each road instance based on the matching degree. When the number of candidate road instances is one, the candidate road instance is directly determined as the target road instance, and the sub-region is filled based on the target road instance; when the number of candidate road instances is at least two, a target road instance that meets the confidence condition is selected from each candidate road instance, and the sub-region is filled based on the target road instance. After all the road instances obtained by segmentation are filled into other road areas, an initial instantiated image is obtained.

[0092] It should be noted that, in the embodiment of the present application, filling the sub-area based on the target road instance can specifically be filling the sub-area based on the instance identification of the target road instance. For example, different target road instances use different filling patterns, and the sub-area is filled with the filling pattern corresponding to the instance identification of the target road instance, thereby obtaining the filling result shown in Figure 5; in addition, the sub-area can also be filled based on the semantic category of the target road instance. For example, the corresponding filling pattern is determined based on the semantic category of the target road instance, and the corresponding sub-area is filled based on the determined filling pattern.

[0093] In the above embodiment, when the computer device instantiates other road surface areas based on the instance recognition results of the original road image, for any sub-area, it fills it by selecting the target road instance that matches its shape and position, thereby ensuring that the final generated instantiated image retains the road instance information in the original road image, and by selecting the target road instance whose confidence meets the confidence condition to fill it, the accuracy of the filling can be improved, and the consistency between the road instance in the instantiated image and the road instance in the original road image is further improved, thereby improving the accuracy of the final road image.

[0094] In one embodiment, the hole-filling region includes a first hole-filling region and a second hole-filling region; the area of ​​the first hole-filling region is larger than that of the second hole-filling region; the initial instantiated image may also be referred to as a first instantiated image, and the computer device instantiates the hole-filling region in the initial instantiated image, and the process of obtaining the instantiated image includes the following steps: determining the instance category based on the adjacent instances of the first hole-filling region; in the first instantiated image, generating a new road instance in the first hole-filling region according to the instance category to obtain a second instantiated image; determining the road instance to which the second hole-filling region belongs; and in the second instantiated image, instantiating the second hole-filling region based on the attributed road instance to obtain an instantiated image.

[0095] Among them, the adjacent instance is the road instance adjacent to the first hole-filling area in the first instantiated image; the determined instance category is the semantic category of the possible road instance of the first hole-filling area. For example, if the road instances adjacent to a certain long strip of the first hole-filling area are road sections, intersections, etc., then it can be determined that the instance category corresponding to the first hole-filling area is an isolation belt.

[0096] Specifically, after obtaining the first instantiated image, the computer device obtains the adjacent instances of any first hole-filling region in the first instantiated image, determines the instance category corresponding to the first hole-filling region based on the semantic category of the adjacent instances through a pre-trained machine learning model, and generates a new road instance in the first hole-filling region according to the determined instance category. After completing the instantiation of each first hole-filling region, a second instantiated image is obtained. After obtaining the second instantiated image, the computer device determines the adjacent instances of the second hole-filling region for the second hole-filling region in the second instantiated image, and selects the road instance to which the second hole-filling region can be attributed from each adjacent instance according to a preset attribution rule, and attributes the second hole region to the road instance. Specifically, the second hole-filling region can be filled based on the semantics of the attributed road instance, and the common boundary line between the second hole-filling region and the attributed road instance is eliminated to achieve the attribution of the second hole region. After completing the instantiation processing of all second hole-filling regions, a second instantiated image is obtained.

[0097] The attribution rules can be pre-set based on attributes such as type, position, shape, and color of adjacent instances.

[0098] In the above embodiment, the computer device determines the instance category based on the adjacent instances of the first hole-filling area; in the first instantiated image, a new road instance is generated in the first hole-filling area according to the instance category to obtain a second instantiated image; the road instance to which the second hole-filling area belongs is determined; in the second instantiated image, the second hole-filling area is instantiated based on the attributed road instance to obtain an instantiated image, thereby accurately instantiating the hole-filling area, improving the consistency between the road instance in the instantiated image and the road instance in the original road image, and thereby improving the accuracy of the final road image.

[0099] In one embodiment, a computer device determines common elements shared by various road instances based on the edges of the road instances in the instantiated image, including: performing edge detection on the instantiated image to obtain the edges of the road instances in the instantiated image; determining the skeleton of the road instances in the instantiated image based on the edges; and determining the common elements shared by various road instances based on the connection relationships in the road instance skeletons.

[0100] Edge detection is used to identify the outlines of each road instance in the instantiated image. The edges obtained can specifically be the inner edges of the road instance. The road instance skeleton is the main structural representation of the road instance extracted from the road image. Its essence is a series of interconnected line segments. The interconnectedness of the line segments represents the relationship between the road instances.

[0101] The common elements include at least one of key points and key edges. Key points are usually special positions in the hanger, such as intersections, and key edges are line segments connecting key points.

[0102] Specifically, after obtaining the instantiated image, the computer device can use a preset edge detection algorithm to perform edge detection on the instantiated image to obtain the edges of each road instance in the instantiated image, use a semantic skeleton extraction algorithm to process the edges to obtain a road instance skeleton in the instantiated image, analyze the road instance skeleton based on the connection relationship in the road instance skeleton to identify the key points therein, and identify the key edges based on the connection relationship between the key points, thereby obtaining the common elements of each road instance.

[0103] The preset edge detection algorithm may be a Sobel operator, a Canny edge detection algorithm, a Laplacian operator, or an edge detection filter. The semantic skeleton extraction algorithm is used to extract a skeleton structure with semantic information from an instantiated image.

[0104] In the above embodiment, the computer device performs edge detection on the instantiated image to obtain the edges of the road instances in the instantiated image; determines the skeleton of the road instances in the instantiated image based on the edges; and determines the common elements shared by each road instance based on the connection relationship in the road instance skeleton. The common elements can be automatically identified through edge detection and skeleton analysis, thereby improving the accuracy and efficiency of common element identification, so that the semantic information of the common elements can be further determined in the road image, thereby improving the accuracy of the road image and enriching the information content of the road recognition image.

[0105] In one embodiment, common elements include key edges, and the semantic information of the common elements includes edge semantic information. A computer device performs semantic division on the common elements according to each road instance. The process of obtaining the semantic information of the common elements includes the following steps: determining the target road instance to which the key edge belongs; and performing semantic division on the key edge based on the semantic information of the target road instance to obtain edge semantic information.

[0106] Among them, the target road instance is a road instance containing a key edge. In a road scene, the key edge is usually located at the intersection between different road instances. The key edge can express the accurate semantic information of the key edge by associating it with the target road instance to which it belongs.

[0107] Specifically, for any key edge, the computer device can analyze the relationship between the road instances connected to the key edge, and determine the target road instance to which the key edge belongs based on the connection relationship analysis results, obtain the semantic information of the target road instance, and perform semantic division on the key edge based on the semantic information of the target road instance to obtain the edge semantic information of the key edge.

[0108] In the above embodiment, the computer device determines the target road instance to which the key edge belongs, performs semantic segmentation on the key edge based on the semantic information of the target road instance, and obtains edge semantic information. This allows the computer device to associate the key edges in the image with the target road instance and assign them semantic information so that the obtained edge semantic information can be subsequently added to the road image, thereby increasing the information content of the road image.

[0109] In one embodiment, the common elements include key points, and the semantic information of the common elements includes point semantic information. The computer device performs semantic division on the common elements according to each road instance. The process of obtaining the semantic information of the common elements includes the following steps: determining the target road instance to which the key points belong; performing semantic division on the key points based on the semantic information of the target road instance to obtain point semantic information.

[0110] Among them, the target road instance is a road instance containing key points. In the road scene, key points are usually located at the intersection between different road instances. By associating the key points with the target road instances to which they belong, the accurate semantic information of the key points can be represented.

[0111] Specifically, for any key point, the computer device can analyze the relationship between the road instances connected to the key point, and determine the target road instance to which the key point belongs based on the connection relationship analysis results, obtain the semantic information of the target road instance, and perform semantic division on the key point based on the semantic information of the target road instance to obtain the point semantic information of the key point.

[0112] In the above embodiment, the computer device determines the target road instance to which the key points belong, performs semantic division on the key points based on the semantic information of the target road instance, and obtains point semantic information, so that the key points in the image can be associated with the target road instance and given semantic information, so that the obtained point semantic information can be subsequently added to the road image, thereby increasing the information content of the road image.

[0113] In one embodiment, the computer device may also determine a processing method for key edges based on the semantic information of the target road instance; optimize the key edges according to the processing method to obtain processed key edges; and semantically divide the processed key edges based on the semantic information of the target road instance to obtain edge semantic information.

[0114] The processing method refers to the optimization method of the key edge, which can be parameterized processing or smoothing processing. Parameterized processing refers to expressing the shape of the key edge through parameters to achieve optimization of the key edge shape. For example, the key edge of the curved shape is modeled through a curve function to make it more regular. Smoothing processing refers to optimizing the key edge shape by reducing the irregularity of the key edge, for example, making a jagged key edge smoother.

[0115] Specifically, the computer device determines a processing method for the key edges based on the semantic information of the target road instance and the shape information of the key edges, optimizes the key edges according to the processing method, and obtains processed key edges; and semantically divides the processed key edges based on the semantic information of the target road instance to obtain edge semantic information.

[0116] It should be noted that a processing method library can be pre-configured, and the processing method library contains various processing methods and corresponding semantic conditions and shape conditions. The computer device can match the semantic information of the target road instance and the shape information of the key edge with the semantic conditions and shape conditions, select a matching processing method from the processing method library based on the matching result, and optimize the key edge according to the selected processing method.

[0117] For example, if the target road segment instance of a certain key edge is an intersection and non-road segment isolation, and the shape of the key edge is approximately an arc, then it can be determined that the processing method for the key edge is parametric processing. Specifically, the curve of the key edge can be parameterized modeled, that is, the curve of the key edge is expressed as a parametric equation of an arc, and a smooth arc curve is generated according to the parametric equation. The smooth arc curve is the optimized key edge, and the optimized key edge is approximately the same shape as the original key edge, but smoother; if the target road segment instance of a certain key edge is a road segment and isolation, and the shape of the key edge includes jagged features, such as jagged at the corners, then it can be determined that the processing method for the key edge is smoothing. Specifically, the shape of the key edge can be processed to reduce the number and sharpness of jagged edges or corners to obtain a smoothed key edge. The smoothed key edge is the optimized key edge, and the optimized key edge is smoother and more regular than the original key edge.

[0118] In the above embodiment, the computer device determines a processing method for key edges based on the semantic information of the target road instance; optimizes the key edges according to the processing method to obtain processed key edges; and selects an appropriate processing method based on different semantic information to improve the visual effect of the key edges, such as reducing jagged edges, corners, and irregularities, making the road instance smoother and more natural in the image, and improving the visualization effect; and the processed key edges can better reflect the characteristics of the actual road, thereby improving the accuracy of the road image generated based on the processed key edges.

[0119] In one embodiment, a computer device performs image editing on each road instance in an instantiated image based on semantic information of common elements. The process of generating a road image includes the following steps: adding semantic labels to each road instance in the instantiated image based on the semantic information of the common elements to obtain semantically optimized road instances; and generating a road image based on the semantically optimized road instances.

[0120] After obtaining the edge semantic information of the key edge and the point semantic information of the key point, the specific computer device can associate the semantic label corresponding to the edge semantic information of any key edge of the road instance with the key edge, and associate the semantic label corresponding to the point semantic information of the key point with any key point of the road instance to obtain a semantically optimized road instance. When reconstructing the image, the corresponding semantically optimized road instance is drawn according to the semantic labels associated with each element in the points, lines and surfaces of each semantically optimized road instance, thereby generating a road image.

[0121] In the above embodiment, the computer device adds semantic tags to each road instance in the instantiated image based on the semantic information of the common elements to obtain a semantically optimized road instance, so that different elements of the road instance are associated with specific semantic information. Then, a road image is generated based on the semantically optimized road instance, which contains richer information, improves the amount of information and accuracy of the road image, and makes the content of the road image easier to understand and analyze. Moreover, when the road image is applied to fields such as road planning, autonomous driving, and urban planning, it makes it easier for users to understand and analyze the road structure and elements in the image.

[0122] In one embodiment, after the computer device generates a road image, it can also obtain road elements to be added; determine the element addition position of the road elements to be added in the road image; and draw the road elements to be added at the element addition position according to the semantic information of the elements to be added, thereby obtaining an element-optimized road image.

[0123] Among them, the road elements to be added refer to the additional elements that need to be added to the road image. These elements are related to the road scene, and can be new road segments, road signs and traffic signs, pedestrians and vehicles, traffic lights and signals, road markings and curbs, road conditions, buildings and environment, etc. Adding new road segments refers to adding new road segments or road extensions to the road image to expand or connect existing roads; adding road signs and traffic signs refers to adding road signs, traffic signs, signs, etc. to the road image to identify information, rules and road conditions on the road; adding pedestrians and vehicles refers to adding pedestrians, bicycles, cars, buses, etc. to the road image. Adding traffic lights and signals refers to adding traffic lights, pedestrian signals, etc. at intersections or road sections in the road image to simulate traffic control and traffic flow; Adding road markings and curbs refers to drawing road markings, center lines, curbs, etc. in the road image to enhance the recognizability and realism of the road; Adding road conditions refers to adding road conditions such as rain, accumulated water, ice and snow to the road image to simulate different weather and road conditions; Adding buildings and environment refers to adding environmental elements such as buildings, trees, light poles, billboards, etc. around the road in the road image to provide a more realistic scene.

[0124] Specifically, the computer device can obtain the element information of the element to be added, which may include the semantic information, shape, material and other information of the element to be added, and determine the element addition position of the element to be added according to the actual geographic data or scene requirements, and then use the image processing tool to draw the road element to be added at the element addition position according to the semantic information of the element to be added to obtain an element-optimized road image.

[0125] In the above embodiment, the computer device obtains the road element to be added; determines the element addition position of the road element to be added in the road image; and draws the road element to be added at the element addition position according to the semantic information of the element to be added, thereby obtaining an element-optimized road image, thereby increasing the amount of information in the road image and making the element-optimized road image closer to the real road scene.

[0126] In one embodiment, as shown in FIG7 , a road image generation method is further provided. The road image generation method is described by taking the method applied to the computer device in FIG1 as an example, and includes the following steps:

[0127] S702, segmenting different road instances from the road image; performing regional fusion on each road instance to obtain a road surface region that fuses all road instances; and generating a road surface image with the road surface region that fuses all road instances as the foreground.

[0128] S704, performing hole detection on the road surface image to obtain candidate hole areas; determining hole areas in the candidate hole areas that meet filling conditions; performing dilation processing on the road surface areas adjacent to the hole areas in the road surface image to obtain a road surface image after hole filling.

[0129] The road surface image after hole filling includes a hole filling area and other road surface areas; the hole filling area includes a first hole filling area and a second hole filling area, and the area of ​​the first hole filling area is larger than that of the second hole filling area.

[0130] S706 , instantiating other road surface areas based on the instance segmentation result obtained by performing instance segmentation on the road image to obtain an initial instantiated image; the initial instantiated image is a first instantiated image.

[0131] S708 , determining an instance category based on adjacent instances in the first hole-filling area; generating a new road instance in the first hole-filling area according to the instance category in the first instantiated image to obtain a second instantiated image.

[0132] S710 , determining a road instance to which a second hole filling region belongs; and instantiating the second hole filling region based on the road instance in a second instantiated image to obtain an instantiated image.

[0133] S712, performing edge detection on the instantiated image to obtain edges of the road instances in the instantiated image; determining a skeleton of the road instances in the instantiated image based on the edges; and determining common elements shared by all road instances based on connection relationships in the road instance skeletons.

[0134] S714 , determining the target road instance to which the key edge belongs; determining a processing method for the key edge based on the semantic information of the target road instance; and optimizing the key edge according to the processing method to obtain a processed key edge.

[0135] S716 , semantically segmenting the processed key edges based on the semantic information of the target road instance to obtain semantic information of common elements; and performing image editing on each road instance in the instantiated image based on the semantic information of the common elements to generate a road image.

[0136] S718 , determining an element adding position of the road element to be added in the road image; drawing the road element to be added at the element adding position according to the semantic information of the element to be added, and obtaining an element-optimized road image.

[0137] The present application also provides an application scenario, in which the above-mentioned road image generation method can be applied. The method specifically includes the following steps:

[0138] Step 1: Obtain the original satellite image perception results

[0139] The instance segmentation algorithm is used to perform instance segmentation on the road satellite image to obtain the instance segmentation result. The instance segmentation result can also be called the satellite image perception result. The instance segmentation result includes each segmented road instance, such as intersections, road sections, barriers, guide strips, main and auxiliary road entrances and exits, U-turns, roundabouts, etc. It can be understood that if a road image is generated directly based on the instance segmentation result, as shown in (1) in Figure 8, the resulting road image will have problems such as overlapping, non-fitting or voids, and the relationship between adjacent instances cannot be known, and the morphological structure of the instance cannot be guaranteed to be consistent with the actual situation.

[0140] For example, if the large intersection image shown in Figure 9 is directly reconstructed based on the instance segmentation results, there may be problems such as holes (i.e., voids) and poor connectivity as shown in Figure 10 (a). There may also be problems such as holes, missing isolation belts, and poor morphology as shown in Figure 10 (b). There may also be problems such as holes, missing isolation belts with poor morphology, and missing isolation belts as shown in Figure 10 (c).

[0141] Step 2: Generate the road surface of the drivable area after filling the holes

[0142] All road instances identified in step 1 are fused into a mask image, which becomes a foreground-background semantic mask, to obtain a road surface image containing the road area, and detect candidate hole areas in the road surface image. The hole areas that meet the conditions are selected from the candidate hole areas, and the foreground mask in the road surface image is expanded and eroded to fill the hole areas in the road surface image. The road surface image after hole filling is obtained, as shown in (2) in Figure 8, where there is no hole in the road surface area of ​​the road surface image.

[0143] It can be understood that the drivable area in the hole-filled road surface image generated in this step is the entire road surface, which is the overall data base for subsequent generation of the road image.

[0144] Step 3: Road surface instantiation

[0145] Based on the instance segmentation result obtained in step 1, the original road surface area in the road surface image is instance-filled, that is, the covering road surface is assigned instance semantics; a new instance generation operation is performed on the large area in the hole-filling area in the road surface image, and the small area in the hole-filling area is assigned to other nearby instances, thereby achieving instance filling of the entire road surface area of ​​the road surface image, that is, the instance semantic pixels fill the road surface without leaving any gaps, and an instanced image is obtained, as shown in (3) in Figure 8, where the entire road surface area is completely filled with instances.

[0146] Step 4: Extract instance edges

[0147] Edge extraction is performed on the instantiated image in step 3. The extracted edges can be the inner edges of the instance, as shown in (4) in Figure 8. All edges of the instance are generated with an edge thickness of 2. There is no common edge expression between instances, and no common point expression between multiple instances. Therefore, the edges in step 4 are only basic edges.

[0148] Step 5: Extract common elements between instances

[0149] Using the semantic skeleton edge extraction algorithm, the connectivity relationship is determined based on the extracted edges, and the key points and key edges of all instances are solved based on the connectivity relationship, as shown in (5) in Figure 8. The black dots in the figure are key points, and the line segments connecting the black dots are key edges. The key points and key edges connect various instances (intersections, road sections, isolation, etc.).

[0150] Step 6: Optimize common elements

[0151] According to the semantics of the instance, key edges are smoothed and fitted, for example, the edges of intersections are parameterized to generate smooth arcs, and the edge lines of some road sections are smoothed or straightened, as shown in (6) in Figure 8, making each instance more beautiful and regular. Among them, each adjustment of a key edge is actually an adjustment of the edges of two instances, and multiple instances can be modified synchronously.

[0152] Step 7: Deconstructing instance edge attributes

[0153] The instantiation parameterized expression analysis of each instance is performed, and the instantiation definition of intersections and sections is as follows: for example, intersections are divided into: intersection faces, intersection points, intersection boundaries, and road surface entry and exit edges; parsing intersection and road segment instances generates vectorized semantic intersection definitions - intersection faces, intersection edges, intersection entry and exit edges, and endpoints.

[0154] Semantic division is performed based on the instances to which common elements belong. For example, the key edges between intersections and road sections are the entry and exit edges of the road surface. The common edges between intersection instances and non-drivable areas are the road surface boundaries. Key points are the connection points of key edges with different attributes. The key points and key edges are combined to form the entire intersection surface.

[0155] A road image is generated based on the semantic segmentation results. The obtained road image is shown in (7) in Figure 8. Each element in the image is filled with semantics, that is, each point, each line, and each surface in (7) in Figure 8 has actual semantics. For details, please refer to Figures 11 and 12. Figure 11 is a road image, and the actual semantics of the points, each line, and each surface in Figure 11 are shown in Figure 12.

[0156] Step 8: Fill in new features

[0157] Based on the instantiation of the intersection section generated in the first seven steps, the elements of lane lines, stop lines, guide strips, isolation strips, and ground vehicle lines are added to improve the information of the road surface and obtain the element-optimized road image, as shown in (8) in Figure 8.

[0158] The present application also provides an application scenario, which can be specifically a map scenario. After obtaining the road image, the map application automatically adopts the road image generation method provided by the present application to automatically realize 6-level multi-lane data mapping to obtain the road image, and simultaneously output the lane lines and their related attributes in the road image; in addition, the automation of the large-scale intersection map can also be realized. Based on the structured intersection section data and combined with the basic information of spatial data (SD), the fully automatic data mapping of the large-scale intersection map is completed to meet the rendering requirements of the large-scale intersection map and render a new generation of real-life intersection maps. Specifically, the computer device can automatically obtain a road image at an intersection, perform instance segmentation processing on the road image, generate a road surface image based on the instance segmentation result, and fill the hole areas in the road surface image (such as holes formed due to occlusion problems in the original road image) to obtain a hole-filled road surface image. The original road area and the hole-filled area in the hole-filled road surface image are respectively instantiated, and the key points and key edges of the road instance in the instantiated image are extracted. The key edges are optimized, the semantic information of each key point and key edge is determined, and the image of each road instance in the instantiated image is edited based on the semantic information of each key point and key edge to generate a road image, thereby realizing the automatic generation of road images. In order to better apply to lane-level navigation, autonomous driving, road planning, and urban planning, road elements can also be added to the road image, such as road signs, traffic signs, road markings, curbs, road conditions, buildings, and the environment, so as to obtain an element-optimized road image (such as a large intersection map, refer to the schematic diagram of the reconstruction result of the large intersection map shown in Figure 13). The element-optimized road image is used as a map image in the electronic map. On the one hand, it can make it easier for users to understand and analyze road structures and elements. On the other hand, when performing lane-level navigation and autonomous driving, lane-level path planning can be implemented based on the map image, and when performing lane-level navigation, lane-level navigation guidance can be displayed in the map image. In addition, in the fields of road planning and urban planning, road images can assist in realizing road network visualization or urban environment visualization, so that planners can simulate and evaluate road network, land use, traffic flow and other data based on road images to formulate reasonable planning schemes.

[0159] In addition, the satellite image shown in (a) of Figure 14 was reconstructed using the road image generation method provided by the present application and the traditional reconstruction method, respectively, to obtain the mapping effect of the traditional scheme as shown in (b) of Figure 14, and the mapping effect of the road image generation method provided by the present application as shown in (c) of Figure 14. It can be seen from the figure that the mapping effect of the road image generation method provided by the present application is better, such as each road instance is more complete and regular.

[0160] It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0161] Based on the same inventive concept, embodiments of the present application also provide a road image generation device for implementing the aforementioned road image generation method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more road image generation device embodiments provided below can be found in the aforementioned limitations of the road image generation method and will not be further elaborated here.

[0162] In one embodiment, as shown in FIG15 , a road image generation device is provided, comprising: a road surface image generation module 1502 , an instantiated image generation module 1504 , a common element determination module 1506 , a semantics determination module 1508 , and an image processing module 1510 , wherein:

[0163] The road surface image generation module 1502 is configured to perform instance segmentation on the road image, obtain instance segmentation results, and generate a road surface image based on the instance segmentation results.

[0164] The instantiated image generation module 1504 is configured to fill the hole areas in the road surface image and instantiate the hole-filled road surface image to obtain an instantiated image.

[0165] The common element determination module 1506 is configured to determine common elements shared by all road instances based on the edges of the road instances in the instantiated image.

[0166] The semantic determination module 1508 is used to perform semantic division on the common elements according to each road instance to obtain semantic information of the common elements.

[0167] The image processing module 1510 is configured to perform image editing on each road instance in the instantiated image based on the semantic information of the common elements to generate a road image.

[0168] In the above embodiment, instance segmentation is performed on the road image to obtain an instance segmentation result, and a road surface image is generated based on the instance segmentation result. Holes in the road surface image are filled and the hole-filled road surface image is instantiated to obtain an instantiated image. Common elements shared by each road instance are determined based on the edges of the road instance in the instantiated image. The common elements are semantically divided according to each road instance to obtain semantic information of the common elements. Image editing is performed on each road instance in the instantiated image based on the semantic information of the common elements to generate a road image. Hole filling and instantiation processing can avoid holes in the reconstructed image caused by occlusion problems of the road image, ensure the integrity and continuity of each road instance (such as road section pavement and intersection pavement) in the final generated road image, and improve the accuracy of the road image. By determining the common elements and performing semantic division, the relationship between adjacent instances can be determined. Further combined with image editing based on semantic information, the final generated road image can contain more accurate and rich semantic information, thereby improving the information richness and accuracy of the road image.

[0169] In one embodiment, the road surface image generation module 1502 is further used to: segment different road instances from the road image; perform regional fusion on each road instance to obtain a road surface area that is fused with at least one road instance; and generate a road surface image with the road surface area that is fused with at least one road instance as the foreground.

[0170] In one embodiment, the instantiated image generation module 1504 is further used to: perform hole detection on the road surface image to obtain candidate hole areas; determine the hole areas in the candidate hole areas that meet the filling conditions; and perform expansion processing on the road surface areas adjacent to the hole areas in the road surface image to obtain the road surface image after the holes are filled.

[0171] In one embodiment, the road surface image after hole filling includes the hole-filled area and other road surface areas; the instantiated image generation module 1504 is further used to: instantiate other road surface areas to obtain an initial instantiated image; instantiate the hole-filled area in the initial instantiated image to obtain an instantiated image.

[0172] In one embodiment, the instantiated image generation module 1504 is further used to: obtain instance segmentation results obtained by performing instance segmentation on the road image; the instance segmentation results include at least two road instances and corresponding confidence levels; for a sub-region of each road surface area in the road surface image after hole filling, determine a candidate road instance that matches the shape and position of the sub-region in at least two road instances; when the number of candidate road instances is at least two, select a target road instance whose confidence level meets the confidence level condition from the candidate road instances; and fill the sub-region based on the target road instance to obtain an initial instantiated image.

[0173] In one embodiment, the hole filling region includes a first hole filling region and a second hole filling region, and the area of ​​the first hole filling region is larger than the area of ​​the second hole filling region; the initial instantiated image is the first instantiated image; the instantiated image generation module 1504 is further used to: determine the instance category based on the adjacent instances of the first hole filling region; in the first instantiated image, generate a new road instance in the first hole filling region according to the instance category to obtain a second instantiated image; determine the road instance to which the second hole filling region belongs; in the second instantiated image, instantiate the second hole filling region based on the attributed road instance to obtain an instantiated image.

[0174] In one embodiment, the common element determination module 1506 is further used to: perform edge detection on the instantiated image to obtain the edges of the road instances in the instantiated image; determine the skeleton of the road instances in the instantiated image based on the edges; and determine the common elements shared by all road instances based on the connection relationships in the road instance skeletons.

[0175] In one embodiment, the common elements include key edges, and the semantic information of the common elements includes edge semantic information. The semantic determination module 1508 is further used to: determine the target road instance to which the key edge belongs; and perform semantic division on the key edge based on the semantic information of the target road instance to obtain edge semantic information.

[0176] In one embodiment, the semantic determination module 1508 is further used to: determine a processing method for key edges based on the semantic information of the target road instance; optimize the key edges according to the processing method to obtain processed key edges; and semantically divide the processed key edges based on the semantic information of the target road instance to obtain edge semantic information.

[0177] In one embodiment, the image processing module 1510 is further configured to: add semantic labels to each road instance in the instantiated image based on semantic information of the common elements to obtain semantically optimized road instances; and generate a road image based on the semantically optimized road instances.

[0178] In one embodiment, the image processing module 1510 is further used to: obtain the road element to be added; determine the element addition position of the road element to be added in the road image; and draw the road element to be added at the element addition position according to the semantic information of the element to be added, so as to obtain an element-optimized road image.

[0179] Each module in the aforementioned road image generation device may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor within a computer device in the form of hardware, or may be stored in a computer device memory in the form of software, allowing the processor to call and execute the corresponding operations of each module.

[0180] In one embodiment, a computer device is provided, which may be a server. Its internal structure diagram may be as shown in FIG16 . The computer device includes a processor, memory, an input / output (I / O) interface, and a communication interface. The processor, memory, and I / O interface are connected via a system bus, and the communication interface is connected to the system bus via the I / O interface. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operating system and computer program in the non-volatile storage medium. The database of the computer device stores data required for executing a road image generation method and / or road images generated by executing the road image generation method. The I / O interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a road image generation method.

[0181] Those skilled in the art will understand that the structure shown in Figure 16 is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0182] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0183] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0184] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0185] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0186] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0187] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0188] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A method for generating a road image, executed by a computer device, the method comprising: Performing instance segmentation on the road image to obtain instance segmentation results, and generating a road surface image based on the instance segmentation results; Filling holes in the hole areas in the road surface image, and instantiating the hole-filled road surface image to obtain an instantiated image; Determining common elements shared by each of the road instances based on edges of the road instances in the instantiated image; Semantically dividing the common elements according to each of the road instances to obtain semantic information of the common elements; and Image editing is performed on each road instance in the instantiated image based on the semantic information of the common element to generate a road image.

2. The method according to claim 1, wherein the instance segmentation is performed on the road image to obtain the instance segmentation result, and the road surface image is generated based on the instance segmentation result, comprising: Segment different road instances from road images; Performing regional fusion on each of the road instances to obtain a road surface region fused with at least one of the road instances; A road surface image is generated with a road surface area fused with at least one of the road instances as a foreground.

3. The method according to claim 1 or 2, wherein the step of filling the hole area in the road surface image comprises: Performing hole detection on the road surface image to obtain candidate hole areas; Determine a hole region among the candidate hole regions that meets a filling condition; The road surface area adjacent to the hole area in the road surface image is expanded to obtain a road surface image after the hole is filled.

4. The method according to any one of claims 1 to 3, wherein the road surface image after hole filling includes a hole filling area and other road surface areas; The instantiating the hole-filled road surface image to obtain an instantiated image includes: Instantiating the other road surface areas to obtain an initial instantiation image; The hole-filled region in the initial instantiated image is instantiated to obtain an instantiated image.

5. The method according to claim 4, wherein instantiating the other road surface area to obtain an initial instantiated image comprises: Obtaining an instance segmentation result obtained by performing instance segmentation on the road image; The instance segmentation result includes at least two road instances and corresponding confidence levels; For a sub-region of each road surface region in the road surface image after hole filling, determining at least one candidate road instance in the at least two road instances that matches the shape and position of the sub-region; When the number of the at least one candidate road instance is at least two, selecting a target road instance whose confidence satisfies a confidence condition from the candidate road instances; The sub-region is filled based on the target road instance to obtain an initial instantiation image.

6. The method according to claim 4 or 5, wherein the hole filling region comprises a first hole filling region and a second hole filling region, and an area of ​​the first hole filling region is larger than an area of ​​the second hole filling region; The initial instantiated image is a first instantiated image; The instantiating the hole filling region in the initial instantiated image to obtain the instantiated image includes: Determining an instance category based on adjacent instances of the first hole filling area; In the first instantiated image, a new road instance is generated in the first hole filling area according to the instance category to obtain a second instantiated image; Determine the road instance to which the second hole filling area belongs; In the second instantiated image, the second hole filling region is implemented based on the attributed road instance. Instantiate to obtain an instantiated image.

7. The method according to any one of claims 1 to 6, wherein determining the common elements shared by the road instances based on the edges of the road instances in the instantiated image comprises: Performing edge detection on the instantiated image to obtain edges of road instances in the instantiated image; determining a road instance skeleton in the instantiated image based on the edge; Based on the connection relationship in the road instance skeleton, common elements shared by each of the road instances are determined.

8. The method according to any one of claims 1 to 7, wherein the common element comprises a key edge, and the semantic information of the common element comprises edge semantic information; and the semantic division of the common element according to each of the road instances to obtain the semantic information of the common element comprises: Determine a target road instance to which the key edge belongs; The key edges are semantically divided based on the semantic information of the target road instance to obtain edge semantic information.

9. The method according to claim 8, further comprising: Determining a processing method of the key edge based on the semantic information of the target road instance; Optimizing the key edge according to the processing method to obtain a processed key edge; The semantically dividing the key edge based on the semantic information of the target road instance to obtain edge semantic information includes: The processed key edges are semantically divided based on the semantic information of the target road instance to obtain edge semantic information.

10. The method according to any one of claims 1 to 9, wherein the step of performing image editing on each road instance in the instantiated image based on the semantic information of the common element to generate a road image comprises: Adding semantic labels to each road instance in the instantiated image based on the semantic information of the common element to obtain semantically optimized road instances; A road image is generated based on the semantically optimized road instance.

11. The method according to any one of claims 1 to 10, further comprising: Get the road elements to be added; Determine an element adding position of the road element to be added in the road image; According to the semantic information of the element to be added, the road element to be added is drawn at the element adding position to obtain an element-optimized road image.

12. A road image generating device, the device comprising: A road surface image generation module is used to perform instance segmentation on the road image, obtain instance segmentation results, and generate a road surface image based on the instance segmentation results; An instantiated image generation module is used to fill the hole areas in the road surface image and instantiate the road surface image after the hole filling to obtain an instantiated image; A common element determination module, configured to determine common elements shared by each of the road instances based on the edges of the road instances in the instantiated image; A semantic determination module, used for performing semantic division on the common elements according to each of the road instances to obtain semantic information of the common elements; and The image processing module is used to perform image editing on each road instance in the instantiated image based on the semantic information of the common element to generate a road image.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method according to any one of claims 1 to 11 when executing the computer program.

14. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 11.

15. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images