Image data processing method, apparatus, device, and storage medium

Abstract

Embodiments of the present disclosure provide an image data processing method, device and equipment and a computer readable storage medium. The method provided by the embodiments of the present disclosure takes a road unit as a unit, identifies road boundary attributes based on a single image from a plurality of image sequences associated with the road unit, extracts road boundary attribute features in a driving direction and a direction perpendicular to the driving direction of the road unit in each image, and aggregates the extracted road boundary attribute features in all images in the image sequences to obtain a road boundary attribute vector of the road unit. The embodiments of the present disclosure can be applied to the field of maps. The method provided by the embodiments of the present disclosure can quickly and automatically update road edge data in a global or specified area in a road unit, and provide fine-grained road edge information for map rendering, so that the presentation of the map is more realistic.

Description

Technical Field

[0001] This disclosure relates to the fields of mapping or transportation, and more specifically, to an image data processing method, apparatus, device, and storage medium. Background Technology

[0002] Maps are used to realistically depict the world, and the increase in the dimensions of map data storage and the increase in update channels have gradually enhanced map rendering capabilities, making the construction of map digital twins increasingly possible.

[0003] Road edge information is an important form of implicit traffic information. It represents the boundaries between lanes with different traffic capacities, such as the boundaries between motor vehicle lanes (distinguishing between forward and reverse traffic), bicycle lanes, and pedestrian lanes. Depending on the lane type, the components of the road edge may vary. These components typically include traffic sign markings (e.g., double yellow lines, solid white lines), facilities (e.g., guardrails, shrubs, grass, curbs), and natural boundaries (e.g., boundaries formed by changes in material). Extracting road edge information during map rendering helps improve map rendering capabilities.

[0004] Therefore, an image data processing method is needed to enable efficient and accurate extraction of road edge information based on images. Summary of the Invention

[0005] To this end, this disclosure utilizes a deep learning-based boundary instance segmentation network to identify the attribute features of road edges in different directions from images, and fuses road edge information based on the identified attribute features, thereby achieving efficient and fine-grained extraction of road edge information.

[0006] Embodiments of this disclosure provide an image data processing method, apparatus, device, and computer-readable storage medium.

[0007] Embodiments of this disclosure provide an image data processing method, comprising: acquiring a plurality of image sequences associated with a road unit; for each of the plurality of image sequences, identifying road boundary attributes in the image using a boundary instance segmentation network for each image in the image sequence, the road boundary attributes including boundary line attributes and boundary feature attributes, wherein the boundary line attributes are used to indicate boundary attribute change information along the driving direction of the road unit, and the boundary feature attributes are used to indicate the positional relationship of boundary features along the perpendicular driving direction of the road unit; and for the plurality of image sequences, performing multi-sequence attribute aggregation based on the road boundary attributes of each image therein to generate a road boundary attribute vector for the road unit.

[0008] Embodiments of this disclosure provide an image data processing apparatus, comprising: an image acquisition module configured to acquire a plurality of image sequences associated with a road unit; an attribute recognition module configured to, for each of the plurality of image sequences, identify road boundary attributes in the image using a boundary instance segmentation network for each image in the image sequence, the road boundary attributes including boundary line attributes and boundary feature attributes, wherein the boundary line attributes are used to indicate boundary attribute change information along the driving direction of the road unit, and the boundary feature attributes are used to indicate the positional relationship of boundary features along the perpendicular driving direction of the road unit; and an attribute aggregation module configured to, for the plurality of image sequences, perform multi-sequence attribute aggregation based on the road boundary attributes of each image therein to generate a road boundary attribute vector for the road unit.

[0009] Embodiments of this disclosure provide an image data processing apparatus, including: one or more processors; and one or more memories, wherein the one or more memories store a computer-executable program, and when the processor executes the computer-executable program, it performs the image data processing method as described above.

[0010] Embodiments of this disclosure provide a computer-readable storage medium having computer-executable instructions stored thereon, which, when executed by a processor, are used to implement the image data processing method described above.

[0011] Embodiments of this disclosure provide a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform an image data processing method according to an embodiment of this disclosure.

[0012] Compared to traditional data processing methods for map rendering, the method provided by the embodiments of this disclosure can quickly update road edge data in the entire area or a specified area, and provide fine-grained road edge information for map rendering, thereby improving map rendering capabilities.

[0013] The method provided in the embodiments of this disclosure operates on a road unit basis. It identifies road boundary attributes from multiple image sequences associated with a road unit, extracting road boundary attribute features along the driving direction and perpendicular to the driving direction from each image. These extracted road boundary attribute features are then aggregated across all images in the image sequences to obtain the road boundary attribute vector for that road unit. The method provided in the embodiments of this disclosure enables rapid and automatic updates of road edge data within a global or specified area, on a road unit basis, and provides fine-grained road edge information for map rendering. Through this method, fine-grained elements constituting road edges can be depicted in map rendering based on road edge information, thereby further restoring the real-world road scene, making the map presentation more realistic, and making map-based navigation functions more intuitive and easier to understand, thus improving the user experience. Attached Figure Description

[0014] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some exemplary embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0015] Figure 1 This is a schematic diagram illustrating a scenario of obtaining road edge information based on a sequence of acquired images according to an embodiment of the present disclosure;

[0016] Figure 2A This is a flowchart illustrating an image data processing method according to an embodiment of the present disclosure;

[0017] Figure 2B This is a schematic flowchart illustrating an image data processing method according to an embodiment of the present disclosure;

[0018] Figure 3A This is a schematic diagram illustrating an example structure of a boundary instance segmentation network according to an embodiment of the present disclosure;

[0019] Figure 3B This is a schematic diagram illustrating a first subnetwork of a feature extraction structure in a boundary instance segmentation network according to an embodiment of the present disclosure;

[0020] Figure 3C This is a schematic diagram illustrating a second sub-network of a feature extraction structure in a boundary instance segmentation network according to an embodiment of the present disclosure;

[0021] Figure 3D This is a schematic diagram illustrating a high-dimensional feature learning subnetwork in a boundary instance segmentation network according to an embodiment of the present disclosure;

[0022] Figure 3E This is a schematic diagram illustrating a boundary line prediction subnetwork in a boundary instance segmentation network according to an embodiment of the present disclosure;

[0023] Figure 3F This is a schematic diagram illustrating a boundary feature prediction subnetwork in a boundary instance segmentation network according to an embodiment of the present disclosure;

[0024] Figure 4 This is a schematic diagram illustrating the road boundary attribute identification results according to an embodiment of the present disclosure;

[0025] Figure 5A This is a flowchart illustrating a multi-sequence attribute aggregation process according to an embodiment of the present disclosure;

[0026] Figure 5B This is a schematic diagram illustrating a multi-sequence attribute aggregation process according to an embodiment of the present disclosure;

[0027] Figure 6 This is a schematic diagram illustrating the results of road edge information extraction according to an embodiment of the present disclosure;

[0028] Figure 7 This is a schematic diagram illustrating an image data processing apparatus according to an embodiment of the present disclosure;

[0029] Figure 8 A schematic diagram of an image data processing apparatus according to an embodiment of the present disclosure is shown;

[0030] Figure 9 A schematic diagram of the architecture of an exemplary computing device according to embodiments of the present disclosure is shown; and

[0031] Figure 10 A schematic diagram of a storage medium according to an embodiment of the present disclosure is shown. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.

[0033] In this specification and accompanying drawings, steps and elements that are substantially the same or similar are indicated by the same or similar reference numerals, and repeated descriptions of these steps and elements are omitted. Furthermore, in the description of this disclosure, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance or order.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0035] To facilitate the description of this disclosure, the following concepts related to this disclosure are introduced.

[0036] The image data processing method disclosed herein can be based on artificial intelligence (AI). Artificial intelligence utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to obtain optimal results—theories, methods, technologies, and application systems. In other words, artificial intelligence is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine capable of reacting in a manner similar to human intelligence. For example, an AI-based image data processing method can extract changes in road edge attributes and the positional relationships between various boundary elements from an image, similar to how humans visually identify the various components of a road edge and their positional relationships, thereby generating fine-grained road edge information for map rendering. By studying the design principles and implementation methods of various intelligent machines, artificial intelligence enables the image data processing method of this disclosure to automatically extract the road boundary attributes of a road unit in real time and accurately based on the acquired image sequence associated with that road unit.

[0037] Optionally, the boundary instance segmentation network used in the image data processing method of this disclosure can be based on image instance segmentation technology. Image instance segmentation is a further refinement of object detection, separating the foreground and background of objects to achieve pixel-level object separation. Therefore, image instance segmentation is an improvement on object detection and has applications in target detection, face detection, expression recognition, medical image processing and disease-assisted diagnosis, and shelf vacancy recognition in retail scenarios. In the image data processing method of this disclosure, a boundary instance segmentation network based on image instance segmentation technology can be used to obtain boundary line attribute prediction and boundary feature attribute prediction from the image.

[0038] The image data processing method disclosed herein can be applied to the field of mapping, and can be based on Intelligent Traffic Systems (ITS). Intelligent Traffic Systems, also known as Intelligent Transportation Systems, effectively integrate advanced technologies (information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operations research, artificial intelligence, etc.) into transportation, service control, and vehicle manufacturing, strengthening the connection between vehicles, roads, and users, thereby forming a comprehensive transportation system that ensures safety, improves efficiency, improves the environment, and saves energy. For example, the image data processing method disclosed herein can automatically extract rendering-level road edge information based on acquired road image sequences for real-time updating and rendering of road edges, facilitating its use in intelligent traffic systems.

[0039] In summary, the solutions provided by the embodiments of this disclosure involve technologies such as artificial intelligence and intelligent transportation systems. The embodiments of this disclosure will be further described below with reference to the accompanying drawings.

[0040] Figure 1 This is a schematic diagram illustrating a scenario of obtaining road edge information based on a sequence of acquired images according to an embodiment of the present disclosure.

[0041] like Figure 1 As shown, a user can send one or more image sequences to be processed (e.g., one or more vehicle-mounted image sequences acquired via an in-vehicle terminal) to a server via a user terminal. The server then filters out image sequences related to the area to be processed (e.g., the area to be mapped) based on the received image sequences, which are then used for subsequent road edge recognition and rendering of that area. Subsequently, the server can return the obtained road edge information (and / or road edge data directly used for rendering at the user terminal) to the user terminal via the network for display.

[0042] Optionally, the user terminal may specifically include smartphones, tablets, laptops, wearable devices, smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc., but is not limited to these. For example, the user terminal may also be an image or video acquisition device to directly send the acquired images or videos to the processor for image processing. Optionally, the network may be an Internet of Things (IoT) based on the Internet and / or telecommunications networks. It can be a wired network or a wireless network. For example, it may be a local area network (LAN), metropolitan area network (MAN), wide area network (WAN), cellular data communication network, or other electronic networks capable of information exchange. The user terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in this application. The server may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0043] As mentioned above, road edge information serves as an important form of implicit traffic information in map rendering. It can characterize the boundaries between lanes with different traffic capacities, and its geometric shape can depict road patterns and contain traffic guidance information. Therefore, extracting road edge information during map rendering helps improve map rendering capabilities, which is crucial for applications such as urban design, traffic planning, disaster detection, and geographic information updates.

[0044] However, existing data processing methods for map rendering typically focus on drawing multiple lanes, generating a road skeleton network based on the detected road distribution to guide applications such as urban planning, road traffic, and disaster detection. However, these data processing methods often handle the drawing of lane edges in a relatively simple way, failing to reproduce the feature morphology at lane boundaries and thus failing to restore the real and comprehensive road information of the real world.

[0045] Therefore, in order to achieve better map rendering effects, it is necessary to consider extracting fine-grained information of road edges during the map rendering process, so as to depict the fine-grained elements that constitute the road edges, restore the scene of real roads, and make the map more realistic.

[0046] In other words, at the server used in the image data processing method of this disclosure, fine-grained information extraction of road edges is required based on the acquired image sequence. In the fine-grained information extraction of road edges, the following potential problems need to be considered: the constituent elements of road edges are diverse in type and form, and these constituent elements often coexist. Specifically, depending on the lane type, the constituent elements of the road edge of a lane will change accordingly (for example, the road edges of lanes such as motor vehicle lanes, bicycle lanes, and pedestrian lanes typically have different constituent elements), and one or more of these elements may appear simultaneously at the road edge (for example, one side of the road edge of a motor vehicle lane may simultaneously have traffic signs and markings as well as facilities such as guardrails). That is, the composition of road edge elements may exhibit a coexistence of multiple elements, and there are specific positional relationships between these elements.

[0047] Furthermore, since different components of the road edge may share certain commonalities and characteristics, achieving refined identification and rendering of these components for the analysis of rendering-level road boundary attributes requires distinguishing the major categories, minor categories, and additional attributes of the road edge components. For example, when the components include double yellow lines, it is necessary to analyze the major category (e.g., traffic sign markings), minor category (e.g., the solid or dashed state of the markings), and additional attributes (e.g., color attributes) of the components. Similarly, when the components include urban guardrails, it is necessary to analyze the major category (e.g., facilities) and minor category (e.g., New Jersey guardrails). This can be achieved in the image data processing method disclosed herein by using multi-dimensional information (i.e., multi-dimensional boundary attributes) to characterize the road edge components. In addition, to achieve accurate reproduction of these components, an orderly analysis of them is also required; that is, the positional relationships of these components at the road edge need to be identified.

[0048] Based on this, the present disclosure provides an image data processing method that uses a deep learning-based boundary instance segmentation network to identify the attribute features of road edges in different directions from an image, and fuses road edge information based on the identified attribute features, thereby achieving efficient and fine-grained extraction of road edge information.

[0049] Compared to traditional data processing methods for map rendering, the method provided by the embodiments of this disclosure can quickly update road edge data in the entire area or a specified area, and provide fine-grained road edge information for map rendering, thereby improving map rendering capabilities.

[0050] The method provided in the embodiments of this disclosure operates on a road unit basis. It identifies road boundary attributes from multiple image sequences associated with a road unit, extracting road boundary attribute features along the driving direction and perpendicular to the driving direction from each image. These extracted road boundary attribute features are then aggregated across all images in the image sequences to obtain the road boundary attribute vector for that road unit. The method provided in the embodiments of this disclosure enables rapid and automatic updates of road edge data within a global or specified area, on a road unit basis, and provides fine-grained road edge information for map rendering. Through this method, fine-grained elements constituting road edges can be depicted in map rendering based on road edge information, thereby further restoring the real-world road scene, making the map presentation more realistic, and making map-based navigation functions more intuitive and easier to understand, thus improving the user experience.

[0051] Figure 2A This is a flowchart illustrating an image data processing method 200 according to an embodiment of the present disclosure. Figure 2B This is a schematic flowchart illustrating an image data processing method according to an embodiment of the present disclosure.

[0052] As described above, this disclosure achieves road edge information extraction from the image domain, wherein, as Figure 2B As shown, road edge information is extracted from the area to be mapped. This area can be the entire region or a specific region specified as needed; this disclosure does not impose any limitations on this. In the image data processing method of this disclosure, road edge information extraction can be performed within any region on a road unit basis, thereby achieving road edge rendering for that region. The road unit can be a predetermined road unit (e.g., a road unit with a predetermined length). This disclosure only describes road edge information extraction for road units as an example, but does not specifically limit the setting or acquisition method of the road unit.

[0053] Therefore, the image data processing method of this disclosure can take the geographical location range information of the area to be mapped as input, and output all road edge information within the area to be mapped, wherein the output of this road edge information can be in units of road cells. Optionally, as Figure 2B As shown, in the image data processing method disclosed herein, based on the input region to be mapped, the extraction of road edge information within the region to be mapped can be achieved through image sequence filtering associated with the region, road edge recognition, and multi-sequence attribute aggregation of the recognition information.

[0054] Next, we will refer to Figure 2B as well as Figures 3A-5B Come to Figure 2AEach step in the process will be described in detail.

[0055] In step 201, multiple image sequences associated with the road unit can be acquired.

[0056] like Figure 2B As shown, based on the geographic location range information of the area to be mapped, image sequence filtering can be performed by image filtering based on geographic location information (e.g., image filtering based on Global Positioning System (GPS)). For example, all road units in the area to be mapped and one or more associated image sequences can be determined from the road database using GPS range information of the area to be mapped.

[0057] Therefore, for each road unit in the area to be mapped, according to embodiments of this disclosure, obtaining multiple image sequences associated with the road unit may include: filtering multiple image sequences associated with the road unit from the road database based on the geographical location information of the road unit and the acquisition information of image sequences in the road database, wherein the acquisition information of the image sequences may include one or more of the acquisition geographical location information, acquisition time information, and acquisition device information of the image sequences. That is, the filtering of image sequences may be performed one by one among all road units in the area to be mapped, on a per-road-unit basis. For each road unit, multiple image sequences associated with it may be filtered from the road database based on the geographical location information of that road unit (e.g., GPS range information), wherein the association between the road unit and the image sequences may be determined based on the geographical location information of the road unit and the acquisition information of the image sequences.

[0058] Optionally, the aforementioned image sequences can be acquired via mobile devices, and therefore the acquisition information of these image sequences can include motion information of the mobile device. For example, for vehicle image sequences acquired via in-vehicle equipment, the acquisition information can include driving trajectory information, such as GPS information, acquisition time, and acquisition camera parameters.

[0059] According to embodiments of this disclosure, the filtering can be based on the data coverage and data freshness of the image sequences in the road database with respect to the road units.

[0060] Optionally, the image sequence filtering for each road unit can first determine the number of image sequences to be selected (e.g., based on a pre-set maximum number of image sequences N, where N can be reasonably set according to actual conditions (e.g., N can be 5)). Then, multiple image sequences are selected from the road database according to predetermined filtering conditions. For example, the predetermined filtering conditions can be requirements for the data coverage and data freshness of the image sequences in the road database relative to the road unit. Data coverage can be related to the GPS information of the image sequence and the road unit, reflecting the degree to which the image sequence data covers the target scene of the road unit (i.e., the availability of the image sequence for extracting road edge information of the current road unit). Data freshness can be related to the acquisition time of the image sequence, reflecting the reliability (real-time performance) of the image sequence for extracting road edge information of the current road unit.

[0061] Therefore, through the above image sequence filtering process, multiple valid image sequences can be selected for each road unit for subsequent road edge feature extraction processing.

[0062] In step 202, for each of the plurality of image sequences, road boundary attributes in the image can be identified by a boundary instance segmentation network for each image in the image sequence. The road boundary attributes may include boundary line attributes and boundary feature attributes, wherein the boundary line attributes are used to indicate boundary attribute change information along the driving direction of the road unit, and the boundary feature attributes are used to indicate the positional relationship of boundary features along the perpendicular driving direction of the road unit.

[0063] Optionally, in the image data processing method of this disclosure, the problem of parsing road edges can be decomposed into the problem of identifying boundary lines and boundary features. Identifying boundary lines allows for the extraction of road edge information along the driving direction of the road unit, facilitating boundary association between adjacent image frames. Identifying boundary features allows for the extraction of the order of boundary features perpendicular to the driving direction (i.e., along the direction perpendicular to the driving direction of the road unit). By identifying boundary lines and boundary features, the changes in boundary attributes along the driving direction of the road unit and the positional relationships of boundary features along the direction perpendicular to the driving direction of the road unit can be determined, thereby achieving fine-grained extraction of road edge information in a single image frame.

[0064] As described above, the identification of boundary lines and boundary features can be achieved based on image instance segmentation methods through a boundary instance segmentation network. According to embodiments of this disclosure, the boundary instance segmentation network can take a single image as input and output the boundary line attributes and boundary feature attributes in the single image. That is, Figure 2BThe road edge recognition part of the process is performed on a single image frame basis, and each image sequence in the multiple image sequences obtained from the above screening is recognized frame by frame.

[0065] Optionally, to achieve fine-grained attribute prediction of the boundary line, the boundary line attributes may include geometric attributes, material attributes, orientation attributes, and color attributes of the boundary line in the image. The material attributes may include multiple dimensions representing the various materials simultaneously included in the boundary line in the image. As described above, for each image frame of a road unit, multi-dimensional attribute information can be extracted along the driving direction of the road unit. For example, this includes the geometric attributes, material attributes, orientation attributes, and color attributes of the boundary line in the image. The geometric attributes of the boundary line in the image characterize the specific location of the boundary line in the image; the material information characterizes the material composition of the elements on the boundary line; the orientation attribute characterizes the orientation of the boundary line in the image (e.g., the boundary line is the left or right boundary line of the road unit along the driving direction); and the color attribute, as an additional attribute to the aforementioned attributes, is used to supplement the material attributes to further refine the boundary line attribute features.

[0066] Optionally, the material and color attributes of the boundary line can vary along the driving direction of the road unit, thereby reflecting the boundary attribute changes in the driving direction of the road unit. Specifically, the material attribute can be the category of the constituent elements on the boundary line (i.e., the aforementioned major and minor categories of constituent elements, such as all minor categories under the major category of traffic sign markings, facilities, and natural boundaries, such as double solid lines, double dashed lines, ..., four lines, New Jersey guardrails, municipal guardrails, ..., anti-glare panels, or intersection break virtual lines, etc.). Furthermore, to reflect the coexistence state of the constituent elements on the boundary line, a multi-dimensional representation can be provided for the boundary line's material attributes to characterize the combination of material attributes of two or more constituent elements coexisting on the boundary line. This multi-dimensional representation can be based on the distance relationship between these constituent elements and the image acquisition point. For example, the boundary line's material attributes can include three dimensions representing the near, medium, and far distances of multiple coexisting constituent elements from the image acquisition point, thereby enabling the prediction of coexisting state material attributes.

[0067] However, since the attribute recognition results of the above boundary lines are linear representations along the driving direction of the road unit, they cannot reflect the specific positional relationships of all constituent elements on the boundary lines along the perpendicular driving direction of the road unit. This positional relationship information is essential for road edge rendering. Therefore, in addition to boundary line attribute recognition along the driving direction of the road unit, it is also necessary to perform attribute recognition of the boundary elements based on their positional relationships within the road unit.

[0068] Optionally, to achieve fine-grained attribute prediction of boundary features, the boundary feature attributes may include the material attribute, orientation attribute, and color attribute of each boundary feature in the image. The material attribute of a boundary feature can be its category (e.g., including major and minor categories), as described above; the orientation attribute can be its location within the road unit (e.g., left or right side of the road unit); and the color attribute can also supplement its material attribute, further defining the boundary feature. By performing instance segmentation on each boundary feature in the image, the specific location and attributes of each boundary feature can be identified, thereby determining the positional relationship of each boundary feature within the road unit.

[0069] As mentioned above, the identification of the boundary line attributes and boundary feature attributes is achieved through a boundary instance segmentation network based on image instance segmentation technology. Therefore, next, we can use... Figures 3A-3F The network structure shown is presented as an example of the boundary instance segmentation network used in this disclosure. It should be understood that the design of this boundary instance segmentation network and its specific subnetworks is used in this disclosure only as an example and not as a limitation. Other network structures that can achieve the same boundary attribute recognition effect can also be used in the image data processing method of this disclosure.

[0070] According to embodiments of this disclosure, the boundary instance segmentation network may include a boundary line prediction subnetwork for predicting the geometry and attributes of boundary lines, a boundary feature prediction subnetwork for instance segmenting boundary features in the image and predicting the position and attributes of the boundary features, and a high-dimensional feature learning subnetwork for learning high-dimensional instance features of boundary lines. Figure 3A This is a schematic diagram illustrating an example structure of a boundary instance segmentation network according to an embodiment of the present disclosure. Figure 3B This is a schematic diagram illustrating a first subnetwork of a feature extraction structure in a boundary instance segmentation network according to an embodiment of the present disclosure. Figure 3C This is a schematic diagram illustrating a second subnetwork of a feature extraction structure in a boundary instance segmentation network according to an embodiment of the present disclosure. Figure 3D This is a schematic diagram illustrating a high-dimensional feature learning subnetwork in a boundary instance segmentation network according to an embodiment of the present disclosure. Figure 3E This is a schematic diagram illustrating a boundary line prediction subnetwork in a boundary instance segmentation network according to an embodiment of the present disclosure. Figure 3F This is a schematic diagram illustrating a boundary feature prediction subnetwork in a boundary instance segmentation network according to an embodiment of the present disclosure.

[0071] like Figure 3AAs shown, the boundary instance segmentation network of this disclosure may include two branches, one for identifying boundary line attributes and the other for identifying boundary feature attributes, after performing basic feature extraction on the input image (i.e., through a feature extraction sub-network). The feature extraction sub-network may be a neural network that extracts basic image features. Specifically, the feature extraction sub-network may be built using, for example, an HRNet (High-Resolution Net) network, and may be a shared network in the subsequent two branches. As an example, the feature extraction sub-network may employ a network such as... Figure 3B The ResNet101 network shown is Figure 3C The stacked structure of the FPN network shown in the diagram involves superimposing the feature outputs of the conv2_x, conv3_x, conv4_x, and conv5_x layers of the ResNet101 network (C2, C3, C4, and C5, respectively) onto the connections of the FPN network. This outputs the basic feature prediction results of the input image (S4, S8, and S1, respectively). 16 and S 32 ), where S4 represents the 4-dimensional image basic feature, S8 represents the 8-dimensional image basic feature, and S 16 Representing the fundamental features of a 16-dimensional image and S 32 Represents the basic features of a 32-dimensional image.

[0072] According to embodiments of this disclosure, for each of the plurality of image sequences, identifying road boundary attributes in the image through a boundary instance segmentation network for each image in the image sequence may include: determining the boundary lines in the image and their vertical attribute associations through the boundary line prediction subnetwork and the high-dimensional feature learning subnetwork to determine the boundary line attributes in the image; and determining the lateral positional relationships of the boundary elements in the image through the boundary element prediction subnetwork to determine the boundary element attributes in the image.

[0073] Optionally, in the boundary line attribute recognition branch, basic features can be extracted from the image to be processed to obtain basic image features. Then, based on these basic image features, a high-dimensional feature learning subnetwork and a boundary line prediction subnetwork are used to obtain the features of each pixel in the image to be processed, as well as the boundary line segmentation features and boundary line attribute features. For example, specifically, the basic image features can be used for multi-task processing, i.e., using the basic image features to perform pixel embedding representation to obtain the features of each pixel in the image to be processed. Simultaneously, the basic image features can be used for boundary line instance segmentation and classification recognition to obtain the boundary line segmentation features and boundary line attribute features corresponding to the image to be processed. That is, each pixel in the image to be processed can be classified and recognized to identify whether each pixel belongs to the boundary line, and simultaneously identify the attribute category of each pixel belonging to the boundary line. Next, optionally, the features of each boundary line pixel can be calculated based on the boundary line segmentation features and the features of each pixel. Then, by clustering the features of each boundary line pixel, the boundary line region in the image to be processed (that is, the location of the boundary line in the image) can be determined. Specifically, by using boundary line segmentation features and individual pixel features to calculate the features of each boundary line pixel, the obtained boundary line pixel features can be made more accurate. Furthermore, by using clustering to identify the boundary line regions in the image to be processed, the obtained boundary line regions can be made more accurate. Finally, attribute recognition can be performed based on the determined boundary line regions and boundary line attribute features to determine the boundary line attribute information corresponding to the boundary line regions.

[0074] Optionally, the high-dimensional feature learning subnetwork can employ, for example... Figure 3D The network structure shown describes a method that uses fundamental features of the input image received from the feature extraction subnetwork (i.e., S4, S8, S1). 16 and S 32 Pixel feature extraction is performed to obtain the features of each pixel in the image to be processed. Optionally, the boundary line prediction subnetwork can adopt a method such as... Figure 3E The Deeplab v3+ (a semantic segmentation network) network structure shown is used to predict the geometric and attribute features of boundary lines. Specifically, as... Figure 3E As shown, basic image features at different scales can be input into the boundary line prediction sub-network. For example, inputting S32 and S4 into the boundary line prediction sub-network yields pixel-level classification and recognition results, including individual pixel features, boundary line segmentation features, and boundary line attribute features. Each pixel feature in the image, i.e., the segmentation feature, has a dimension of 1, indicating whether it belongs to a road boundary line. The dimension of each attribute feature is its attribute category number plus one. This boundary line prediction sub-network enables multi-label classification of pixels in the image to determine boundary line attribute features.

[0075] Optionally, in the boundary feature attribute recognition branch, boundary feature attribute features in the image to be processed can be obtained based on the basic image features through a boundary feature prediction subnetwork. Specifically, boundary feature segmentation can be performed on the basic image features to obtain the segmentation results of each boundary feature in the image, which represent the corresponding boundary feature instances identified in the image. This boundary feature prediction subnetwork can employ methods such as... Figure 3F The transform decoder network structure shown is used to perform instance segmentation of boundary features in order to predict the location and attributes of each boundary feature in the image.

[0076] According to embodiments of this disclosure, the boundary instance segmentation network can be trained based on the cross-entropy loss function of the boundary line prediction subnetwork and the boundary feature prediction subnetwork, as well as the discriminative loss function of the high-dimensional feature learning subnetwork.

[0077] Optionally, during the training of the boundary instance segmentation network, the image training loss information can be used to back-update the boundary instance segmentation network using an algorithm such as stochastic gradient descent. For example, a discriminative loss function can be used to calculate the loss between the boundary line region and the boundary line instance label, while a cross-entropy loss function can be used to calculate the loss between the boundary line attribute information and the boundary line attribute label. The cross-entropy loss function can also be used to calculate the loss between the boundary feature segmentation result and the boundary feature segmentation label. Therefore, by combining all loss functions, the image training loss information can be determined.

[0078] Figure 4 This is a schematic diagram illustrating the road boundary attribute identification results according to an embodiment of the present disclosure.

[0079] Specifically, Figure 4 (a) shows the results of the boundary line attribute identification, while Figure 4 (b) shows the results of the identification of boundary feature attributes. For example... Figure 4 As shown in (a), the identification results of boundary line attributes include the specific location of the boundary line in the image (e.g., two straight lines on the left and right sides of the road in the image), attribute changes on the boundary line (e.g., line segments with different attributes on the boundary line represented by line types of different gray levels), and specific boundary line attributes shown in textual results. Optionally, multi-dimensional attribute prediction of road edge information can directly predict attribute combinations. For example, in Figure 4 The road unit shown in (a) includes a left boundary line and a right boundary line. The left boundary line includes two attribute combinations: "municipal guardrail + center" and "double solid yellow & municipal guardrail + center". The right boundary line has two attribute combinations: "curbstone + right side" and "virtual intersection break - marking type + right side".

[0080] like Figure 4 As shown in (b), the multi-dimensional attribute prediction of boundary features in road units can be performed using a binary-like encoding and decoding method for type mapping, so that the attribute combination of each boundary feature can correspond to a unique number. For example, "double solid line_yellow_center" corresponds to the boundary feature attribute combination with material attribute of double solid line, orientation attribute of center and color attribute of yellow.

[0081] It can be seen that the attribute recognition of boundary lines focuses on depicting the complete shape of the road edge, which can represent longitudinal attribute relationships and identify virtual lines at intersection breaks; while the attribute recognition of boundary elements focuses on depicting the positional relationships of boundary elements, which can represent the combined relationships of lateral elements (i.e., boundary elements along the perpendicular driving direction of the road unit). Through testing of the above recognition methods, the boundary instance segmentation network achieved a recall rate of 96.1% and an accuracy rate of 81.3% for boundary lines, and a recall rate of 95.3% and an accuracy rate of 86.2% for boundary elements.

[0082] Therefore, by performing boundary line and boundary feature recognition on the image as described above, each boundary line can be segmented from the image and its corresponding boundary line attributes can be identified, as well as each boundary feature can be segmented and its corresponding boundary feature attributes (including the position of these boundary features). This allows us to obtain information on the change of boundary attributes along the driving direction of the road unit in a single image frame, as well as information on the positional relationship of each boundary feature along the perpendicular driving direction of the road unit.

[0083] Next, the road boundary attributes of all images in all the above image sequences can be fused to generate the road boundary attributes of the complete road unit. In step 203, multi-sequence attribute aggregation can be performed on the multiple image sequences based on the road boundary attributes of each image to generate the road boundary attribute vector of the road unit.

[0084] As described above, the road edge information contained in a single image can be extracted using a boundary instance segmentation network. In the image data processing method disclosed in this paper, the following multi-sequence attribute aggregation algorithm is also proposed to integrate and calculate the boundary attribute recognition results of all image sequences associated with a road unit, thereby obtaining the rendering-level road edge attributes of the road unit.

[0085] According to embodiments of this disclosure, for the plurality of image sequences, performing multi-sequence attribute aggregation based on the road boundary attributes of each image to generate the road boundary attribute vector of the road unit may include: semantically aligning the plurality of image sequences based on the road boundary attributes in each image of the plurality of image sequences; and performing information fusion based on the road boundary attributes in each image of the semantically aligned plurality of image sequences to generate the road boundary attribute vector of the road unit.

[0086] Optionally, such as Figure 2B As shown, the multi-sequence attribute aggregation algorithm disclosed herein can include two parts: feature alignment and information fusion. By semantically aligning multiple image sequences associated with a road unit and fusing road edge information, the road edge information of that road unit is obtained, thereby generating rendering-level attributes of the road boundary. The generated rendering-level attributes of the road boundary can be, for example, obtained through... Figure 2B The process involves manual verification, comparison of rendered materials, and other operations to generate the correct rendering data package for image rendering on terminals such as user devices.

[0087] The following will refer to Figure 5A and Figure 5B This section provides a detailed description of the complete processing of multi-sequence attribute aggregation. Figure 5A This is a flowchart illustrating a multi-sequence attribute aggregation process according to an embodiment of the present disclosure. Figure 5B This is a schematic diagram illustrating a multi-sequence attribute aggregation process according to an embodiment of the present disclosure.

[0088] like Figure 5A As shown, the multi-sequence attribute aggregation processing of this disclosure may include steps 2031-2035, wherein steps 2031-2034 correspond to the feature alignment processing in the multi-sequence attribute aggregation processing, and the semantic alignment of the multiple image sequences based on the road boundary attributes in each image in the multiple image sequences may include steps 2031-2034, while step 2035 corresponds to the information fusion processing in the multi-sequence attribute aggregation processing, and the information fusion based on the road boundary attributes in each image in the semantically aligned multiple image sequences to generate the road boundary attribute vector of the road unit may include step 2035.

[0089] Generally, the boundary attributes within a road unit can exhibit a segmented pattern. That is, within a continuous road segment of a road unit, the road boundary can remain unchanged (i.e., exhibit the same attribute characteristics within that continuous road segment). Therefore, for multiple images associated with that continuous road segment in an image sequence associated with that road unit, each image will also exhibit the same road boundary attributes. As one moves from that continuous road segment to the next adjacent continuous road segment, the road boundary attributes change but remain unchanged within that next continuous road segment. Therefore, each image associated with that next continuous road segment will exhibit different road boundary attributes than the images associated with that continuous road segment, and these images will still possess the same road boundary attributes. Thus, the expression of road edge information in all images within each image sequence can be simplified to the expression of road edge information in the sequence segments of each image sequence. In other words, by segmenting the image sequence, the semantic-level alignment problem of multiple image sequences can be transformed into a semantic-level alignment problem of sequence segments; that is, the sequence segments between different image sequences can be aligned based on the correlation between sequence segments and road segments.

[0090] Specifically, in step 2031, the boundary attribute feature expression of the image can be obtained by sampling the road boundary attributes along the boundary line in each image based on the road boundary attributes in the image. The boundary attribute feature expression may include the multi-label attribute set of the boundary line and the multi-label attribute set of the boundary element.

[0091] Optionally, in order to segment the image sequence, images with the same road boundary attributes can be merged within the image sequence to form image segments, and the sequence segmentation can be completed based on all images in the image sequence by expanding and creating image segments.

[0092] Therefore, firstly, optionally, a segmentation pattern based on the boundary attributes of road units can compress data along the geometric dimension of numerous images in an image sequence. In other words, the road edge attributes in each image can be sampled to simplify the representation of road edges.

[0093] For example, based on the predicted values ​​of the orientation attributes of boundary lines and boundary features in an image, the road boundary attribute recognition results for the left and right sides of the road can be extracted, and reasonable sampling rules can be set to simplify the representation of road edges. Specifically, as an example, when the intrinsic and extrinsic parameters of the acquisition camera are determined, the distance to pixels in the image can be estimated using a monocular ranging method. By setting an appropriate distance threshold (i.e., the distance from the image acquisition point, which corresponds to a specific y-value in the image), the attribute recognition results of boundary line instances and boundary feature instances appearing at specific y-values ​​can be obtained along the image's x-axis. Furthermore, when the intrinsic and extrinsic parameters of the acquisition camera are not determined, the vanishing point can be calculated using the geometric attributes of the boundary lines. This can be achieved by estimating the y-values ​​at the nearest 1 / 3 of the left and right boundaries of the road. L and y R The value is used to obtain the attribute identification results of the adjacent boundary lines and boundary features.

[0094] Through the above sampling process, the boundary attribute feature expression of a single image can be obtained based on the attribute results of the boundary lines and boundary features within a selected range. This includes the multi-label attribute set of the boundary lines and the multi-label attribute set of the boundary features.

[0095] Alternatively, based on the representation of attribute combinations described above, the multi-tag attribute set of the boundary line can be represented as:

[0096] B = [t1, t2, ..., t] i (1)

[0097] Furthermore, the multi-label attribute set of boundary features can be represented as:

[0098] E = [t1, t2, ..., t] j (2)

[0099] In this context, any multi-label attribute t in sets B and E can represent the multi-label information of the road edge in the same form; for example, they can all be represented as a combination of "material-color-orientation" attributes. For instance, in... Figure 4 In the boundary attribute identification results shown, the values ​​of the multi-label attribute t can include "municipal guardrail - none - center" and "double solid line - yellow - center," etc. Note that the multi-label attribute set of the boundary line and the multi-label attribute set of the boundary features may overlap when represented according to the above attribute combination format, but they are not completely identical. Furthermore, the number of elements in sets B and E depends on the actual sampling situation, where i and j may not be equal.

[0100] Therefore, based on the above attribute sampling, the boundary attribute feature representation of a single image can be expressed as: F = B + E.

[0101] In step 2032, the similarity between adjacent images in each image sequence and the similarity between any two images from different image sequences can be estimated based on the boundary attribute feature representation of each image.

[0102] After determining the boundary attribute feature representation of each image, the above-mentioned image merging can be performed based on this information. Optionally, in the embodiments of this disclosure, the image merging can be performed based on the similarity between the boundary attribute feature representations of different images.

[0103] Optionally, cosine similarity can be used to estimate the feature similarity between different images. For example, F1 and F2 can be used to represent the boundary attribute feature representations of two images, respectively. By combining like terms in F1 and F2, common multi-label attributes in the boundary attribute feature representations of these two images can be determined.

[0104] T(F1)+T(F2)=[t1, t2,...,t k (3)

[0105] Then, the common multi-label attributes t1, t2, ..., t of F1 and F2 can be calculated separately. k The frequency of occurrence, i.e.:

[0106] Freq1 = (f 11 f 12 , ..., f 1k (4)

[0107] and

[0108] Freq2 = (f 21 f 22 , ..., f 2k (5)

[0109] Therefore, the similarity of boundary attribute features between these two images can be expressed as:

[0110]

[0111] The closer the similarity value is to 1, the higher the similarity of the boundary attribute features between the two images.

[0112] In step 2033, each image sequence can be segmented based on the estimation of similarity between adjacent images in each image sequence to form one or more sequence segments, wherein the images on each sequence segment have the same boundary attribute feature expression.

[0113] Optionally, by calculating similarity and making appropriate similarity determinations (e.g., similarity determination based on a predetermined similarity threshold, where two images are determined to be similar if the similarity of their boundary attribute features reaches the predetermined similarity threshold, and therefore can be aggregated), adjacent image frames within a single image sequence can be aggregated to form one or more sequence segments. For example, Figure 5B The image shows the sequence segmentation of three image sequences acquired for a road unit, where the boundary attribute features contained in the images on the line segments between adjacent nodes (shown as the origin) are consistent, that is, they have the same boundary attribute feature expression.

[0114] In step 2034, sequence segments with the same boundary attribute feature expression between different image sequences can be aligned based on the boundary attribute feature expression of one or more sequence segments in each image sequence, so as to align the multiple image sequences with respect to road boundary attributes.

[0115] Optionally, after forming sequence segments within each image sequence, sequence segment alignment can be performed based on the correlation between different sequence segments from different image sequences.

[0116] For example, establishing associations between sequence segments can involve pairwise comparisons of the boundary attribute feature representations of sequence segments across different image sequences, aligning sequence segments with the same boundary attribute feature representations, thereby achieving alignment of multiple image sequences in the dimension of road edge information based on the alignment of boundary attribute feature representations between sequence segments. For example, in Figure 5B In this context, for line segments in different image sequences, the same combination of start and end points is used to indicate mutually aligned sequence segments. And because the image sequences are acquired in different ways, these sequence segments may correspond to different road segments on a road unit, but these road segments should all correspond to the same continuous road segments.

[0117] In step 2035, the road boundary attributes of the aligned sequence segments between different image sequences can be merged based on the semantically aligned multiple image sequences to generate the road boundary attribute vector of the road unit.

[0118] Therefore, after aligning the various image sequences with respect to road edge information (i.e., semantic alignment), the road boundary attributes of the aligned sequence segments can be fused. In other words, multi-path road edge information with the same features can be merged to generate complete road edge information for road units.

[0119] According to embodiments of this disclosure, the road boundary attribute vector of the road unit may include a combination of one or more road boundary attributes of the road unit, wherein the one or more road boundary attributes of the road unit may respectively correspond to different scopes in the road unit determined based on the alignment result of the sequence segmentation, and the two road boundary attributes corresponding to two adjacent scopes in the one or more road boundary attributes of the road unit are different; wherein, the scope may include the start coordinates and end coordinates of the road boundary attribute corresponding to the scope appearing in the road unit.

[0120] Optionally, based on the segmentation pattern of the road unit's boundary attributes, the complete road edge information of the road unit can include a rendering-level road boundary attribute vector and its scope. The road boundary attribute vector can also include a combination of several road boundary attributes based on the segmentation pattern, with each road boundary attribute acting on its respective scope, such as... Figure 5B As shown, the road unit may include two road segments (i.e., road segments 1 and 2), and these two road segments have different road boundary attributes. Therefore, the road boundary attribute sequence of the road unit may include a combination of the two road boundary attributes.

[0121] Optionally, the scope of the road boundary attribute can be calculated based on the positions (e.g., including start and end coordinates) of multiple road segments corresponding to multiple aligned sequence segments associated with the road boundary attribute within the road unit. For example, the start and end coordinates of the road unit can be used as a reference frame to determine the start and end coordinates of the road segments corresponding to each aligned sequence segment within that reference frame (e.g., based on the distance between a specific image in the sequence segment and the start of the road unit in the reference frame), and suitable start and end points can be selected from these road segment start and end points to ensure the most accurate location coordinates where the road boundary attribute changes. For example, in Figure 5B In this system, a family of points of the same color can select a suitable point based on the distance value under this reference system, so that the location coordinates of the road segment changes are most accurate.

[0122] Optionally, for each road unit, the road edge rendering-level attributes output by the above image data processing method can be represented by the road boundary attribute vector Border. i This can be represented as follows:

[0123]

[0124] And the road boundary attribute vector Border i Scope Range i It can be represented as:

[0125]

[0126] Where i represents the sequence number of the road unit in the area to be mapped, and j represents the j-th road segment in the i-th road unit. The road boundary attributes of the j-th road segment can be used... This indicates that it records the attribute information of the left boundary of the road segment. and the attribute information of the right boundary The j-th road segment represents the extent of a road section, and its scope can be determined by... It means that, among them, Let J be the coordinates of the starting point of the j-th road segment. Let λ represent the coordinates of the endpoint of the j-th road segment, where λ represents longitude. This represents latitude. Therefore, when the image sequence is basically continuous, the above expression can be simplified so that the endpoint coordinate of the (j-1)th road segment is equal to the starting coordinate of the jth road segment.

[0127] The above method can obtain road edge information for the area to be mapped, and the topological structure of this information is unit road cells. In practical applications, similar materials can be selected for map rendering based on the semantic information of the road edges. Furthermore, in advanced applications, it is possible to select feature images from multiple image sequences, extract their boundary features based on predicted locations, and automatically perform similarity matching with a material library, thereby automating the selection of rendering materials.

[0128] As described above, the image data processing method disclosed herein, as a method for updating rendering-level road edge data, can achieve automated updating of rendering-level road edge attribute information across the entire domain or within a specific range, thus helping map rendering to develop towards restoring the real world.

[0129] Figure 6 This is a schematic diagram illustrating the results of road edge information extraction according to an embodiment of the present disclosure.

[0130] Considering that the image data processing method of this disclosure can be used to provide refined road edge recognition results for map rendering, the effect of fine-grained attribute prediction of road edges can be displayed to the user through a specific graphical user interface (GUI), such as... Figure 6 As shown.

[0131] exist Figure 6The image shows the road edge information recognition results for the currently selected road unit (e.g., determined by clicking on the road on the left by the indicated selection icon), which displays the boundary attribute information of the road unit, including the boundary feature names and their confidence levels for the left and right boundaries of the road unit.

[0132] This visualization displays comprehensive road edge information for the selected road unit, facilitating verification of the accuracy of the information provided for map rendering (e.g., for purposes such as...). Figure 2B (The above visualization method involves manual verification and comparison of rendered materials to generate the correct rendering data package). It should be understood that the above visualization method is not the only way to display the processing results of the image data processing method disclosed herein. Different visualization interfaces can be designed according to specific needs, and this disclosure is not limited to this.

[0133] Figure 7 This is a schematic diagram illustrating an image data processing apparatus 700 according to an embodiment of the present disclosure.

[0134] According to embodiments of the present disclosure, the image data processing apparatus 700 may include an image acquisition module 701, an attribute recognition module 702, and an attribute aggregation module 703.

[0135] The image acquisition module 701 can be configured to acquire multiple image sequences associated with road units. Optionally, the image acquisition module 701 can perform the operations described above with reference to step 201.

[0136] Optionally, the associated image sequences can be filtered one by one among all road units in the area to be mapped, on a road unit basis. For each road unit, multiple associated image sequences can be filtered from the road database based on the road unit's geographic location information (e.g., GPS range information). The association between the road unit and the image sequence can be determined based on the road unit's geographic location information and the image sequence's acquisition information. For example, the image sequences can be acquired via mobile devices, and therefore the acquisition information of these image sequences can include the mobile device's motion information. For example, for vehicle-mounted image sequences acquired via in-vehicle equipment, the acquisition information can include driving trajectory information, such as GPS information, acquisition time, and acquisition camera parameters.

[0137] Optionally, the image sequence selection for each road unit can begin by determining the number of image sequences to be selected, and then selecting multiple image sequences from the road database according to predetermined selection criteria. For example, the predetermined selection criteria could be requirements for the data coverage and data freshness of the image sequences in the road database relative to the road units. Data coverage could be related to the GPS information of the image sequences and the road units, reflecting the extent to which the image sequence data covers the target scene of the road unit (i.e., the availability of the image sequence for extracting road edge information for the current road unit). Data freshness could be related to the acquisition time of the image sequences, reflecting the reliability (real-time performance) of the image sequence for extracting road edge information for the current road unit.

[0138] Therefore, by filtering the image sequences, multiple valid image sequences can be selected for each road unit for subsequent road edge feature extraction processing.

[0139] The attribute recognition module 702 can be configured to, for each of the plurality of image sequences, identify road boundary attributes in the image using a boundary instance segmentation network for each image in the image sequence. The road boundary attributes include boundary line attributes and boundary feature attributes, wherein the boundary line attributes indicate boundary attribute changes along the driving direction of the road unit, and the boundary feature attributes indicate the positional relationships of boundary features along the perpendicular driving direction of the road unit. Optionally, the attribute recognition module 702 can perform the operations described above with reference to step 202.

[0140] Optionally, the problem of resolving road edges can be decomposed into the problem of identifying boundary lines and boundary features. Identifying boundary lines allows for the extraction of road edge information along the driving direction of the road unit, facilitating boundary association between adjacent image frames. Identifying boundary features allows for the extraction of the order of boundary features perpendicular to the driving direction (i.e., along the direction perpendicular to the driving direction of the road unit). By identifying boundary lines and boundary features, the changes in boundary attributes along the driving direction of the road unit and the positional relationships of boundary features perpendicular to the driving direction of the road unit can be determined, thereby achieving fine-grained extraction of road edge information in a single image frame.

[0141] Optionally, to achieve fine-grained attribute prediction of the boundary line, the boundary line attributes may include geometric attributes, material attributes, orientation attributes, and color attributes of the boundary line in the image. The material attributes may include multiple dimensions representing the various materials simultaneously included in the boundary line in the image. As described above, for each image frame of a road unit, multi-dimensional attribute information can be extracted along the driving direction of the road unit. For example, this includes the geometric attributes, material attributes, orientation attributes, and color attributes of the boundary line in the image. The geometric attributes of the boundary line in the image characterize the specific location of the boundary line in the image; the material information characterizes the material composition of the elements on the boundary line; the orientation attribute characterizes the orientation of the boundary line in the image (e.g., the boundary line is the left or right boundary line of the road unit along the driving direction); and the color attribute, as an additional attribute to the aforementioned attributes, is used to supplement the material attributes, further refining the boundary line attribute features.

[0142] However, since the attribute recognition results of the above boundary lines are linear representations along the driving direction of the road unit, they cannot reflect the specific positional relationships of all constituent elements on the boundary lines along the perpendicular driving direction of the road unit. This positional relationship information is essential for road edge rendering. Therefore, in addition to boundary line attribute recognition along the driving direction of the road unit, it is also necessary to perform attribute recognition of the boundary elements based on their positional relationships within the road unit.

[0143] Optionally, to achieve fine-grained attribute prediction of boundary features, the boundary feature attributes may include the material attribute, orientation attribute, and color attribute of each boundary feature in the image. The material attribute of a boundary feature can be its category (e.g., including major and minor categories), as described above; the orientation attribute can be its location within the road unit (e.g., left or right side of the road unit); and the color attribute can also supplement its material attribute, further defining the boundary feature. By performing instance segmentation on each boundary feature in the image, the specific location and attributes of each boundary feature can be identified, thereby determining the positional relationship of each boundary feature within the road unit.

[0144] As described above, the identification of boundary lines and boundary features can be achieved through a boundary instance segmentation network based on an image instance segmentation method. This network takes a single image as input and outputs the boundary line attributes and boundary feature attributes within that single image. By identifying boundary lines and boundary features in the image, each boundary line can be segmented and its corresponding boundary line attributes identified, as well as each boundary feature can be segmented and its corresponding boundary feature attributes identified (including the position of these boundary features). This yields information on the boundary attribute changes along the driving direction of the road unit and the positional relationships of each boundary feature along the perpendicular driving direction of the road unit in a single image frame.

[0145] The attribute aggregation module 703 can be configured to perform multi-sequence attribute aggregation based on the road boundary attributes of each of the plurality of image sequences to generate a road boundary attribute vector for the road unit. Optionally, the attribute aggregation module 703 can perform the operations described above with reference to step 203.

[0146] In the attribute aggregation module 703, road boundary attributes in all images of all the aforementioned image sequences can be fused to generate complete road boundary attributes for a road unit. Optionally, fusing road boundary attributes in all images of all the aforementioned image sequences may include two parts: feature alignment and information fusion. By semantically aligning multiple image sequences associated with a road unit and fusing road edge information, the road edge information of that road unit can be obtained.

[0147] Considering that the boundary attributes in a road unit can exhibit a segmented pattern, that is, within a continuous road segment in a road unit, the road boundary can remain unchanged, but as one moves from that continuous road segment to the next adjacent continuous road segment, the road boundary attributes will change and remain unchanged in that next continuous road segment. Therefore, the expression of road edge information in all images in each image sequence can be simplified to the expression of road edge information in the sequence segments of each image sequence. In other words, by segmenting the image sequence, the semantic-level alignment problem of multiple image sequences can be transformed into the semantic-level alignment problem of sequence segments. That is, based on the correlation between sequence segments and road segments, the sequence segments between different image sequences can be aligned.

[0148] Optionally, in order to segment the image sequence, images with the same road boundary attributes can be merged within the image sequence to form image segments, and the sequence segmentation can be completed based on all images in the image sequence by expanding and creating image segments.

[0149] Specifically, firstly, based on the segmentation pattern of road unit boundary attributes, data compression can be performed along the geometric dimension of numerous images in an image sequence. In other words, road edge attributes in each image can be sampled to simplify the representation of road edges. Through sampling processing, the boundary attribute feature representation of an image can be obtained within a selected range based on the attribute results of boundary lines and boundary features, including the multi-label attribute set of the boundary lines and the multi-label attribute set of the boundary features.

[0150] After determining the boundary attribute feature representation of each image, the aforementioned image merging can be performed based on this information. Optionally, in embodiments of this disclosure, the image merging can be performed based on the similarity between the boundary attribute feature representations of different images. For example, cosine similarity can be used to estimate the feature similarity between different images. Therefore, by utilizing similarity calculation and appropriate similarity determination (e.g., similarity determination based on a predetermined similarity threshold, where two images are determined to be similar if the similarity of their boundary attribute feature representations reaches the predetermined similarity threshold, and thus aggregation can be performed), adjacent image frames within a single image sequence can be aggregated to form one or more sequence segments.

[0151] After segmenting each image sequence into segments, alignment can be performed based on the correlation between different sequence segments from different image sequences. For example, establishing correlation between sequence segments can involve pairwise comparison of the boundary attribute feature expressions of sequence segments between different image sequences, and aligning sequence segments with the same boundary attribute feature expressions. This alignment of multiple image sequences in the dimension of road edge information can be achieved based on the alignment of boundary attribute feature expressions between sequence segments.

[0152] Therefore, after aligning the various image sequences with respect to road edge information (i.e., semantic alignment), the road boundary attributes of the aligned sequence segments can be fused. In other words, multi-path road edge information with the same features can be merged to generate complete road edge information for road units.

[0153] As described above, the image data processing apparatus of this disclosure uses image recognition technology to rationally model road boundary information within the entire or specified area, thereby extracting fine-grained information about road edges from road image sequences. This helps improve the realism of map rendering and makes user interaction functions such as navigation more intuitive and user-friendly.

[0154] According to another aspect of this disclosure, an image data processing apparatus is also provided. Figure 8 A schematic diagram of an image data processing apparatus 2000 according to an embodiment of the present disclosure is shown.

[0155] like Figure 8 As shown, the image data processing device 2000 may include one or more processors 2010 and one or more memories 2020. The memories 2020 store computer-readable code, which, when executed by the one or more processors 2010, can perform the image data processing method described above.

[0156] The processor in the embodiments of this disclosure can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor, and can be based on an x86 architecture or an ARM architecture.

[0157] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0158] For example, the method or apparatus according to embodiments of this disclosure can also be used by means of Figure 9 The architecture of the computing device 3000 shown is used for implementation. For example... Figure 9 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage devices in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication in the image data processing method provided in this disclosure, as well as program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 9 The architecture shown is merely exemplary and can be omitted as needed when implementing different devices. Figure 9 One or more components in the computing device shown.

[0159] According to another aspect of this disclosure, a computer-readable storage medium is also provided. Figure 10 A schematic diagram 4000 of a storage medium according to the present disclosure is shown.

[0160] like Figure 10 As shown, the computer storage medium 4020 stores computer-readable instructions 4010. When the computer-readable instructions 4010 are executed by a processor, the image data processing method according to embodiments of the present disclosure described with reference to the above figures can be performed. The computer-readable storage medium in the embodiments of the present disclosure may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). It should be noted that the memory used in the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0161] Embodiments of this disclosure also provide a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform an image data processing method according to embodiments of this disclosure.

[0162] Embodiments of this disclosure provide an image data processing method, apparatus, device, and computer-readable storage medium.

[0163] Compared to traditional data processing methods for map rendering, the method provided by the embodiments of this disclosure can quickly update road edge data in the entire area or a specified area, and provide fine-grained road edge information for map rendering, thereby improving map rendering capabilities.

[0164] The method provided in the embodiments of this disclosure operates on a road unit basis. It identifies road boundary attributes from multiple image sequences associated with a road unit, extracting road boundary attribute features along the driving direction and perpendicular to the driving direction from each image. These extracted road boundary attribute features are then aggregated across all images in the image sequences to obtain the road boundary attribute vector for that road unit. The method provided in the embodiments of this disclosure enables rapid and automatic updates of road edge data within a global or specified area, on a road unit basis, and provides fine-grained road edge information for map rendering. Through this method, fine-grained elements constituting road edges can be depicted in map rendering based on road edge information, thereby further restoring the real-world road scene, making the map presentation more realistic, and making map-based navigation functions more intuitive and easier to understand, thus improving the user experience.

[0165] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing at least one executable instruction for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0166] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0167] The exemplary embodiments of this disclosure described in detail above are merely illustrative and not restrictive. Those skilled in the art will understand that various modifications and combinations can be made to these embodiments or their features without departing from the principles and spirit of this disclosure, and such modifications should fall within the scope of this disclosure.

Claims

1. An image data processing method, comprising: Acquire multiple image sequences associated with road units; For each of the plurality of image sequences, for each image in the image sequence, a boundary instance segmentation network is used to identify road boundary attributes in the image. The road boundary attributes include boundary line attributes and boundary feature attributes. The boundary line attributes are used to indicate boundary attribute change information along the driving direction of the road unit, and the boundary feature attributes are used to indicate the positional relationship of boundary features along the perpendicular driving direction of the road unit. as well as For the plurality of image sequences, multi-sequence attribute aggregation is performed based on the road boundary attributes of each image to generate the road boundary attribute vector of the road unit.

2. The method as described in claim 1, wherein, The boundary instance segmentation network takes a single image as input and outputs the boundary line attributes and boundary feature attributes in the single image. The boundary line attributes include geometric attributes, material attributes, orientation attributes, and color attributes of the boundary line in the image, wherein the material attributes include multiple dimensions representing the various materials simultaneously included in the boundary line of the image; and The boundary element attributes include the material attribute, orientation attribute, and color attribute of each boundary element in the image.

3. The method as described in claim 2, wherein, The boundary instance segmentation network includes a boundary line prediction sub-network for predicting the geometry and attributes of the boundary line, a boundary feature prediction sub-network for instance segmenting boundary features in the image and predicting the position and attributes of the boundary features, and a high-dimensional feature learning sub-network for learning high-dimensional instance features of the boundary line. Specifically, for each of the plurality of image sequences, identifying road boundary attributes in the image using a boundary instance segmentation network for each image in the image sequence includes: The boundary line prediction subnetwork and the high-dimensional feature learning subnetwork determine the boundary lines in the image and their vertical attribute associations, thereby determining the boundary line attributes in the image; and The lateral positional relationship of boundary features in the image is determined by the boundary feature prediction subnetwork, thereby determining the boundary feature attributes in the image.

4. The method of claim 3, wherein, The boundary instance segmentation network is trained based on the cross-entropy loss function of the boundary line prediction subnetwork and the boundary feature prediction subnetwork, as well as the discriminative loss function of the high-dimensional feature learning subnetwork.

5. The method of claim 1, wherein, For the plurality of image sequences, multi-sequence attribute aggregation is performed based on the road boundary attributes of each image to generate the road boundary attribute vector of the road unit, including: Based on the road boundary attributes in each image of the plurality of image sequences, semantic alignment is performed on the plurality of image sequences; and Information fusion is performed based on the road boundary attributes in each of the semantically aligned multiple image sequences to generate the road boundary attribute vector of the road unit.

6. The method of claim 5, wherein, Semantic alignment of the multiple image sequences based on road boundary attributes in each image of the multiple image sequences includes: Based on the road boundary attributes in each image, the boundary attribute feature representation of the image is obtained by sampling the road boundary attributes along the boundary line. The boundary attribute feature representation includes the multi-label attribute set of the boundary line and the multi-label attribute set of the boundary element. Based on the boundary attribute feature representation of each image, the similarity between adjacent images in each image sequence and the similarity between any two images from different image sequences are estimated; Based on the estimation of similarity between neighboring images in each image sequence, each image sequence is segmented to form one or more sequence segments, where the images in each sequence segment have the same boundary attribute feature representation; Based on the boundary attribute feature representation of one or more sequence segments in each image sequence, sequence segments with the same boundary attribute feature representation between different image sequences are aligned to align the multiple image sequences with respect to road boundary attributes.

7. The method of claim 6, wherein, Information fusion based on road boundary attributes in each of the semantically aligned plurality of image sequences to generate the road boundary attribute vector of the road unit includes: Based on the semantically aligned multiple image sequences, the road boundary attributes of the aligned sequence segments between different image sequences are merged to generate the road boundary attribute vector of the road unit. The road boundary attribute vector of the road unit includes a combination of one or more road boundary attributes of the road unit. The one or more road boundary attributes of the road unit correspond to different scopes in the road unit determined based on the alignment result of the sequence segmentation. Furthermore, the two road boundary attributes of the one or more road boundary attributes of the road unit that correspond to two adjacent scopes are different. The scope includes the starting and ending coordinates of the road boundary attributes corresponding to the scope in the road unit.

8. The method of claim 1, wherein, Acquiring multiple image sequences associated with road units includes: Based on the geographical location information of the road unit and the acquisition information of image sequences in the road database, multiple image sequences associated with the road unit are filtered from the road database. The acquisition information of the image sequences includes one or more of the acquisition geographical location information, acquisition time information, and acquisition device information of the image sequences.

9. The method of claim 8, wherein, The filtering is based on the data coverage and data freshness of the image sequences in the road database with respect to the road units.

10. An image data processing apparatus, comprising: The image acquisition module is configured to acquire multiple image sequences associated with road units; The attribute recognition module is configured to, for each of the plurality of image sequences, identify road boundary attributes in the image through a boundary instance segmentation network for each image in the image sequence. The road boundary attributes include boundary line attributes and boundary feature attributes, wherein the boundary line attributes are used to indicate boundary attribute change information along the driving direction of the road unit, and the boundary feature attributes are used to indicate the positional relationship of boundary features along the perpendicular driving direction of the road unit. as well as The attribute aggregation module is configured to perform multi-sequence attribute aggregation based on the road boundary attributes of each of the plurality of image sequences to generate the road boundary attribute vector of the road unit.

11. The apparatus of claim 10, wherein, The boundary instance segmentation network takes a single image as input and outputs the boundary line attributes and boundary feature attributes in the single image. The boundary line attributes include geometric attributes, material attributes, orientation attributes, and color attributes of the boundary line in the image, wherein the material attributes include multiple dimensions representing the various materials simultaneously included in the boundary line of the image; and The boundary element attributes include the material attribute, orientation attribute, and color attribute of each boundary element in the image.

12. The apparatus of claim 10, wherein, For the plurality of image sequences, multi-sequence attribute aggregation is performed based on the road boundary attributes of each image to generate the road boundary attribute vector of the road unit, including: Based on the road boundary attributes in each image of the plurality of image sequences, semantic alignment is performed on the plurality of image sequences; and Information fusion is performed based on the road boundary attributes in each of the semantically aligned multiple image sequences to generate the road boundary attribute vector of the road unit.

13. An image data processing apparatus, comprising: One or more processors; as well as One or more memories storing a computer-executable program, which, when executed by the processor, performs the method of any one of claims 1-9.

14. A computer program product stored on a computer-readable storage medium and comprising computer instructions that, when executed by a processor, cause a computer device to perform the method of any one of claims 1-9.

15. A computer-readable storage medium having stored thereon computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-9.

Images