Label image generation and semantic segmentation model training method and device, and storage medium

Abstract

Provided are a label image generation and semantic segmentation model training method and device and a storage medium. The label image generation method comprises: performing multi-level semantic labeling on a region unit, thereby obtaining one or more labeled semantic categories of each region unit; obtaining an image region corresponding to each semantic category according to the one or more labeled semantic categories of each region unit; generating a single semantic category parsing image for each semantic category according to the image region corresponding to each semantic category; selecting, from the parsing image, a parsing image corresponding to the semantic category mapped by the label semantics of one or more label semantics corresponding to the label image; performing an operation on the pixel value of the selected parsing image to determine an image region corresponding to each label semantics; and assigning a value to the image region corresponding to each different label semantics respectively to generate a label image. The present application can greatly reduce the time for image relabeling and shorten the research and development cycle.

Description

Technical Field

[0001] The embodiments of the present invention generally relate to the field of image semantic annotation technology, and more specifically, to a method for generating labeled images and a method, apparatus and storage medium for training semantic segmentation models. Background Technology

[0002] Semantic segmentation refers to the process of dividing an image into semantically meaningful regions using specific algorithms, identifying the semantic category of each region, and enabling semantic reasoning across different image regions. The final result is a segmented image with pixel-by-pixel semantic annotations, thus achieving functions such as image classification, object detection, and image segmentation. Image semantic segmentation can be divided into traditional methods and methods based on convolutional neural networks (CNNs). In recent years, with the continuous development of deep learning technology, semantic segmentation technology has made great progress. CNNs can automatically learn image features and perform semantic segmentation, significantly improving the accuracy of semantic segmentation.

[0003] With the emergence of more and more advanced semantic segmentation algorithms, the research and application of semantic segmentation algorithms in high-level autonomous driving are constantly increasing.

[0004] However, firstly, the annotation cost of semantic segmentation datasets is very high. For semantic segmentation algorithms, training a high-accuracy neural network model requires a large number of original images and their labeled images. The process of generating labeled images mainly involves labeling each pixel in the original image with its semantic category and representing it with a specific numerical code to generate a labeled image. These categories could be pedestrians, vehicles, buildings, sky, lane lines, etc., defined by the user according to their needs. Furthermore, in order for the model to effectively recognize various scene categories, images from a large number of different scenes need to be extracted and labeled. In summary, the annotation complexity of a single image is high, and the need to annotate a large number of images results in significant time and labor costs.

[0005] Secondly, different semantic segmentation tasks require different label categories. In practice, we face various semantic segmentation training tasks, each requiring different labeled data. For example, we might train one model for semantic segmentation of dynamic targets such as pedestrians, motor vehicles, bicycles, and motorcycles, thus only needing to label these semantic categories; another model might segment one or more semantic categories such as drivable areas, lane lines, and traffic lights, also requiring only labeling of these categories. Therefore, the training tasks can be diverse. Furthermore, if the semantic category definitions cause ambiguity in the algorithm, resulting in poor model training results, we might need to redefine the data label categories. All these requirements necessitate changes in the dataset label category definitions, requiring relabeling and incurring significant costs.

[0006] Currently, publicly available datasets all employ single-level semantic annotation methods. For example, the CitySpace dataset defines eight groups: plane, people, vehicles, buildings, objects, nature, and sky (default). Each group is further subdivided into 30 semantic categories, with a one-to-one correspondence between image pixels and semantic categories. The annotation files can be directly used for semantic segmentation training. Furthermore, by mapping these semantic categories, the 30 categories can be mapped to new semantic categories, enabling their application to various training tasks to a certain extent.

[0007] Commonly known domain solutions all employ single-class annotation schemes, where each pixel is defined as a semantic category. However, in reality, a single pixel may correspond to different required semantic categories in different training tasks. Therefore, common-domain annotation schemes have certain limitations and cannot satisfy all training requirements with a single annotation. Summary of the Invention

[0008] To address the aforementioned problems in existing technologies, it is desirable to invent a flexible dataset annotation and mapping method that can generate labeled images of different semantic types from a single image data annotation, thereby achieving different training tasks or optimizing training models. To this end, embodiments of the present invention provide a labeled image generation method, the method comprising: performing semantic annotation at multiple levels on regions to be annotated in an original image; determining the semantic category of each region to be annotated in each of the multiple levels, thereby obtaining one or more labeled semantic categories for each region to be annotated; wherein each of the multiple levels includes multiple semantic categories, the multiple levels being selected from a group including one or more levels of ground regions, one or more levels of dynamic targets, one or more levels of static targets, and one or more levels of overall image attributes; obtaining an image region corresponding to each semantic category in each of the multiple levels based on the one or more labeled semantic categories of each region to be annotated in the original image; and based on... For each semantic category, a single semantic category parsing image is generated for each semantic category, where the image region corresponding to that semantic category in the single semantic category parsing image has a specific pixel value. For one or more label semantics corresponding to the label image to be generated, a single semantic category parsing image corresponding to the semantic category mapped to the one or more label semantics is selected from the single semantic category parsing images according to the mapping relationship between the label semantics and the labeled semantic categories. The pixel values ​​of the selected single semantic category parsing images are calculated to determine the image region corresponding to each of the one or more label semantics. Different pixel values ​​are assigned to the image regions corresponding to each different label semantic in the one or more label semantics to generate the label image.

[0009] In some implementations, the calculation of the pixel values ​​of the selected single semantic category parsing image includes: dividing the selected single semantic category parsing image into parsing images to be superimposed and parsing images to be subtracted according to the association between the label semantics and the labeled semantic categories; performing superposition operation on the regions with specific pixel values ​​in all the parsing images to be superimposed, and performing subtraction operation on the regions with specific pixel values ​​in all the parsing images to be subtracted.

[0010] In some implementations, the single semantic category parsing image is a binary image, and the image region corresponding to its semantic category has a specific pixel value of 1.

[0011] In some implementations, one or more layers of the ground area include one or more layers selected from the functional layers of the ground, the material layers of the ground, and the feature layers of the ground.

[0012] In some embodiments, one or more levels of the ground area include one or more levels selected from ground accessibility levels, ground marking levels, ground flatness levels, and ground height levels.

[0013] In some implementations, calculating the pixel values ​​of the selected single semantic category parsing image includes: calculating the pixel values ​​of the selected single semantic category parsing image according to a predefined category mapping relationship between the labeled semantics and the annotated semantic categories.

[0014] In some implementations, the method further includes storing a single semantic category parsed image for each semantic category and its semantic category as a labeling file.

[0015] In some implementations, the semantic meaning of one or more labels corresponding to the label image to be generated is determined based on a specific semantic segmentation training task.

[0016] In a second aspect, embodiments of the present invention propose a semantic segmentation model training method, the method comprising: using a labeled image generated according to any of the above embodiments as a training sample to input into a semantic segmentation model for training.

[0017] In a third aspect, embodiments of the present invention provide a labeled image generation apparatus, the apparatus comprising: a multi-level semantic annotation module configured to: perform multi-level semantic annotation on a region unit to be annotated in an original image, determine the semantic category of each region unit to be annotated in each of the multi-levels, thereby obtaining one or more annotated semantic categories for each region unit to be annotated, wherein each of the multi-levels includes multiple semantic categories, the multi-levels being selected from a group including one or more levels of ground regions, one or more levels of dynamic targets, one or more levels of static targets, and one or more levels of overall image attributes; a single semantic image region acquisition module configured to: obtain an image region corresponding to each semantic category in each of the multi-levels based on one or more annotated semantic categories of each region unit to be annotated in the original image; and a parsing image generation module configured to: The system is configured to: generate a single semantic category parsing image for each semantic category based on the image region corresponding to each semantic category, wherein the image region corresponding to the semantic category in the single semantic category parsing image has a specific pixel value; a parsing image selection module, configured to: select, based on the mapping relationship between the label semantics and the labeled semantic categories, a single semantic category parsing image corresponding to the one or more label semantics corresponding to the label image to be generated from the single semantic category parsing image; a parsing image calculation module, configured to calculate the pixel values ​​of the selected single semantic category parsing image to determine the image region corresponding to each of the one or more label semantics; and a region assignment module, configured to: assign different pixel values ​​to the image regions corresponding to each different label semantic in the one or more label semantics, thereby generating the label image.

[0018] In a fourth aspect, embodiments of the present invention provide a semantic segmentation model training apparatus, the apparatus being used to: input a label image generated according to the label image generation method described in any of the above embodiments as a training sample into a semantic segmentation model for training.

[0019] In a fifth aspect, embodiments of the present invention provide a storage medium storing machine-readable instructions that, when executed by a processor, perform the label image generation method or semantic segmentation model training method described in any of the above embodiments.

[0020] The label image generation method, semantic segmentation model training method, apparatus, and storage medium proposed in the embodiments of this invention can address users' needs in semantic segmentation training. Through multi-category semantic annotation, all semantic categories that users wish to train can be expressed and labeled. The annotation files obtained using the method of this invention can be mapped to obtain various label images required for actual training, thereby greatly reducing the time for image re-annotation, shortening the development cycle, and reducing development costs. Attached Figure Description

[0021] The above and other objects, features, and advantages of embodiments of the present invention will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of the invention are illustrated in the drawings by way of example and not limitation, wherein:

[0022] Figure 1 A flowchart of a label image generation method according to an embodiment of the present invention is shown;

[0023] Figure 2 An example image illustrating multi-level semantic annotation according to an embodiment of the present invention is shown;

[0024] Figure 3 A schematic diagram of the annotation file data structure according to an embodiment of the present invention is shown;

[0025] Figure 4 An example of a single semantic category parsing image is shown according to an embodiment of the present invention;

[0026] Figure 5 An example of a label image according to an embodiment of the present invention is shown;

[0027] Figure 6 A schematic block diagram of a label image generation apparatus according to an embodiment of the present invention is shown.

[0028] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0029] The principles and spirit of the invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement the invention, and are not intended to limit the scope of the invention in any way.

[0030] In one aspect, embodiments of the present invention provide a method for generating a label image. (See reference...) Figure 1 This illustrates a flowchart of a label image generation method 100 according to an embodiment of the present invention. Figure 1 As shown, method 100 includes steps S101-S106.

[0031] In step S101, semantic annotation of multiple levels is performed on the region units to be annotated in the original image to determine the semantic category of each region unit in each of the multiple levels, thereby obtaining one or more annotation semantic categories for each region unit. Each of the multiple levels includes multiple semantic categories, and the multiple levels are selected from a group including one or more levels of ground regions, one or more levels of dynamic targets, one or more levels of static targets, and one or more levels of overall image attributes.

[0032] Overall image attributes can include global parameters of the image. These parameters mainly describe the overall properties of a single image, and these attributes are important for filtering specific datasets. These parameters can mainly include: city, weather, lighting, etc., as shown in Table 1.

[0033] Table 1. Global Parameter Definition Table

[0034]

[0035] In the field of autonomous driving, ground information is diverse and has the most direct connection with vehicles; therefore, interpreting ground information is crucial for autonomous driving. As one embodiment of the present invention, one or more layers of the ground area may include one or more layers selected from the ground's functional layer, ground's material layer, and ground's feature layer. As another embodiment of the present invention, one or more layers of the ground area may include one or more layers selected from the ground's accessibility layer, ground's marking layer, ground's flatness layer, and ground's height layer.

[0036] To address the semantic information of images on the ground that requires attention, as a specific example, the ground region is divided into three groups (A, B, and C) based on three levels: function, material, and features. These groups complement each other; through logical operations between groups, diverse ground information can be generated, thus achieving a complete representation of the ground information. The area outside the ground can be divided into two groups: dynamic targets and static targets. A specific example of the definition and description is shown in Table 2 below.

[0037] Table 2. Definition of Multi-level Semantic Groups

[0038]

[0039] For each semantic group within a multi-category semantic grouping framework, the semantic category information required for autonomous driving semantic segmentation training will be comprehensively analyzed. The semantic categories needed during multi-class training will be filtered out, even though these categories may overlap at the image pixel level. However, while meeting requirements, the number of semantic categories can be minimized to reduce annotation complexity and cost. For each semantic category, a category number, category name, and description can be defined to guide subsequent image annotation work. An example definition method is as follows:

[0040] A. Ground: Function

[0041] The semantic categories in this group are used to divide ground areas according to function. The required categories are defined and described by the developers according to actual needs. A specific example of semantic category definition is shown in Table 3. The category numbers start from 1 for example:

[0042] Table 3. Ground-Functional Semantic Category Definitions

[0043] category Class Name describe 1 Driveable roads Including urban roads, highways and other regular driving roads 2 Driving area All areas where motor vehicles can drive, except for drivable roads. 3 Non-driving area All areas where motor vehicles are prohibited 4 Parking area Parking spaces are distinguished by certain features or materials. 5 sidewalk Sidewalks or bike lanes along the road 6 pedestrian crossing pedestrian crossing area

[0044] B. Ground: Material

[0045] The semantic categories in this group are used to divide ground areas according to material. The required categories are defined and described by the developers according to actual needs. A specific example of semantic category definition is shown in Table 4, with numbers starting from 21 for example:

[0046] Table 4. Ground - Material Semantic Category Definitions

[0047] category Class Name describe 21 asphalt asphalt surface 22 Paving Paved surfaces with repeating textures. 23 Natural surfaces Natural surfaces such as grasslands, fields, and lakes 24 gravel gravel ground 25 cement cement floor

[0048] C. Ground: Features

[0049] The semantic categories in this group are used to divide ground regions according to features. The required categories are defined and described by the developers according to actual needs. A specific example of semantic category definition is shown in Table 5, with the numbers starting from 31 for example:

[0050] Table 5. Ground-Feature Semantic Category Definitions

[0051]

[0052]

[0053] D. Dynamic Objectives

[0054] The semantic categories in this group are used to describe dynamic targets outside the ground. The required categories are defined and described by the developers according to actual needs, but all possible dynamic entities that may appear on the ground must be classified. A specific example of semantic category definition is shown in Table 6, with numbers starting from, for example, 51:

[0055] Table 6 Definition of Semantic Categories of Dynamic Targets

[0056] category Class Name describe 51 car Various types of cars 52 Passenger and freight vehicles All kinds of trucks, buses, and vans 53 rail vehicles Various types of rail vehicles 54 motorcycle motorcycle 55 pedestrian pedestrian 56 bike bike 57 animal Animals that may appear on the road 58 Special vehicles Various construction vehicles, agricultural vehicles, garbage trucks, etc. 59 other Other types of dynamic entities

[0057] E. Static Target

[0058] The semantic categories in this group are used to describe static targets outside the ground. The required categories are defined and described by the developers according to actual needs, but all possible static entities that may appear on the ground must be classified. A specific example of semantic category definition is shown in Table 7, with numbers starting from, for example, 71:

[0059] Table 7 Definition of Static Target Semantic Categories

[0060]

[0061]

[0062] Besides the semantic categories mentioned above, there are some regions that can confuse the training algorithm and should be ignored; their category can be defined as 0. For example, some distant and blurry objects cannot be accurately labeled.

[0063] Step S101 involves labeling each region unit (or pixel region) in the original image according to the semantic categories defined above. Each pixel may correspond to multiple semantic categories, as detailed below. Figure 2 Let's take a pedestrian crossing area as an example to illustrate. Figure 2 An example image is shown illustrating multi-level semantic annotation according to an embodiment of the present invention.

[0064] like Figure 2 As shown, assuming there are white stripes for pedestrian crossings on a conventional asphalt road, the entire road is divided into three color zones: I, II, and III. For each zone, the multi-category labeling method according to an embodiment of the present invention is as follows:

[0065] Area I (black) is marked as: 1 (driving road), 6 (pedestrian crossing), 21 (asphalt).

[0066] For Zone II (white), it is marked as: 1 (driving road), 6 (pedestrian crossing), 33 (marking - zebra crossing).

[0067] For area III (grey), it is marked as: 1 (driving road), 21 (asphalt).

[0068] To illustrate the multi-category semantic annotation method of this invention, the above description is based on the smallest region, with no overlap between regions. In practice, we can annotate large regions according to categories. As shown in the example above, we can uniformly label regions I, II, and III as 1 (driving road), and uniformly label regions I and II as 6 (pedestrian crossing), ensuring the completeness of the annotation.

[0069] Dynamic targets can also be labeled at multiple levels; for example, an ambulance can be labeled as both a truck and an ambulance.

[0070] By performing multi-analogous semantic annotations on the original images, we can obtain the final annotation file, whose data structure can typically be expressed using XML or JSON files. (Reference) Figure 3 This illustrates a schematic diagram of the annotation file data structure according to an embodiment of the present invention. Figure 3 As shown, the data structure includes global parameters and annotation information. Annotation information for a specific region can include: region name, region extent, and semantic category. The semantic category includes a category ID (identifier) ​​and a category name.

[0071] In the annotation scheme of this invention, the annotation file is expressed as follows: a single image region corresponds to one or more semantic categories.

[0072] In step S102, based on one or more semantic categories of each region unit to be labeled in the original image, the image region corresponding to each semantic category in each of the multiple levels is obtained. The goal of this step is to transform the annotation file into one where a single semantic category corresponds to multiple (>=0) image regions.

[0073] In step S103, a single semantic category parsing image is generated for each semantic category based on the image region corresponding to each semantic category, wherein the image region corresponding to the semantic category in the single semantic category parsing image has a specific pixel value.

[0074] As one embodiment of the present invention, a single semantic category parsing image can be a binary image, and the image region corresponding to its semantic category has a specific pixel value of 1, while the pixel values ​​of other regions are 0.

[0075] Assuming the image has a height of H and a width of W, and we define N semantic categories, we can obtain an N*W*H*1 dimensional data structure through certain logical operations. Each dimension of N represents the image region corresponding to a certain semantic category. That is, for each semantic category, it is described as a W*H*1 binary image, with the region containing the semantic category set to 1 and the region not containing the semantic category set to 0.

[0076] refer to Figure 4 This illustrates an example of parsing an image by a single semantic category according to an embodiment of the present invention. Taking the aforementioned pedestrian crossing as an example, it has four semantic categories, which can be parsed into a 4*W*H*1 data structure. For semantic category 33 (marking - zebra crossing), the parsed data structure is represented as follows: Figure 4 As shown, the values ​​in the white areas are all 1, and the values ​​in the black areas are all 0.

[0077] As one embodiment of the present invention, the method may further include: storing a single semantic category parsing image for each semantic category and its semantic category as a annotation file.

[0078] In step S104, for one or more label semantics corresponding to the label image to be generated, based on the mapping relationship between label semantics and labeled semantic categories, one or more semantic category parsing images corresponding to the semantic categories mapped by the label semantics are selected from the single semantic category parsing images.

[0079] As one embodiment of the present invention, the semantic meaning of one or more labels corresponding to the label image to be generated is determined according to a specific semantic segmentation training task.

[0080] In step S105, the pixel values ​​of the selected single semantic category parsing image are calculated to determine the image region corresponding to each label semantic in one or more label semantics.

[0081] As one embodiment of the present invention, the calculation of the pixel value of the selected single semantic category parsing image includes: dividing the selected single semantic category parsing image into parsing images to be superimposed and parsing images to be subtracted according to the association relationship between the label semantics and the labeled semantic categories; performing superposition operation on the regions with specific pixel values ​​in all the parsing images to be superimposed, and performing subtraction operation on the regions with specific pixel values ​​in all the parsing images to be subtracted.

[0082] As one embodiment of the present invention, the calculation of the pixel value of the selected single semantic category parsing image includes: calculating the pixel value of the selected single semantic category parsing image according to the predefined category mapping relationship between the label semantics and the labeled semantic categories.

[0083] In step S106, different pixel values ​​are assigned to the image regions corresponding to each different label semantic in one or more label semantics, thereby generating a label image.

[0084] In the actual algorithm training process, each pixel can only correspond to one semantic category. Therefore, it is necessary to define the mapping relationship between the annotation file and the label image used for training based on the specific training task, and generate the required label image through the corresponding parsing operation.

[0085] Because the annotation file uses a multi-level annotation method, there is overlap and intersection between different semantic categories. Therefore, a simple one-to-one mapping cannot complete the parsing of the annotation file. The following section explains steps S104-S106 with specific examples.

[0086] Assuming the semantic categories required for training are T, then for each semantic category C... j ∈T(C j The dataset contains k categories (k = m + n), corresponding to the categories in the m + n annotation files. Here, m represents the region to be overlaid, marked with 1 (a union of 1s, i.e., retaining the pixel values ​​of the binary image), and n represents the region to be removed, marked with -1 (in the binary image, a pixel value of 1 is subtracted by 1, resulting in a blank image after removal). Finally, by parsing the annotation files, the pixels marked with 1 in the semantic categories of the annotation files are overlaid, and the pixels marked with -1 in the semantic categories of the annotation files are removed. The resulting pixel portion represents the semantic category C. j The corresponding pixel area.

[0087] Taking the pedestrian crossing markings mentioned above as an example, if a semantic segmentation model needs to be trained to identify the drivable area and road markings in the image, although the road traffic signs also belong to the drivable area, they need to be removed from the drivable area. The mapping relationship is shown in Table 8 below:

[0088] Table 8. Category Mapping Table Based on Specific Tasks

[0089]

[0090] In the example shown in Table 8, k is 2, which is the category associated with the drivable area, m is 1, which represents the resolved image to be overlaid, and n is 1, which represents the resolved image to be subtracted.

[0091] In this way, an N*W*H*1 dimensional parsing file can be mapped into a W*H*1 dimensional data structure, where the value of each pixel is the semantic category used for training, forming a label image for semantic segmentation training.

[0092] Taking the mapping in Table 8 as an example, the resulting training label image is illustrated as follows. Figure 5 This illustrates an example of a labeled image according to an embodiment of the present invention. All pixel values ​​in region 1 are assigned a value of 1, and all pixel values ​​in region 2 are assigned a value of 2. These assignments are formed by identifying the corresponding image regions and labeling them with different numbers. Since the number of semantic categories required above is T, the maximum value can be T, which is 1—T, or 0—(T-1), representing a total of T categories.

[0093] The label image generation method proposed in the embodiments of the present invention involves a multi-category semantic segmentation dataset annotation and mapping scheme. The generated label images can be used as semantic segmentation datasets and as training samples for specific semantic segmentation model tasks.

[0094] The label image generation method proposed in the embodiments of the present invention involves a multi-category semantic annotation and mapping scheme. Its core idea is that ground pixels can be described by multiple semantic categories, meaning that one pixel can correspond to multiple semantic categories. Specifically, it includes the following technical elements:

[0095] 1) Multi-level semantic group definition, for example, the ground pixel part is divided into three major categories: function, material and feature, and the ground information is described through multiple levels, and other pixel parts are divided into two major categories: dynamic target and static target;

[0096] 2) Define semantic categories for multiple needs. Define all possible semantic categories under different groups. Categories may overlap or intersect with each other in image regions.

[0097] 3) Define a multi-category annotation scheme that uses a single pixel to correspond to one or more semantic categories to generate multi-level annotation data;

[0098] 4) Based on task-specific category mapping, category mapping rules for labeled files and labeled images can be defined in combination with specific training requirements, and labeled images for training can be generated.

[0099] The label image generation method proposed in the embodiments of this invention mainly describes a scheme for multi-category semantic annotation and mapping. Any changes to the group definition and semantic category definition within the scheme fall within the scope of this invention, for example:

[0100] 1) For multi-level semantic group definitions, the ground part can be added, deleted, reorganized or redefined in other ways, such as: ground-accessibility, ground-marking, ground-flatness, ground-height, etc.

[0101] 2) Regarding the semantic category definition for multiple needs, the above implementation only illustrates the principles of category definition and provides sufficient category definitions. The defined semantic categories can be added, deleted, or modified according to actual training needs. For example, for semantic category 74 (traffic lights), if different states of traffic lights need to be obtained during actual training, it can be decomposed into: traffic light - red, traffic light - yellow, traffic light - green, traffic light - red and yellow, and traffic light - other states. For group D dynamic targets, if it is necessary to identify emergency vehicles, such as ambulances, fire trucks, and police cars, a new category of emergency vehicles can be added, or even different emergency vehicles can be treated as separate categories.

[0102] The label image generation method proposed in the embodiments of the present invention adopts a multi-category semantic annotation and mapping scheme. Compared with the existing semantic annotation scheme, its substantial differences include: a. labeling multiple semantic categories with one pixel; b. defining a multi-category semantic group and category scheme; c. defining semantic categories based on all possible training tasks; d. obtaining label images for all training tasks through category mapping definition.

[0103] The label image generation method proposed in this invention can address users' needs in semantic segmentation training by expressing and labeling all semantic categories that users wish to train through multi-category semantic annotation. The annotation files obtained using this method can be mapped to various label images required for actual training, thereby significantly reducing image re-annotation time, shortening the development cycle, and lowering development costs.

[0104] The label image generation method proposed in the embodiments of the present invention can be applied to semantic segmentation model training in the field of autonomous driving. For other industries, the present invention can also be used to implement semantic segmentation annotation and mapping, so as to enhance the scalability of the labeled data, reduce the annotation cost, and accelerate the research and development progress.

[0105] In a second aspect, embodiments of the present invention propose a semantic segmentation model training method, which includes: using a label image generated according to the label image generation method described in any of the above embodiments as a training sample to input into the semantic segmentation model for training.

[0106] In a third aspect, embodiments of the present invention provide a label image generation apparatus. (See reference...) Figure 6 This illustrates a schematic block diagram of a label image generation apparatus according to an embodiment of the present invention. Figure 6 As shown, the label image generation device 600 may include modules 601-606.

[0107] The multi-level semantic annotation module 601 can be configured to: perform multi-level semantic annotation on the region units to be annotated in the original image, determine the semantic category of each region unit to be annotated in each of the multiple levels, thereby obtaining one or more annotation semantic categories for each region unit to be annotated, wherein each of the multiple levels includes multiple semantic categories, and the multiple levels are selected from a group including one or more levels of ground regions, one or more levels of dynamic targets, one or more levels of static targets, and one or more levels of overall image attributes.

[0108] The single semantic image region acquisition module 602 can be configured to: obtain the image region corresponding to each semantic category in each of multiple levels, based on one or more labeled semantic categories of each region unit to be labeled in the original image.

[0109] The parsed image generation module 603 can be configured to: generate a single semantic category parsed image for each semantic category based on the image region corresponding to each semantic category, wherein the image region corresponding to the semantic category in the single semantic category parsed image has a specific pixel value.

[0110] The parsing image selection module 604 can be configured to: select, based on the mapping relationship between the label semantics and the labeled semantic categories, one or more single semantic category parsing images corresponding to the semantic categories mapped by the label semantics from the single semantic category parsing images, for one or more label semantics corresponding to the label image to be generated.

[0111] The image parsing and processing module 605 can be configured to perform calculations on the pixel values ​​of the parsed image for a selected single semantic category to determine the image region corresponding to each of the one or more label semantics.

[0112] The region assignment module 606 can be configured to: assign different pixel values ​​to the image regions corresponding to each different label semantic in one or more label semantics, thereby generating a label image.

[0113] In a fourth aspect, embodiments of the present invention provide a semantic segmentation model training apparatus, which is used to: input a label image generated according to the label image generation method described in any of the above embodiments as a training sample into the semantic segmentation model for training.

[0114] In a fifth aspect, embodiments of the present invention provide a storage medium storing machine-readable instructions that, when executed by a processor, perform the label image generation method or semantic segmentation model training method described in any of the above embodiments.

[0115] The foregoing description of embodiments of the invention has been given for illustrative purposes and is not exhaustive, nor is it intended to limit the invention to the exact forms disclosed. Those skilled in the art will understand that various changes can be made without departing from the scope of the invention, and elements therein can be substituted with equivalents. Furthermore, many modifications can be made to adapt particular situations or materials to the teachings of the invention without departing from the basic scope of the invention. Therefore, the invention is not intended to be limited to the specific embodiments disclosed as the best mode contemplated for carrying out the invention; the invention will include all embodiments falling within the scope of the appended claims.

Claims

1. A method for generating label images, characterized in that, The method includes: For the region units to be labeled in the original image, semantic annotation at multiple levels is performed to determine the semantic category of each region unit in each of the multiple levels, thereby obtaining one or more labeled semantic categories for each region unit. Each of the multiple levels includes multiple semantic categories, and the multiple levels are selected from a group including one or more levels of ground regions, one or more levels of dynamic targets, one or more levels of static targets, and one or more levels of overall image attributes. Based on one or more semantic categories of each region unit to be labeled in the original image, obtain the image region corresponding to each semantic category in each of the multiple levels; Based on the image region corresponding to each semantic category, a single semantic category parsing image is generated for each semantic category, wherein the image region corresponding to the semantic category in the single semantic category parsing image has a specific pixel value; For one or more label semantics corresponding to the label image to be generated, based on the mapping relationship between label semantics and labeled semantic categories, select the single semantic category parsing image corresponding to the semantic category to which the one or more label semantics are mapped from the single semantic category parsing image; The pixel values ​​of the image are calculated for the selected single semantic category to determine the image region corresponding to each of the one or more label semantics; Different pixel values ​​are assigned to the image regions corresponding to each of the one or more different label semantics to generate the label image.

2. The method according to claim 1, characterized in that, The calculations performed on the pixel values ​​of the selected single semantic category parsed image include: Based on the relationship between the semantics of the labels and the semantic categories of the annotations, the selected single semantic category parsing images are divided into parsing images that need to be overlaid and parsing images that need to be subtracted. The system performs a superposition operation on regions with specific pixel values ​​in all the analytical images that need to be superimposed, and a subtraction operation on regions with specific pixel values ​​in all the analytical images that need to be subtracted.

3. The method according to claim 1, characterized in that, The single semantic category parsing image is a binary image, and the image region corresponding to its semantic category has a specific pixel value of 1.

4. The method according to claim 1, characterized in that, The ground area includes one or more layers selected from the functional layers, material layers, and feature layers of the ground.

5. The method according to claim 1, characterized in that, The ground area includes one or more levels selected from the ground accessibility level, ground marking level, ground flatness level, and ground height level.

6. The method according to claim 1, characterized in that, The calculations performed on the pixel values ​​of the selected single semantic category parsed image include: The pixel values ​​of the image are calculated based on the predefined category mapping relationship between the label semantics and the labeled semantic categories.

7. The method according to claim 1, characterized in that, The method further includes: The image parsed for each semantic category and its semantic category are stored as a labeled file.

8. The method according to claim 1, characterized in that, The semantic meaning of one or more labels corresponding to the label image to be generated is determined based on a specific semantic segmentation training task.

9. A semantic segmentation model training method, characterized in that, The method includes: The labeled images generated by the method according to any one of claims 1-8 are used as training samples to train the semantic segmentation model.

10. A label image generation apparatus, characterized in that, The device includes: A multi-level semantic annotation module is configured to: perform multi-level semantic annotation on a region unit to be annotated in the original image, determine the semantic category of each region unit to be annotated in each of the multiple levels, thereby obtaining one or more annotation semantic categories for each region unit to be annotated, wherein each of the multiple levels includes multiple semantic categories, and the multiple levels are selected from a group including one or more levels of ground regions, one or more levels of dynamic targets, one or more levels of static targets, and one or more levels of overall image attributes; A single semantic image region acquisition module is configured to: obtain the image region corresponding to each semantic category in each of the multiple levels based on one or more labeled semantic categories of each region unit to be labeled in the original image; The parsing image generation module is configured to: generate a single semantic category parsing image for each semantic category based on the image region corresponding to each semantic category, wherein the image region corresponding to the semantic category in the single semantic category parsing image has a specific pixel value; The image selection module is configured to: select, based on the mapping relationship between the label semantics and the labeled semantic categories, a single semantic category parsing image corresponding to the semantic category mapped by the one or more label semantics corresponding to the label image to be generated; The image parsing module is configured to perform calculations on the pixel values ​​of the parsed image for the selected single semantic category, and determine the image region corresponding to each of the one or more label semantics. The region assignment module is configured to: assign different pixel values ​​to the image regions corresponding to each different label semantic in the one or more label semantics, thereby generating the label image.

11. A semantic segmentation model training device, characterized in that, The apparatus is used to: input the labeled image generated by the method according to any one of claims 1-8 as a training sample into the semantic segmentation model for training.

12. A storage medium storing machine-readable instructions that, when executed by a processor, perform the method according to any one of claims 1-9.

Images