Model training method and device, equipment, storage medium and vehicle

Abstract

The invention relates to a model training method and device, equipment, a storage medium and a vehicle. The method comprises the following steps: acquiring a training image of a road image set; obtaining a full-artificial label image and a semi-artificial label image through the training image; a first label image is obtained through the semi-artificial label image, a category label of an image recognition difficulty area of the first label image is obtained through combination of manual labeling and automatic labeling of a preset model, and category labels of remaining areas are obtained through automatic labeling of the preset model; converting a diversion line label of the first label image and a diversion line label of the full-artificial label image into diversion area labels to obtain a second label image with updated category labels; and training the semantic segmentation initial model through the second tag image, the updated category tag and the preset weight based on the category tag to obtain the semantic segmentation model, and the embodiment of the invention improves the performance of the semantic segmentation model.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a model training method, device, equipment, storage medium and vehicle. Background Art

[0002] Image semantic segmentation is a computer vision task that is widely used in autonomous driving, remote sensing image processing, medical image diagnosis and other fields. Image semantic segmentation is to assign a semantic category to each pixel in the image and give it a category label. Each pixel has only one category label. Image semantic segmentation can be used to separate different objects in the image.

[0003] In the field of autonomous driving, the semantic segmentation model of images can be used to divide the drivable area, pedestrians, vehicles, traffic signs and other road elements into regions, providing support for vehicles to obtain environmental perception information. The semantic segmentation model needs to be trained before use, and during training, the pixel data in the collected images needs to be manually labeled with semantic categories.

[0004] However, manual labeling requires a lot of manpower and time, is costly, and the labeling quality is difficult to guarantee and prone to errors. In addition, in semantic segmentation tasks, the segmentation performance of certain scenes or categories (such as complex traffic scenes, small targets, occluded targets, etc.) is poor, resulting in poor performance of the semantic segmentation model. Summary of the invention

[0005] One of the objects of the present invention is to provide a model training method to solve the problem of poor performance of existing semantic segmentation models. The second object is to provide a model training device. The third object is to provide an electronic device. The fourth object is to provide a computer-readable storage medium. The fifth object is to provide a vehicle.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] A first aspect of an embodiment of the present invention provides a model training method, the method comprising:

[0008] Acquire a road image set, wherein the road image set includes a training image;

[0009] Obtaining fully artificially labeled images and semi-artificially labeled images through the training images, wherein the fully artificially labeled images refer to road images with artificially labeled category labels for the entire image area, and the semi-artificially labeled images refer to road images with artificially labeled category labels for pre-defined image recognition difficult areas;

[0010] Obtaining a first labeled image through the semi-artificial labeled image, wherein the category labels of the image recognition difficult areas of the first labeled image are obtained by combining manual labeling and automatic labeling of a preset model, and the category labels of the remaining areas are obtained by automatic labeling of a preset model;

[0011] Obtain guide line labels from the category labels of the first label image and the category labels of the fully manual label image respectively;

[0012] Converting the guide line labels of the first label image and the guide line labels of the fully manual label image into guide area labels to obtain a second label image with updated category labels;

[0013] The semantic segmentation initial model is trained using the second label image, the updated category label and the preset weight based on the category label to obtain a semantic segmentation model.

[0014] Optionally, the acquiring of a road image set includes:

[0015] Acquire road image videos containing different road scenes;

[0016] Randomly selecting a number of road images from the road image video to obtain a road image set;

[0017] The road image set is divided into training images and test images according to a preset ratio.

[0018] Optionally, obtaining a fully artificially labeled image and a semi-artificially labeled image through the training image includes:

[0019] Screening out a first road image belonging to a difficult-to-recognize scene from the training images;

[0020] Obtaining a predefined segmentation and annotation document, wherein the segmentation and annotation document defines different category labels and different annotation methods corresponding to the different category labels;

[0021] According to the second road image, a fully manually labeled image is obtained, wherein the fully manually labeled image is obtained by manually labeling the entire image area of ​​the second road image using the segmented and labeled document;

[0022] determining an image recognition difficult area of ​​the first road image;

[0023] According to the first road image, a semi-artificially labeled image is acquired, where the semi-artificially labeled image is obtained by manually labeling the image recognition difficult area of ​​the first road image using the segmented and labeled document.

[0024] Optionally, obtaining a first label image through the semi-artificial label image includes:

[0025] Through the semi-artificial label image, a pre-brush image and a difficult image are obtained, wherein the pre-brush image is an image obtained by automatically annotating the entire image area of ​​the semi-artificial label image through a preset model, and the difficult image is an image manually annotated with an image recognition difficult area in the semi-artificial label image;

[0026] A first label image is acquired through the pre-flash image and the difficult image.

[0027] Optionally, obtaining a first label image through the pre-flash image and the difficult image includes:

[0028] Filtering out category labels belonging to image recognition difficult areas from the pre-brush image;

[0029] Obtaining manually annotated category labels in the difficult image;

[0030] Get different priorities for different category labels in difficult images and pre-brush images;

[0031] Determine a target category label having a higher priority than the pre-flash image in the difficult image and a first region where the target category label is located;

[0032] Acquire a corresponding second area from the pre-flash image;

[0033] The category label of the second area is replaced with the target category label of the first area to obtain a first label image.

[0034] Optionally, converting the guide line labels of the first label image and the guide line labels of the fully manual label image into guide area labels to obtain a second label image with updated category labels includes:

[0035] According to the guide line label, converting the first label image and the fully artificial label image into binary images of the guide line and the background;

[0036] Determine the edge contour of the guide line from the binary image;

[0037] Connect the edge contours of adjacent flow guide lines with lines to obtain the initial flow guide area;

[0038] Filtering the initial diversion area in combination with the target scene to obtain a new diversion area;

[0039] The predefined segmentation annotation document is updated according to the new diversion area, wherein the diversion line labels in the updated segmentation annotation document are converted into diversion area labels;

[0040] The first label image and the fully manual label image are labeled and updated using the updated segmented label document to obtain a second label image with updated category labels.

[0041] Optionally, before the initial semantic segmentation model is trained using the second label image, the updated category label and the preset weight based on the category label to obtain the semantic segmentation model, the method further includes:

[0042] Pre-set multi-scale network;

[0043] Adding multiple scale output branches in the multi-scale network according to preset scales;

[0044] Add feature fusion branches between output branches of different scales;

[0045] After randomly initializing the parameters of the multi-scale network, an initial semantic segmentation model is obtained.

[0046] Optionally, the road image set also includes a test image;

[0047] After the semantic segmentation model is obtained by training the initial semantic segmentation model using the second label image, the updated category label and the preset weight based on the category label, the method further includes:

[0048] Set different color labels for different category labels in advance to obtain category-color pairs;

[0049] Inputting the test image into the semantic segmentation model to obtain a semantic segmentation image of the test image and a category label of the semantic segmentation image;

[0050] Determining a target color label corresponding to the category label of the semantic segmentation image according to the category color pair;

[0051] According to the target color label, obtaining a color label image of the semantic segmentation image;

[0052] The test image and the color label image are fused to obtain a visualization image of the semantic segmentation image.

[0053] A second aspect of an embodiment of the present invention provides a model training device, the device comprising:

[0054] A first acquisition module, used to acquire a road image set, wherein the road image set includes a training image;

[0055] A second acquisition module is used to acquire a fully artificially labeled image and a semi-artificially labeled image through the training image, wherein the fully artificially labeled image refers to a road image with category labels manually annotated on the entire image area, and the semi-artificially labeled image refers to a road image with category labels manually annotated on a pre-defined image recognition difficult area;

[0056] A third acquisition module is used to acquire a first labeled image through the semi-artificial labeled image, wherein the category label of the image recognition difficult area of ​​the first labeled image is obtained by combining the manual labeling and the automatic labeling of the preset model, and the category label of the remaining area is obtained by the automatic labeling of the preset model;

[0057] A fourth acquisition module, used to respectively acquire flow line labels from the category labels of the first label image and the category labels of the fully manual label image;

[0058] A conversion module, used to convert the guide line labels of the first label image and the guide line labels of the fully manual label image into guide area labels, so as to obtain a second label image with updated category labels;

[0059] The training module is used to train the initial semantic segmentation model through the second label image, the updated category label and the preset weight based on the category label to obtain the semantic segmentation model.

[0060] A third aspect of an embodiment of the present invention provides an electronic device, including:

[0061] processor;

[0062] a memory for storing instructions executable by the processor;

[0063] Wherein, the processor is configured to execute the instructions to implement the model training method as described in the first aspect.

[0064] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, which, when instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to execute a model training method.

[0065] According to a fifth aspect of an embodiment of the present application, a vehicle is provided, comprising the model training device described in the second aspect of the present application.

[0066] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0067] The present invention obtains a road image set, which includes training images. Through the training images, a full-artificial label image and a semi-artificial label image are obtained, wherein the full-artificial label image refers to a road image with category labels manually annotated for the entire image area, and the semi-artificial label image refers to a road image with category labels manually annotated for a predefined image recognition difficult area. Through the semi-artificial label image, a first label image is obtained, and the category label of the image recognition difficult area of ​​the first label image is obtained by combining manual labeling and automatic labeling of a preset model, and the category label of the remaining area is obtained by automatic labeling of a preset model; not only the cost of manual labeling is reduced, but also because only part of the area is annotated, errors are reduced, and the recognition effect of the image recognition difficult area is improved; guide line labels are respectively obtained from the category labels of the first label image and the category labels of the full-artificial label image; the guide line labels of the first label image and the guide line labels of the full-artificial label image are both converted into guide area labels to obtain a second label image with updated category labels, thereby reducing errors in segmentation results and improving segmentation accuracy; a semantic segmentation initial model is trained through the second label image, the updated category labels and the preset weights based on the category labels to obtain a semantic segmentation model. The embodiment of the present invention reduces the cost of manual labeling and reduces errors by using training images of semi-artificially labeled images, thereby ensuring the quality of labeling. By converting the guide line labels of the training images into guide area labels, the errors in the segmentation results are reduced, the segmentation accuracy is improved, and ultimately the performance of the obtained semantic segmentation model is improved.

[0068] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0070] Figure 1 is a flowchart of a model training method according to an exemplary embodiment;

[0071] Figure 2 yes Figure 1 The flowchart of step 102 in a model training method according to an exemplary embodiment is shown;

[0072] Figure 3 yes Figure 1 The flowchart of step 103 in a model training method according to an exemplary embodiment is shown;

[0073] Figure 4 yes Figure 1The flowchart of step 105 in a model training method according to an exemplary embodiment is shown;

[0074] Figure 5 is a block diagram of a model training device according to an exemplary embodiment;

[0075] Figure 6 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0076] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0077] The first embodiment of the present application relates to a model training method. Figure 1 is a flow chart of a model training method according to an exemplary embodiment. Figure 1 As shown, the following steps are included:

[0078] Step 101: Acquire a road image set, where the road image set includes training images.

[0079] The embodiment of the present invention is to train the semantic segmentation model. In the field of intelligent driving, the semantic segmentation of road images is mainly performed. Therefore, road images of different scenes will be collected during training. In order to achieve this purpose, first, video data can be obtained from sources such as public road image databases, traffic monitoring cameras, and autonomous driving vehicle recorders to ensure that the video data contains a variety of different factors, such as urban roads, highways, rural roads, tunnels, bridges, and other scenes, including different weather conditions such as sunny days, rainy days, snowy days, and foggy days, covering different time periods such as daytime, night, and dusk, and then the long video is divided into short clips for subsequent processing and analysis, and the video quality is checked at the same time, and blurred, jittered or damaged video clips are removed. Then, according to the frame rate of the video, a frame of image is randomly selected every certain number of frames (for example, one frame per second) to ensure that the selected image set covers different road scenes, weather conditions and time periods, and at the same time ensure that the number of images in the selected image set is large enough. Then, the road image set is divided into training images and test images, for example, 80% of the images are used for training and 20% of the images are used for testing. Of course, the ratio can also be adjusted according to specific needs, and the present invention is not specifically limited here. When dividing, ensure that the division process is random to avoid selection bias. The specific steps include:

[0080] Acquire road image videos containing different road scenes;

[0081] Randomly select a number of road images from the road image video to obtain a road image set;

[0082] The road image set is divided into training images and test images according to a preset ratio.

[0083] The above process can provide high-quality data support for the training and evaluation of semantic segmentation models. This process ensures the diversity and randomness of the data, which helps to improve the generalization ability of the model and the reliability of the evaluation results.

[0084] Step 102, obtaining fully artificially labeled images and semi-artificially labeled images through training images, wherein the fully artificially labeled images refer to road images with category labels manually annotated on the entire image area, and the semi-artificially labeled images refer to road images with category labels manually annotated on pre-defined image recognition difficult areas.

[0085] In order to reduce the cost of manual labeling in the embodiment of the present invention, images of scenes with difficulty in recognition are selected for partial area labeling. These images may contain complex backgrounds, such as trees, buildings, pedestrians, etc., may be taken at night or under low light conditions, may be taken under adverse weather conditions such as rain, snow, fog, etc., and may also contain dynamic elements, such as moving vehicles, pedestrians, etc. Using the pre-defined features of scenes with difficulty in recognition, images that meet the conditions are automatically screened out through image processing algorithms, and the results of automatic screening are manually reviewed to ensure that the screened images do belong to scenes with difficulty in recognition. Then a segmentation labeling document is pre-defined, in which different category labels and corresponding labeling methods are defined. For example, the defined category labels include but are not limited to roads, vehicles, pedestrians, traffic signs and backgrounds, and the defined labeling methods include but are not limited to polygonal labeling (applicable to objects with irregular shapes, such as pedestrians and traffic signs), rectangular labeling (applicable to objects with regular shapes, such as vehicles), and pixel-level labeling (applicable to areas that require fine labeling, such as roads and backgrounds). When labeling, select images that need to be manually labeled in all areas, and obtain fully manually labeled images after labeling, and images that need to be manually labeled in some areas, and obtain semi-manually labeled images after labeling.

[0086] Further, step 102, such as Figure 2 As shown:

[0087] Step 1021 , screening out a first road image belonging to a difficult-to-recognize scene and a second road image belonging to a normal scene from the training images.

[0088] Step 1022: Obtain a predefined segmentation and annotation document, in which different category labels and different annotation methods corresponding to the different category labels are defined.

[0089] Step 1023: acquiring a fully manually labeled image based on the second road image. The fully manually labeled image is obtained by manually labeling the entire image area of ​​the second road image using the segmented labeling document.

[0090] Step 1024: determine the image recognition difficult area of ​​the first road image.

[0091] Step 1025 , obtaining a semi-artificially labeled image based on the first road image, wherein the semi-artificially labeled image is obtained by manually labeling the image recognition difficult area of ​​the first road image using the segmented labeling document.

[0092] Through the above steps, the first road image with difficult recognition scenes can be effectively screened out, and the corresponding fully artificially labeled images and semi-artificially labeled images can be obtained. These labeled data will provide high-quality annotation information for the training of the semantic segmentation model, which will help improve the robustness and generalization ability of the model.

[0093] Step 103, obtaining a first labeled image through a semi-manual labeled image, wherein the category labels of the image recognition difficult areas of the first labeled image are obtained by combining manual labeling and automatic labeling of a preset model, and the category labels of the remaining areas are obtained by automatic labeling of a preset model.

[0094] After obtaining the semi-artificial label image in the embodiment of the present invention, since only the image recognition difficulty area is manually labeled in the semi-artificial label image, and the rest is not labeled, the unlabeled area needs to be processed. To this end, the embodiment of the present invention uses a pre-trained semantic segmentation model (such as DeepLab, U-Net, etc.) to automatically label the entire area of ​​the semi-artificial label image, and the model predicts the category of each pixel in the image to generate a pre-brush image. It should be noted that the pre-brush image is a newly generated image and will not modify the original content of the semi-artificial label image. The category label of the image recognition difficulty area is extracted from the original semi-artificial label image, the recognition difficulty area is extracted from the pre-brush image, and the category label in the recognition difficulty area is screened out, and then according to the accuracy and reliability of the annotation, the priority of different category labels in the difficult image and the pre-brush image is defined, and the category labels are prioritized to ensure that the high-priority category labels are given priority in the merging process, and the category label priorities of the same area in the difficult image and the pre-brush image are compared, and the target category label and the first area with a priority higher than the pre-brush image are determined, and the second area corresponding to the first area is found from the pre-brush image, and the category label of the second area is replaced with the target category label of the first area.

[0095] Further, step 103, such as Figure 3 As shown:

[0096] Step 1031, obtaining a pre-flash image and a difficult image through the semi-artificial labeled image. The pre-flash image is an image obtained by automatically annotating the entire image area of ​​the semi-artificial labeled image through a preset model, and the difficult image is an image manually annotated with the image recognition difficult area in the semi-artificial labeled image.

[0097] Step 1032: Obtain a first label image by pre-sweeping the image and the difficult image.

[0098] In the embodiment of the present invention, the category label of the first label image is obtained by merging the category label of the pre-flash image and the category label of the difficult image. Specifically, obtaining the first label image through the pre-flash image and the difficult image includes:

[0099] Filter out category labels belonging to the difficult areas of image recognition from the pre-brushed images;

[0100] Obtain manually annotated category labels in difficult images;

[0101] Get different priorities for different category labels in difficult images and pre-brush images;

[0102] Determine in the difficult image a target category label having a higher priority than the pre-flash image and a first region where the target category label is located;

[0103] Acquire a corresponding second area from the pre-flash image;

[0104] The category label of the second region is replaced with the target category label of the first region to obtain a first label image.

[0105] For example, the priority setting rule is: the lane line labels, pedestrian labels, and traffic sign labels in the difficult image have higher priority than the pre-brushed image, and the road edge labels in the pre-brushed image have higher priority than the difficult image. Therefore, when it is determined that the manually annotated category labels in the difficult image include lane line labels and road edge labels, the labels of the corresponding areas in the pre-brushed image are replaced with lane line labels, but the road edge labels remain unchanged. That is, the first label image is the modified pre-brushed image obtained by covering some category labels in the difficult image with the pre-brushed image, which is the first label image. Alternatively, the unlabeled part of the difficult image can also be covered with the annotations of the pre-brushed image, and the annotated part is partially replaced according to the priority judgment. At this time, the replaced difficult image can also be called the first label image.

[0106] Step 104 , respectively obtain flow line labels from the category labels of the first label image and the category labels of the fully manual label image.

[0107] In the embodiment of the present invention, the guide line label will also be transformed, so the guide line label needs to be found first. Because not every road image has a guide line label, the road images with guide lines will be screened out first, and then the guide line label will be found according to the category label marked thereon.

[0108] Step 105: Convert the guide line labels of the first label image and the guide line labels of the fully manual label image into guide area labels to obtain a second label image with updated category labels.

[0109] In an embodiment of the present invention, when performing guide line label conversion, in order to better process the guide lines, it is necessary to convert the first label image and the fully artificial label image (at this time, the first label image and the fully artificial label image are both pixel true value images) into binary images, in which the guide lines and the background are represented by different values ​​respectively. For example, the pixel values ​​marked as guide lines in the first label image and the fully artificial label image are set to 1, and the pixel values ​​marked as background in the first label image and the fully artificial label image are set to 0, and a binary image about the guide line and the background is generated to ensure that the guide line and the background are clearly distinguished. In the binary image, the edge contour of the guide line is key information and needs to be accurately extracted, so an edge detection algorithm (such as Canny edge detection) is used to extract the edge contour of the guide line from the binary image. The edge contour of the guide line is extracted to ensure the accuracy and completeness of the contour. Each edge contour can be represented by a 4-point box, and the adjacent guide line edge contours are identified. The adjacent guide line edge contours are connected with lines to form an initial guide area. In order to ensure the accuracy of the guide area, the initial guide area needs to be filtered in combination with the target scene, wherein the target scene includes but is not limited to roads, lane line areas and guide area areas, etc. According to these contents, the misidentified parts are filtered to ensure the accuracy and applicability of the new guide area. Afterwards, the guide line label in the segmentation annotation document is converted into a guide area label, and the updated segmentation annotation document is verified to ensure the correctness and consistency of the label. After updating the segmentation annotation document, the first labeled image and the fully manually labeled image need to be annotated and updated to ensure that the category labels in these images are consistent with the updated segmentation annotation document. Read and parse the updated segmentation annotation document, obtain the new category labels and annotation methods, establish a mapping relationship from the old category labels to the new category labels, and update the category labels of the first labeled image and the fully manually labeled image according to the mapping relationship.

[0110] Further, step 105, such as Figure 4 As shown:

[0111] Step 1051: According to the guide line labels, the first label image and the fully artificial label image are converted into binary images of the guide lines and the background.

[0112] Step 1052: determine the edge contour of the guide line from the binary image.

[0113] Step 1053, connecting the edge contours of adjacent guide lines with lines to obtain an initial guide area.

[0114] Step 1054 , filtering the initial diversion area in combination with the target scene to obtain a new diversion area.

[0115] Step 1055 , updating the predefined segmentation annotation document according to the new diversion area, wherein the diversion line labels in the updated segmentation annotation document are converted into diversion area labels.

[0116] Step 1056 , update the annotations of the first labeled image and the fully manually labeled image using the updated segmented annotation document to obtain a second labeled image with updated category labels.

[0117] The above conversion of guide line labels into guide area labels is because the definition of guide areas can divide road areas more finely and improve the accuracy of semantic segmentation. By clarifying the boundaries of guide areas, the errors in segmentation results can be reduced, especially for complex road scenes. In addition, the information of guide areas can provide richer contextual information to help the model better understand the road scenes. It can also provide richer contextual information for the decision-making of the autonomous driving system and improve the accuracy and safety of the decision-making.

[0118] It should be noted that the pixel truth map is the result of classifying each pixel in the image, and the pixels in the first label image and the fully artificial label image are all labeled, so the first label image and the fully artificial label image can be considered as pixel truth maps. Binary images are a special type of image in image processing, in which each pixel has only two possible values: 0 (representing the background) or 1 (representing the foreground). Binary images are often used in image segmentation tasks. By distinguishing the foreground and background, the area of ​​interest is extracted. Binary images can be used to extract features such as edges and contours in the image. The 4-point box is a rectangular bounding box defined by four vertex coordinates. A rectangular box is drawn in the image using annotation tools (such as LabelMe, VGG Image Annotator (VIA), etc.) to represent the boundary of the object.

[0119] Step 106: Train the initial semantic segmentation model using the second label image, the updated category label and the preset weight based on the category label to obtain a semantic segmentation model.

[0120] After obtaining the second label image, the embodiment of the present invention can perform scaling, cropping and other processing, and then input it into the semantic segmentation initial model for training. However, before obtaining the semantic segmentation initial model, its framework will be set first, and a multi-scale network architecture suitable for the semantic segmentation task will be selected, such as U-Net, DeepLab, etc. A multi-scale processing module is introduced into the network to capture image features of different scales. In the multi-scale network, adding multiple scale output branches can better process image information of different scales. For example, in the main network, convolution is used for feature downsampling extraction, and 1 / 2, 1 / 4, ..., 1 / 2 is used. y When the scale is doubled, an output branch is added to output features of a specific scale, and a jump connection and upsampling feature fusion module are added between the scale output branches to obtain fusion features. Then the parameters in the multi-scale network are randomly initialized. The specific steps include:

[0121] Pre-set multi-scale network;

[0122] Add multiple scale output branches according to preset scales in the multi-scale network;

[0123] Add feature fusion branches between output branches of different scales;

[0124] The parameters of the multi-scale network are randomly initialized to obtain the initial semantic segmentation model.

[0125] This process ensures that the model can handle image information of different scales and improves the performance and robustness of the model.

[0126] In the embodiment of the present invention, the preset weights based on the category labels can be pre-set scale weights and loss weights. In addition, other parameters can be set as training parameters of the initial model of semantic segmentation, including initial learning rate, training batch size, loss function, learning rate decay strategy, number of iterations, batchsize, etc. When the second label image and the updated category label are input into the initial model of semantic segmentation for training, forward calculation is performed first, and loss calculation is performed in combination with the true value pixel map of the second label image at each scale, and then the parameters are updated by back propagation to generate a semantic segmentation model.

[0127] After the semantic segmentation model is generated in the embodiment of the present invention, the effect of the semantic segmentation model will be tested. At this time, the divided test image will be obtained from the road image set. The test image is preprocessed (such as normalization, scaling, etc.) to ensure that the image meets the input requirements of the model, and the preprocessed test image is input into the semantic segmentation model. The model infers the test image to generate a semantic segmentation image, and extracts the category label of each pixel from the semantic segmentation image. In order to visualize the semantic segmentation results, it is necessary to convert the category labels in the semantic segmentation image into corresponding color labels, so different category labels in the semantic segmentation task are defined (such as "road", "vehicle", "pedestrian", etc.), and a unique color label is assigned to each category label (such as red for "road", blue for "vehicle", etc.), and a mapping relationship between category labels and color labels is established, and category color pairs are generated, and the category color pairs are saved to ensure the reproducibility and traceability of the mapping relationship. According to the category-color pair, map each category label in the semantic segmentation image to the corresponding color label, determine the target color label of each pixel, add the corresponding color label to the category label of each pixel in the semantic segmentation image, generate a color label image (at this time, the color label image has both category labels and color labels), and ensure that the color label of each pixel is accurate. Select a suitable image fusion method (such as overlay, transparency adjustment, etc.), fuse the test image and the color label image, and ensure that the fused image can clearly display the semantic segmentation results through different colors while retaining the details of the original image. The specific steps include:

[0128] Set different color labels for different category labels in advance to obtain category-color pairs;

[0129] Input the test image into the semantic segmentation model to obtain the semantic segmentation image and the category label of the semantic segmentation image of the test image;

[0130] According to the category color pair, determine the target color label corresponding to the category label of the semantic segmentation image;

[0131] According to the target color label, obtain the color label image of the semantic segmentation image;

[0132] The test image and the color label image are fused to obtain a visualization image of the semantic segmentation image.

[0133] The present invention obtains a road image set, which includes training images. Through the training images, a full-artificial label image and a semi-artificial label image are obtained, wherein the full-artificial label image refers to a road image with category labels manually annotated for the entire image area, and the semi-artificial label image refers to a road image with category labels manually annotated for a predefined image recognition difficult area. Through the semi-artificial label image, a first label image is obtained, and the category label of the image recognition difficult area of ​​the first label image is obtained by combining manual labeling and automatic labeling of a preset model, and the category label of the remaining area is obtained by automatic labeling of a preset model; not only the cost of manual labeling is reduced, but also because only part of the area is annotated, errors are reduced, and the recognition effect of the image recognition difficult area is improved; guide line labels are respectively obtained from the category labels of the first label image and the category labels of the full-artificial label image; the guide line labels of the first label image and the guide line labels of the full-artificial label image are both converted into guide area labels to obtain a second label image with updated category labels, thereby reducing errors in segmentation results and improving segmentation accuracy; a semantic segmentation initial model is trained through the second label image, the updated category labels and the preset weights based on the category labels to obtain a semantic segmentation model. The embodiment of the present invention reduces the cost of manual labeling and reduces errors by using training images of semi-artificially labeled images, thereby ensuring the quality of labeling. By converting the guide line labels of the training images into guide area labels, the errors in the segmentation results are reduced, the segmentation accuracy is improved, and ultimately the performance of the obtained semantic segmentation model is improved.

[0134] The second embodiment of the present application relates to a model training device, Figure 5 is a flow chart of a model training device according to an exemplary embodiment. Figure 5 As shown, the following steps are included:

[0135] The first acquisition module 201 is used to acquire a road image set, where the road image set includes training images.

[0136] The second acquisition module 202 is used to acquire fully artificially labeled images and semi-artificially labeled images through training images, wherein the fully artificially labeled images refer to road images with category labels manually annotated on the entire image area, and the semi-artificially labeled images refer to road images with category labels manually annotated on pre-defined image recognition difficult areas.

[0137] The third acquisition module 203 is used to acquire the first labeled image through the semi-artificial labeled image, the category label of the image recognition difficult area of ​​the first labeled image is obtained by combining manual labeling and automatic labeling of a preset model, and the category label of the remaining area is obtained by automatic labeling of a preset model.

[0138] The fourth acquisition module 204 is used to respectively acquire flow line labels from the category labels of the first label image and the category labels of the fully manual label image.

[0139] The conversion module 205 is used to convert the guide line labels of the first label image and the guide line labels of the fully manual label image into guide area labels to obtain a second label image with updated category labels.

[0140] The training module 206 is used to train the initial semantic segmentation model using the second label image, the updated category label and the preset weight based on the category label to obtain the semantic segmentation model.

[0141] Optionally, the first acquisition module 201 specifically includes:

[0142] The first acquisition submodule is used to acquire road image videos containing different road scenes.

[0143] The first screening submodule is used to randomly screen out a number of road images from the road image video to obtain a road image set.

[0144] The division submodule is used to divide the road image set into training images and test images according to a preset ratio.

[0145] Optionally, the second acquisition module 202 specifically includes:

[0146] The second screening submodule is used to screen out the first road image belonging to the difficult-to-recognize scene from the training image.

[0147] The second acquisition submodule is used to acquire a predefined segmentation and annotation document, in which different category labels and different annotation methods corresponding to different category labels are defined.

[0148] The third acquisition submodule is used to acquire a fully manually labeled image based on the second road image. The fully manually labeled image is obtained by manually labeling the entire image area of ​​the second road image using the segmentation and labeling document.

[0149] The first determining submodule is used to determine the image recognition difficult area of ​​the first road image.

[0150] The fourth acquisition submodule is used to acquire a semi-artificial label image according to the first road image, where the semi-artificial label image is obtained by manually labeling the image recognition difficult area of ​​the first road image by using the segmentation and labeling document.

[0151] Optionally, the third acquisition module 203 specifically includes:

[0152] The fifth acquisition submodule is used to obtain a pre-flash image and a difficult image through a semi-artificial label image. The pre-flash image is an image obtained by automatically annotating the entire image area of ​​the semi-artificial label image through a preset model, and the difficult image is an image manually annotated with an image recognition difficulty area in the semi-artificial label image.

[0153] The merging submodule is used to obtain a first label image through the pre-flash image and the difficult image.

[0154] Optionally, the merged submodules specifically include:

[0155] The screening unit is used to screen out category labels belonging to the difficult area of ​​image recognition from the pre-brush image.

[0156] The first acquisition unit is used to acquire the manually annotated category label in the difficult image.

[0157] The second acquisition unit is used to acquire different priorities of different category labels in the difficult image and the pre-flash image.

[0158] The determination unit is used to determine the target category label having a higher priority than the pre-flash image in the difficult image and the first area where the target category label is located.

[0159] The third acquisition unit is used to acquire the corresponding second area from the pre-flash image.

[0160] The replacement unit is used to replace the category label of the second area with the target category label of the first area to obtain a first label image.

[0161] Optionally, the conversion module 205 specifically includes:

[0162] The conversion submodule is used to convert the first label image and the fully artificial label image into a binary image of the guide line and the background according to the guide line label.

[0163] The second determination submodule is used to determine the edge contour of the guide line from the binary image.

[0164] The connection submodule is used to connect the edge contours of adjacent guide lines with lines to obtain an initial guide area.

[0165] The filtering submodule is used to filter the initial diversion area in combination with the target scene to obtain a new diversion area.

[0166] The first updating submodule is used to update the predefined segmentation annotation document according to the new diversion area, wherein the diversion line labels in the updated segmentation annotation document are converted into diversion area labels.

[0167] The second updating submodule is used to update the annotations of the first label image and the fully manually labeled image through the updated segmentation annotation document to obtain a second label image with updated category labels.

[0168] Optionally, the model training device further includes:

[0169] The first setting module is used to pre-set the multi-scale network.

[0170] The first branch adding module is used to add multiple scale output branches according to preset scales in the multi-scale network.

[0171] The second branch adding module is used to add feature fusion branches between output branches of different scales.

[0172] The initialization module is used to randomly initialize the parameters of the multi-scale network to obtain the initial semantic segmentation model.

[0173] The road image set also includes test images.

[0174] The second setting module is used to pre-set different color labels for different category labels to obtain category-color pairs.

[0175] The input module is used to input the test image into the semantic segmentation model to obtain the semantic segmentation image of the test image and the category label of the semantic segmentation image.

[0176] The determination module is used to determine the target color label corresponding to the category label of the semantic segmentation image according to the category color pair.

[0177] The fifth acquisition module is used to acquire a color label image of the semantic segmentation image according to the target color label.

[0178] The fusion module is used to fuse the test image and the color label image to obtain a visualization image of the semantic segmentation image.

[0179] The present invention obtains a road image set, which includes training images. Through the training images, a full-artificial label image and a semi-artificial label image are obtained, wherein the full-artificial label image refers to a road image with category labels manually annotated for the entire image area, and the semi-artificial label image refers to a road image with category labels manually annotated for a predefined image recognition difficult area. Through the semi-artificial label image, a first label image is obtained, and the category label of the image recognition difficult area of ​​the first label image is obtained by combining manual labeling and automatic labeling of a preset model, and the category label of the remaining area is obtained by automatic labeling of a preset model; not only the cost of manual labeling is reduced, but also because only part of the area is annotated, errors are reduced, and the recognition effect of the image recognition difficult area is improved; guide line labels are respectively obtained from the category labels of the first label image and the category labels of the full-artificial label image; the guide line labels of the first label image and the guide line labels of the full-artificial label image are both converted into guide area labels to obtain a second label image with updated category labels, thereby reducing errors in segmentation results and improving segmentation accuracy; a semantic segmentation initial model is trained through the second label image, the updated category labels and the preset weights based on the category labels to obtain a semantic segmentation model. The embodiment of the present invention reduces the cost of manual labeling and reduces errors by using training images of semi-artificially labeled images, thereby ensuring the quality of labeling. By converting the guide line labels of the training images into guide area labels, the errors in the segmentation results are reduced, the segmentation accuracy is improved, and ultimately the performance of the obtained semantic segmentation model is improved.

[0180] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0181] The third embodiment of the present application relates to an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement any model training method.

[0182] Figure 6 1 is a block diagram of an electronic device 300 according to an exemplary embodiment. For example, the electronic device 300 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0183] Reference Figure 6 , the electronic device 300 may include one or more of the following components: a processing component 302 , a memory 304 , a power component 306 , a multimedia component 308 , an audio component 310 , an input / output interface 312 , a sensor component 314 , and a communication component 316 .

[0184] The processing component 302 generally controls the overall operation of the device 300, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 302 may include one or more processors 320 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 302 may include one or more modules to facilitate the interaction between the processing component 302 and other components. For example, the processing component 302 may include a multimedia module to facilitate the interaction between the multimedia component 308 and the processing component 302.

[0185] The memory 304 is configured to store various types of data to support operations on the device 300. Examples of such data include instructions for any application or method operating on the device 300, contact data, phone book data, messages, pictures, videos, etc. The memory 304 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0186] The power supply component 306 provides power to the various components of the electronic device 300. The power supply component 306 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 300.

[0187] The multimedia component 308 includes a screen that provides an output interface between the electronic device 300 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 308 includes a front camera and / or a rear camera. When the electronic device 300 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0188] The audio component 310 is configured to output and / or input audio signals. For example, the audio component 310 includes a microphone (MIC), and when the electronic device 300 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 304 or sent via the communication component 316. In some embodiments, the audio component 310 also includes a speaker for outputting audio signals.

[0189] The input / output interface 312 provides an interface between the processing component 302 and the peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0190] The sensor assembly 314 includes one or more sensors for providing various aspects of status assessment for the electronic device 300. For example, the sensor assembly 314 can detect the open / closed state of the electronic device 300, the relative positioning of components, such as the display and keypad of the electronic device 300, and the sensor assembly 314 can also detect the position change of the electronic device 300 or a component of the electronic device 300, the presence or absence of user contact with the electronic device 300, the orientation or acceleration / deceleration of the electronic device 300, and the temperature change of the electronic device 300. The sensor assembly 314 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 314 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 314 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0191] The communication component 316 is configured to facilitate wired or wireless communication between the electronic device 300 and other devices. The electronic device 300 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 316 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 316 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0192] In an exemplary embodiment, the electronic device 300 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0193] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 304 including instructions, and the instructions can be executed by a processor 320 of an electronic device 300 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0194] In an exemplary embodiment, a vehicle is also provided, which may specifically include: the above-mentioned model training device.

[0195] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0196] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names. The present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from its scope. The scope of the present invention is limited only by the appended claims.

Claims

1. A model training method, characterized in that: The method comprises: Acquire a road image set, wherein the road image set includes a training image; Obtaining fully artificially labeled images and semi-artificially labeled images through the training images, wherein the fully artificially labeled images refer to road images with artificially labeled category labels for the entire image area, and the semi-artificially labeled images refer to road images with artificially labeled category labels for pre-defined image recognition difficult areas; Obtaining a first labeled image through the semi-artificial labeled image, wherein the category labels of the image recognition difficult areas of the first labeled image are obtained by combining manual labeling and automatic labeling of a preset model, and the category labels of the remaining areas are obtained by automatic labeling of a preset model; Obtain guide line labels from the category labels of the first label image and the category labels of the fully manual label image respectively; Converting the guide line labels of the first label image and the guide line labels of the fully manual label image into guide area labels to obtain a second label image with updated category labels; The semantic segmentation initial model is trained using the second label image, the updated category label and the preset weight based on the category label to obtain a semantic segmentation model.

2. The method according to claim 1, characterized in that The step of acquiring a road image set comprises: Acquire road image videos containing different road scenes; Randomly selecting a number of road images from the road image video to obtain a road image set; The road image set is divided into training images and test images according to a preset ratio.

3. The method according to claim 1, characterized in that The step of obtaining a fully artificially labeled image and a semi-artificially labeled image through the training image includes: Screening out a first road image belonging to a difficult-to-recognize scene and a second road image belonging to a normal scene from the training images; Obtaining a predefined segmentation and annotation document, wherein the segmentation and annotation document defines different category labels and different annotation methods corresponding to the different category labels; According to the second road image, a fully manually labeled image is obtained, where the fully manually labeled image is obtained by manually labeling the entire image area of ​​the second road image using the segmented and labeled document; determining an image recognition difficult area of ​​the first road image; According to the first road image, a semi-artificially labeled image is acquired, where the semi-artificially labeled image is obtained by manually labeling the image recognition difficult area of ​​the first road image using the segmented and labeled document.

4. The method according to claim 1, characterized in that: The step of obtaining a first label image through the semi-artificial label image includes: Through the semi-artificial label image, a pre-brush image and a difficult image are obtained, wherein the pre-brush image is an image obtained by automatically annotating the entire image area of ​​the semi-artificial label image through a preset model, and the difficult image is an image manually annotated with an image recognition difficult area in the semi-artificial label image; A first label image is acquired through the pre-flash image and the difficult image, wherein the category label of the first label image is obtained by merging the category label of the pre-flash image and the category label of the difficult image.

5. The method according to claim 4, characterized in that The step of obtaining a first label image through the pre-flash image and the difficult image includes: Filtering out category labels belonging to image recognition difficult areas from the pre-brush image; Obtaining manually annotated category labels in the difficult image; Get different priorities for different category labels in difficult images and pre-brush images; Determine a target category label having a higher priority than the pre-flash image in the difficult image and a first region where the target category label is located; Acquire a corresponding second area from the pre-flash image; The category label of the second area is replaced with the target category label of the first area to obtain a first label image.

6. The method according to claim 1, characterized in that The step of converting the guide line labels of the first label image and the guide line labels of the fully manual label image into guide area labels to obtain a second label image with updated category labels includes: According to the guide line label, converting the first label image and the fully artificial label image into binary images of the guide line and the background; Determine the edge contour of the guide line from the binary image; Connect the edge contours of adjacent flow guide lines with lines to obtain the initial flow guide area; Filtering the initial diversion area in combination with the target scene to obtain a new diversion area; The predefined segmentation annotation document is updated according to the new diversion area, wherein the diversion line labels in the updated segmentation annotation document are converted into diversion area labels; The first label image and the fully manual label image are labeled and updated using the updated segmented label document to obtain a second label image with updated category labels.

7. The method according to claim 1, characterized in that The method further includes: training the initial semantic segmentation model by using the second label image, the updated category label and the preset weight based on the category label to obtain the semantic segmentation model; Pre-set multi-scale network; Adding multiple scale output branches in the multi-scale network according to preset scales; Add feature fusion branches between output branches of different scales; After randomly initializing the parameters of the multi-scale network, an initial semantic segmentation model is obtained.

8. The method according to claim 1, characterized in that The road image set also includes a test image; After the semantic segmentation model is obtained by training the initial semantic segmentation model using the second label image, the updated category label and the preset weight based on the category label, the method further includes: Set different color labels for different category labels in advance to obtain category-color pairs; Inputting the test image into the semantic segmentation model to obtain a semantic segmentation image of the test image and a category label of the semantic segmentation image; Determining a target color label corresponding to the category label of the semantic segmentation image according to the category color pair; According to the target color label, obtaining a color label image of the semantic segmentation image; The test image and the color label image are fused to obtain a visualization image of the semantic segmentation image.

9. A model training device, characterized in that: include: A first acquisition module, used to acquire a road image set, wherein the road image set includes a training image; A second acquisition module is used to acquire a fully artificially labeled image and a semi-artificially labeled image through the training image, wherein the fully artificially labeled image refers to a road image with category labels manually annotated on the entire image area, and the semi-artificially labeled image refers to a road image with category labels manually annotated on a pre-defined image recognition difficult area; A third acquisition module is used to acquire a first labeled image through the semi-artificial labeled image, wherein the category label of the image recognition difficult area of ​​the first labeled image is obtained by combining the manual labeling and the automatic labeling of the preset model, and the category label of the remaining area is obtained by the automatic labeling of the preset model; A fourth acquisition module, used to respectively acquire flow line labels from the category labels of the first label image and the category labels of the fully manual label image; A conversion module, used to convert the guide line labels of the first label image and the guide line labels of the fully manual label image into guide area labels, so as to obtain a second label image with updated category labels; The training module is used to train the initial semantic segmentation model through the second label image, the updated category label and the preset weight based on the category label to obtain the semantic segmentation model.

10. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to execute the instructions to implement the model training method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by the processor of the mobile terminal, the mobile terminal is enabled to execute the model training method as described in any one of claims 1 to 8.

12. A vehicle, characterized in that: Including the model training device described in claim 9.

Images