Vehicle automatic driving scene data labeling method and system, server and medium

Abstract

The application discloses a kind of vehicle automatic driving scene data labeling method, system, server and medium.The method includes the following steps: in response to automatic driving instruction information, initial image information is collected in real time;Pretreatment initial image information, obtain fusion image information and the sub-image set corresponding to fusion image information;Sub-image set includes: several sub-image information, sub-image information at least includes: segmentation element;Traversal sub-image set, respectively input each segmentation element to identification model, to obtain identification result;Identification result includes identified or unidentified;When judging identification result as unidentified, the sub-image information corresponding to current segmentation element is stored in the to-be-labeled image set;When judging that the to-be-labeled image set has at least one sub-image information, first warning information is issued;First warning information is used to prompt driver to label.The application can expand the coverage scene and quantity of scene data set, to reduce the probability of accident.

Description

Technical Field

[0001] This invention generally relates to the field of autonomous vehicle technology, and specifically to methods, systems, servers, and media for labeling data in autonomous driving scenarios. Background Technology

[0002] As the level of intelligence in automobiles gradually increases, autonomous vehicles are becoming more and more common in our lives. However, this has also led to an increase in accidents caused by the incomplete design of autonomous driving systems. According to the analysis of most accidents, the main cause is that the training data of the intelligent driving models of OEMs is insufficient, resulting in the vehicle's incorrect perception and recognition of road objects.

[0003] Furthermore, due to the limited number of personnel available for data annotation, the annotation work is often completed by a single person, resulting in a limited amount of annotated data and insufficient coverage of scenarios, which can easily lead to serious accidents. Therefore, we propose a data annotation method, system, server, and medium for autonomous driving scenarios to address the aforementioned problems. Summary of the Invention

[0004] In view of the above-mentioned defects or deficiencies in the existing technology, it is desirable to provide a method, system, server and medium for labeling vehicle autonomous driving scenario data to effectively expand the coverage and quantity of scenario datasets, reduce the probability of accidents and increase user stickiness.

[0005] In a first aspect, the present invention provides a method for labeling data in a vehicle autonomous driving scenario, comprising the following steps:

[0006] In response to autonomous driving command information, initial image information is acquired in real time; the initial image information includes at least: image data information and point cloud data information.

[0007] The initial image information is preprocessed to obtain fused image information and a set of sub-images corresponding to the fused image information; the set of sub-images includes: several sub-image information, and the sub-image information includes at least: segmentation elements;

[0008] The sub-image set is traversed, and each segmentation element is input into the recognition model to obtain the recognition result; the recognition result includes whether it has been recognized or not.

[0009] When the recognition result is determined to be unrecognized, the sub-image information corresponding to the current segmentation element is stored in the image set to be labeled;

[0010] When it is determined that the set of images to be labeled has at least one sub-image information, a first warning message is issued; the first warning message is used to prompt the driver to make the label.

[0011] According to the technical solution provided by the embodiments of the present invention, after issuing the first warning information, the method further includes the following steps:

[0012] In response to the confirmation annotation information input by the driver, the fused image information and the set of images to be annotated are sent to the first terminal; the first terminal has a first display area, which is used to display at least one sub-image information from the fused image information and the set of images to be annotated;

[0013] Receive the first annotation information input by the driver, and associate the first annotation information with the segmentation elements of the sub-image information in the image set to be annotated to obtain the first image information after annotation is completed;

[0014] The first image information is sent to the cloud server.

[0015] According to the technical solution provided in the embodiments of the present invention, preprocessing the initial image information to obtain fused image information and a set of sub-images corresponding to the fused image information specifically includes the following steps:

[0016] The image data information and the point cloud data information are fused to obtain fused image information;

[0017] The fused image information is input into a semantic segmentation model to obtain a set of sub-images corresponding to the fused image information.

[0018] According to the technical solution provided in the embodiments of the present invention, each segmentation element is input into the recognition model to obtain the recognition result, specifically including the following steps:

[0019] Obtain the chaos index of the segmented element;

[0020] If the chaos index is less than or equal to the first threshold, the identification result is determined to be unidentified.

[0021] According to the technical solution provided by the embodiments of the present invention, the set of images to be labeled includes a first subset and a second subset;

[0022] The sub-image information corresponding to the current segmentation element is stored in the image set to be labeled, which includes the following steps:

[0023] When the chaos index is determined to be less than or equal to the second threshold, the sub-image information corresponding to the current segmentation element is stored in the first subset;

[0024] When it is determined that the chaos index is less than or equal to the first threshold and greater than the second threshold, the sub-image information corresponding to the current segmentation element is stored in the second subset.

[0025] According to the technical solution provided by the embodiments of the present invention, after determining that the chaos index is less than or equal to the first threshold and greater than the second threshold, and before storing the sub-image information corresponding to the current segmentation element into the second subset, the method further includes the following steps:

[0026] Get the category name corresponding to the current segment element;

[0027] The sub-image information to which the current segmentation element belongs is pre-labeled with first confirmation data; the first confirmation data includes at least the category name.

[0028] According to the technical solution provided in the embodiments of the present invention, after determining that the chaos index is less than or equal to the first threshold, and before storing the sub-image information corresponding to the current segmentation element into the image set to be labeled, the following steps are further included:

[0029] The sub-image information to which the current segmentation element belongs is pre-labeled with second confirmation data; the second confirmation data includes: boundary segmentation contour lines.

[0030] Secondly, the present invention provides a vehicle autonomous driving scene data annotation system, implemented based on the above-mentioned vehicle autonomous driving scene data annotation method, wherein the vehicle autonomous driving scene data annotation system includes:

[0031] The data acquisition module is configured to acquire initial image information in real time in response to autonomous driving command information; the initial image information includes at least: image data information and point cloud data information;

[0032] The processing module is configured to preprocess the initial image information to obtain fused image information and a set of sub-images corresponding to the fused image information; the set of sub-images includes: several sub-image information, and the sub-image information includes at least: segmentation elements;

[0033] The sub-image set is traversed, and each segmentation element is input into the recognition model to obtain the recognition result; the recognition result includes whether it has been recognized or not.

[0034] When the recognition result is determined to be unrecognized, the sub-image information corresponding to the current segmentation element is stored in the image set to be labeled;

[0035] When it is determined that the set of images to be labeled has at least one sub-image information, a first warning message is issued; the first warning message is used to prompt the driver to make the label.

[0036] Thirdly, the present invention provides a server, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method for labeling data in a vehicle autonomous driving scenario.

[0037] Fourthly, the present invention provides a computer-readable storage medium having a computer program, which, when executed by a processor, implements the steps of the above-described method for labeling data in a vehicle autonomous driving scenario.

[0038] In summary, this invention specifically discloses a detailed process for a data annotation method for autonomous driving scenarios. This invention involves: real-time acquisition of initial image information in response to autonomous driving command information; preprocessing the initial image information to obtain fused image information and a set of sub-images corresponding to the fused image information; traversing the sub-image set and inputting each segmentation element into a recognition model to obtain a recognition result; when the recognition result is determined to be unrecognized, storing the sub-image information corresponding to the current segmentation element into the image set to be annotated; and issuing a first warning message when the image set to be annotated has at least one sub-image information.

[0039] Compared to the traditional data annotation method, which suffers from limited data due to dedicated personnel and is prone to accidents, this invention filters out segmentation elements that the recognition model cannot identify, forming a set of images to be annotated. Drivers then annotate this set of images, thereby expanding the coverage and quantity of the scene dataset and reducing the probability of accidents. Attached Figure Description

[0040] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings.

[0041] Figure 1 A flowchart illustrating the data annotation method for autonomous driving scenarios.

[0042] Figure 2 A flowchart illustrating the process of processing a set of images to be labeled for a driver.

[0043] Figure 3 A flowchart illustrating the process of a data annotation system for autonomous driving scenarios.

[0044] Figure 4 This is a schematic diagram of a server-side principle.

[0045] Figure 5 This is a schematic diagram of the first display area.

[0046] The diagram shows: 1. Data acquisition module; 2. Processing module;

[0047] 500. Server; 501. CPU; 502. ROM; 503. RAM; 504. Bus; 505. I / O interface; 506. Input section; 507. Output section; 508. Storage section; 509. Communication section; 510. Driver; 511. Removable media. Detailed Implementation

[0048] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0049] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0050] Example 1

[0051] Please refer to Figure 1 The flowchart shown is a first embodiment of a vehicle autonomous driving scene data annotation method provided by the present invention, which includes the following steps:

[0052] S10. In response to autonomous driving command information, acquire initial image information in real time; the initial image information includes at least: image data information and point cloud data information;

[0053] Among them, autonomous driving instruction information refers to the instruction information for the vehicle to activate autonomous driving mode; image data information is collected by the vehicle-mounted camera, and point cloud data information is collected by the vehicle-mounted LiDAR;

[0054] Here, the model of the vehicle camera is, for example, Lingdu Z300S; the model of the vehicle LiDAR is, for example, Solid State LiDAR-CS20.

[0055] S20. Preprocess the initial image information to obtain fused image information and a set of sub-images corresponding to the fused image information; the set of sub-images includes: several sub-image information, and the sub-image information includes at least: segmentation elements;

[0056] Specifically, it includes the following steps:

[0057] Image data and point cloud data are fused to obtain fused image information. Here, the camera-Lidar fusion method can be used to fuse image data and point cloud data to obtain fused image information.

[0058] The fused image information is input into the semantic segmentation model to obtain a set of sub-images corresponding to the fused image information.

[0059] Among them, the type of semantic segmentation model is, for example, the fully convolutional neural network (FCN) semantic segmentation model.

[0060] The semantic segmentation model described above classifies each segmentation element in the fused image information, determines the category of each segmentation element, and performs region division, that is, divides the fused image information into several sub-image information, each of which has a corresponding segmentation element.

[0061] The semantic segmentation model divides an image into regions with different semantic information and labels each region with a corresponding semantic label. For example, after performing semantic segmentation on an image, semantic labels can be added to objects in the image (such as zebra crossings, pedestrians, vehicles, dogs, etc.). In other words, the semantic segmentation model can add semantic labels to the segmentation elements in each sub-image information in the fused image information to obtain a set of sub-images. This makes it easier for the subsequent recognition model to recognize the segmentation elements of each sub-image information separately, resulting in more accurate recognition results. It avoids the situation where there are multiple segmentation elements in an image, which may lead to misidentification or missed identification due to similarity or overlap of segmentation elements.

[0062] S30. Traverse the sub-image set, input each segmentation element into the recognition model, and obtain the recognition result; the recognition result includes whether it has been recognized or not.

[0063] The recognition model refers to the segmentation element recognition model configured when the vehicle leaves the factory. The recognition model contains the segmentation elements that the developers have previously input and the corresponding categories of the segmentation elements. If the segmentation element received by the recognition model is a segmentation element that already exists in the recognition model, it can match the corresponding category for this segmentation element. If the segmentation element received by the recognition model does not exist in the recognition model, it cannot match the corresponding category for this segmentation element.

[0064] The aforementioned recognition model can identify a certain number of segmentation elements that have been matched with corresponding categories. When a segmentation element of a certain sub-image information is input into the recognition model, the recognition model compares the segmentation element of the sub-image information with the existing segmentation elements in the recognition model. If the segmentation element matches one of the existing segmentation elements, the recognition model can identify the category of the segmentation element. If the segmentation element cannot be found in the recognition model, the recognition model cannot identify the category of the segmentation element, that is, the segmentation element does not exist in the recognition model.

[0065] Accordingly, the criterion for determining whether the recognition result is "recognized" or "unrecognized" is whether the corresponding segmentation element exists in the recognition model. If the corresponding segmentation element exists in the recognition model, the recognition result is "recognized," meaning the recognition model can identify the segmentation element and match the corresponding category. If the segmentation element does not exist in the recognition model, the recognition result is "unrecognized," meaning the recognition model cannot identify the segmentation element and cannot determine the corresponding category. The determination of the "unrecognized" result includes two situations: one is that the recognition model cannot find a category name very similar to the current segmentation element; the other is that although a category name similar to the current segmentation element can be found, the matching degree between the current segmentation element and all elements under that category name is low.

[0066] Specifically, it includes the following steps:

[0067] Obtain the chaos index of the segmented elements;

[0068] When the chaos index is determined to be less than or equal to the first threshold, the identification result is determined to be unidentified; and, those skilled in the art can set the first threshold based on experience, here, the first threshold is, for example, 0.5.

[0069] By acquiring and judging the chaos index of segmentation elements, the sub-image information corresponding to segmentation elements whose categories were not identified by the recognition model is filtered out and stored in the image set to be labeled, waiting for the driver to label them.

[0070] Furthermore, the accuracy of the fused image information can be calculated based on the chaos index. When the accuracy is less than or equal to 80%, it indicates that the visibility of the fused image information is not high, and all its corresponding sub-image information is stored in the image set to be labeled in order to determine the category name of the segmented element. When the accuracy is greater than 80%, it indicates that the visibility of the fused image information is high, and it is not necessary to store all its corresponding sub-image information in the image set to be labeled, which can reduce the user's workload to a certain extent.

[0071] The formula for calculating accuracy is: Accuracy = Number of recognizable segmented elements / Total number of segmented elements. Here, unrecognizable segmented elements are defined as segmented elements with a chaos index less than or equal to 0.3, and recognizable segmented elements are segmented elements with a chaos index greater than 0.3.

[0072] S40. When the recognition result is determined to be unrecognized, the sub-image information corresponding to the current segmentation element is stored in the image set to be labeled.

[0073] The set of images to be labeled includes a first subset and a second subset;

[0074] The sub-image information corresponding to the current segmentation element is stored in the image set to be labeled, which includes the following steps:

[0075] When the chaos index is determined to be less than or equal to the second threshold, the sub-image information corresponding to the current segmentation element is stored in the first subset;

[0076] When the chaos index is determined to be less than or equal to the first threshold and greater than the second threshold, the sub-image information corresponding to the current segmentation element is stored in the second subset; wherein, the first threshold is greater than the second threshold, and those skilled in the art can set the second threshold based on experience, for example, 0.3.

[0077] By further judging the chaos index of the segmentation elements of the sub-image information to be stored in the set of images to be labeled, the sub-image information is divided into two subsets, namely the first subset and the second subset. The sub-image information in the first subset consists of images whose categories cannot be identified at all, requiring the driver to fill in the corresponding category name. The sub-image information in the second subset consists of images whose categories are fuzzy but can be matched with the category name. Through the above division process, the category filling and judgment requirements of the sub-image information are accurately divided, avoiding the need for the driver to fill in all unidentified results. This reduces the workload of labeling while requesting the driver's cooperation in labeling, resulting in a better user experience.

[0078] Furthermore, the sub-image information in the first subset and the second subset must be labeled with second confirmation data before being stored. The second confirmation data includes: boundary segmentation contour lines; the boundary segmentation contour lines refer to the outer edge lines of the segmentation elements in each sub-image information.

[0079] By annotating the boundary segmentation contours of all sub-image information that needs to be stored in the first and second subsets, the segmentation elements in each sub-image information become more obvious and prominent, avoiding the influence of surrounding objects on the classification result of the segmentation element and improving the accuracy of segmentation element classification.

[0080] Furthermore, after determining that the chaos index is less than or equal to the first threshold and greater than the second threshold, and before storing the sub-image information corresponding to the current segmentation element into the second subset, the following steps are also included:

[0081] Get the category name corresponding to the current segment element;

[0082] The sub-image information to which the current segmentation element belongs is pre-labeled with first confirmation data; the first confirmation data includes at least the category name.

[0083] The sub-image information that needs to be stored in the second subset is pre-labeled with category names, so that all unidentified but category-matched sub-image information stored in the second subset only needs to be confirmed by the driver to be accurate, which reduces the driver's labeling workload to a certain extent.

[0084] S50. When it is determined that the set of images to be labeled has at least one sub-image information, a first warning message is issued; the first warning message is used to prompt the driver to label the image.

[0085] Furthermore, such as Figure 2 As shown, after issuing the first warning message, the following steps are also included:

[0086] S60. In response to the confirmation annotation information input by the driver, the fused image information and the set of images to be annotated are sent to the first terminal; the first terminal has a first display area, which is used to display at least one sub-image information in the fused image information and the set of images to be annotated;

[0087] Among them, the confirmation annotation information refers to the information of the driver starting the data annotation; the first terminal refers to the in-vehicle entertainment terminal (IVI), which has an in-vehicle data annotation APP for the driver to perform data annotation; the first terminal can also store fused image information and a set of images to be annotated; the first display area is the display screen of the in-vehicle entertainment terminal.

[0088] S70. Receive the first annotation information input by the driver, and associate the first annotation information with the segmentation elements of the sub-image information in the image set to be annotated to obtain the first image information after annotation is completed.

[0089] S80. Send the first image information to the cloud server.

[0090] The first annotation information includes at least one of the following: confirmation information of category name and category name information. That is, the first annotation information input by the driver may be confirmation information of the category name of the segmenting element of a certain sub-image information (confirmation information includes yes or no), or it may be the category name of the segmenting element of a certain sub-image information.

[0091] Furthermore, the cloud server includes: a telematics BOX (T-BOX), a data annotation and operation platform, and an autonomous driving development platform. The telematics BOX receives the first image information and the sub-image information corresponding to the segmented elements identified by the recognition model. The data annotation and operation platform sends the first image information and the sub-image information corresponding to the segmented elements identified by the recognition model to the autonomous driving development platform. Developers verify the data on the autonomous driving development platform and use the autonomous driving development platform to feed back the verified results to the data annotation and operation platform. The data annotation and operation platform then updates each recognition model to expand the data in each recognition model, thereby achieving the goal of expanding the coverage and quantity of the scene dataset.

[0092] When the driver selects a label based on the first warning information, steps S60-S80 are executed. The driver receives the fused image information and the set of images to be labeled on the first terminal. Based on at least one sub-image information in the fused image information and the set of images to be labeled displayed in the first display area, the driver inputs the first labeling information, that is, checks or fills in the category of the segmentation element in the corresponding sub-image information, and sends the labeled first image information to the cloud server. When the current vehicle recognition model cannot meet the recognition of all segmentation elements, by adding the driver's judgment and filling in of the category of segmentation element in the sub-image information, the range of segmentation element recognition can be expanded and user stickiness can be increased.

[0093] Furthermore, after issuing the first warning information and before responding to the driver's input confirmation annotation information, data anonymization processing can be performed on the fused image information and the image information to be annotated in order to encrypt them and prevent data leakage.

[0094] The specific data annotation process for autonomous driving scenarios is as follows:

[0095] First, the vehicle is put into autonomous driving mode, responding to autonomous driving commands and acquiring initial image information in real time. Then, the image data and point cloud data are fused to obtain fused image information, such as... Figure 5 The original comparison image displayed in the lower left area is the fused image information. The fused image information is then input into the semantic segmentation model to obtain the sub-image set corresponding to the fused image information. Here, the number of fused image information is set to 1, and its corresponding sub-image set includes 10 sub-image information.

[0096] Traverse the sub-image set, input each segmentation element into the recognition model, and obtain the chaos index of the ten segmentation elements, which are 1, 1, 1, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, and 0.3 respectively;

[0097] If the chaos index of a segmentation element is less than or equal to 0.5, the sub-image information corresponding to that segmentation element is obtained. If the information of the 8th, 9th and 10th sub-images matches, the recognition result is "unrecognized".

[0098] Next, determine the sub-image information corresponding to the segmentation elements whose chaos index is less than or equal to 0.5 and greater than 0.3 among the segmentation elements of the 8th, 9th and 10th sub-image information. That is, the sub-image information of the 8th and 9th sub-image information meets the criteria. Correspondingly, the sub-image information corresponding to the segmentation elements whose chaos index is less than or equal to 0.3 is the sub-image information of the 10th sub-image. Then, the 10th sub-image information is labeled with the second confirmation data and stored in the first subset. The 8th and 9th sub-image information is pre-labeled with the first confirmation data and the second confirmation data and stored in the second subset.

[0099] After the above judgment and screening process, it can be seen that the segmentation elements of the information of the first to seventh sub-images are the segmentation elements that the recognition model can recognize, so the recognition result is that it has been recognized.

[0100] When it is determined that the set of images to be labeled has at least one sub-image information, a first warning message is issued to prompt the driver to make the label.

[0101] The driver inputs confirmation annotation information, and the first terminal receives the fused image information and the set of images to be annotated; the first terminal displays the fused image information and a sub-image information in its first display area, such as... Figure 5 The area of ​​the image to be labeled, displayed on the right, is for showing sub-image information; such as... Figure 5 The upper left area displays a user interaction Q&A area, which is used to ask the driver whether the pre-labeled data of the sub-image information is accurate, and to fill in at least one of the following: what is the category corresponding to the segmentation element of the sub-image information.

[0102] When the sub-image information displayed in the first display area is the 10th sub-image information, the first display area displays the category name of the segmentation element in the sub-image information. At the same time, the displayed content is broadcast by voice. The first annotation information entered by the driver is the category name filled in by the driver. This category name is the category determined by the driver based on the segmentation element of the 10th sub-image information.

[0103] When the sub-image information displayed in the first display area is the 8th sub-image information, the first display area displays whether the category name of the segmentation element in the sub-image information is a zebra crossing. At the same time, the displayed content is broadcast by voice, prompting the driver to select yes or no. Then the first annotation information input by the driver is the confirmation result given by the driver based on the pre-annotation information of the 8th sub-image information.

[0104] After the driver judges and fills in the set of images to be labeled as described above, three first image information with labels completed are obtained;

[0105] All first image information and fused image information are sent to the cloud server. The cloud server verifies the first image information and the identified sub-image information, and updates the recognition model based on the verification results to expand the data in the recognition model, thereby expanding the coverage and quantity of the scene dataset and reducing the probability of accidents.

[0106] Example 2

[0107] like Figure 3 As shown, a vehicle autonomous driving scene data annotation system is implemented based on the vehicle autonomous driving scene data annotation method described in Embodiment 1. The vehicle autonomous driving scene data annotation system includes:

[0108] Data acquisition module 1 is configured to acquire initial image information in real time in response to autonomous driving command information; the initial image information includes at least: image data information and point cloud data information;

[0109] Processing module 2 is configured to preprocess the initial image information to obtain fused image information and a set of sub-images corresponding to the fused image information; the set of sub-images includes: several sub-image information, and the sub-image information includes at least: segmentation elements;

[0110] Traverse the sub-image set, input each segmentation element into the recognition model, and obtain the recognition result; the recognition result includes whether it has been recognized or not.

[0111] When the recognition result is not recognized, the sub-image information corresponding to the current segmentation element is stored in the image set to be labeled;

[0112] When it is determined that the set of images to be labeled has at least one sub-image information, a first warning message is issued; the first warning message is used to prompt the driver to make the labeling.

[0113] The data acquisition module 1 consists of an in-vehicle camera and an in-vehicle LiDAR. For example, the model of the in-vehicle camera is Lingdu Z300S; and the model of the in-vehicle LiDAR is, for example, solid-state LiDAR-CS20.

[0114] The type of processing module 2 is, for example, an autonomous driving domain controller (Multi-Domain Controller, MDC).

[0115] Example 3

[0116] A server includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of a vehicle autonomous driving scene data annotation method as described in the above embodiments.

[0117] In this embodiment, as Figure 4 As shown, the server 500 includes a CPU 501 (Central Processing Unit), which can perform various appropriate actions and processes based on programs stored in ROM 502 (Read-Only Memory) or programs loaded from storage into RAM 503 (Random Access Memory). RAM 503 also stores various programs and data required for system operation. The CPU 501, ROM 502, and RAM 503 are interconnected via bus 504. An I / O interface 505 (Input / Output) is also connected to bus 504.

[0118] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive is also connected to I / O interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 510 as needed so that computer programs read from it can be installed into storage section 508 as needed.

[0119] In particular, according to embodiments of the present invention, the above-described reference process Figure 1 The described process can be implemented as a computer software program. For example, Embodiment 3 of the present invention includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by CPU 501, it performs the functions defined in the system of the present invention.

[0120] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, RAM (random access memory), ROM (read-only memory), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0121] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0122] The units described in the embodiments of the present invention can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself. The described units or modules can also be located in a processor; for example, a processor can be described as including a first generation module, an acquisition module, a search module, a second generation module, and a merging module. The names of these units or modules do not necessarily limit the specific unit or module itself; for example, the acquisition module can also be described as "an acquisition module for acquiring multiple instances to be probed in the base table".

[0123] Example 4

[0124] The present invention also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to implement a vehicle autonomous driving scene data annotation method as described in the above embodiments.

[0125] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention is not limited to the specific combination of the above-described technical features, but also includes other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in this invention.

Claims

1. A method for labeling data in a vehicle autonomous driving scenario, characterized in that, Includes the following steps: In response to autonomous driving command information, it acquires initial image information in real time; The initial image information includes at least: image data information and point cloud data information; The initial image information is preprocessed to obtain fused image information and a set of sub-images corresponding to the fused image information; the set of sub-images includes: several sub-image information, and the sub-image information includes at least: segmentation elements; The sub-image set is traversed, and each segmentation element is input into the recognition model to obtain the recognition result; the recognition result includes whether it has been recognized or not. When the recognition result is determined to be unrecognized, the sub-image information corresponding to the current segmentation element is stored in the image set to be labeled; When it is determined that the set of images to be labeled has at least one sub-image information, a first warning message is issued; the first warning message is used to prompt the driver to label the image. By further judging the chaos index of the segmentation elements of the sub-image information that needs to be stored in the set of images to be labeled, the sub-image information is divided into a first subset and a second subset; the first subset is used to store sub-image information whose categories cannot be identified at all, and the second subset is used to store sub-image information whose categories are vaguely identified but pre-labeled.

2. The method for labeling vehicle autonomous driving scenario data according to claim 1, characterized in that, After issuing the first warning, the following steps are also included: In response to the confirmation annotation information input by the driver, the fused image information and the set of images to be annotated are sent to the first terminal; the first terminal has a first display area, which is used to display at least one sub-image information from the fused image information and the set of images to be annotated; Receive the first annotation information input by the driver, and associate the first annotation information with the segmentation elements of the sub-image information in the image set to be annotated to obtain the first image information after annotation is completed; The first image information is sent to the cloud server.

3. The method for labeling vehicle autonomous driving scenario data according to claim 1, characterized in that, Preprocessing the initial image information to obtain fused image information and a set of sub-images corresponding to the fused image information specifically includes the following steps: The image data information and the point cloud data information are fused to obtain fused image information; The fused image information is input into a semantic segmentation model to obtain a set of sub-images corresponding to the fused image information.

4. The method for labeling vehicle autonomous driving scenario data according to claim 1, characterized in that, Each segmentation element is input into the recognition model to obtain the recognition result, which includes the following steps: Obtain the chaos index of the segmented element; If the chaos index is less than or equal to the first threshold, the identification result is determined to be unidentified.

5. The method for labeling vehicle autonomous driving scenario data according to claim 4, characterized in that, The set of images to be labeled includes a first subset and a second subset; The sub-image information corresponding to the current segmentation element is stored in the image set to be labeled, which includes the following steps: When the chaos index is determined to be less than or equal to the second threshold, the sub-image information corresponding to the current segmentation element is stored in the first subset; When it is determined that the chaos index is less than or equal to the first threshold and greater than the second threshold, the sub-image information corresponding to the current segmentation element is stored in the second subset; Wherein, the first threshold is greater than the second threshold.

6. The method for labeling vehicle autonomous driving scenario data according to claim 5, characterized in that, After determining that the chaos index is less than or equal to the first threshold and greater than the second threshold, and before storing the sub-image information corresponding to the current segmentation element into the second subset, the following steps are also included: Get the category name corresponding to the current segment element; The sub-image information to which the current segmentation element belongs is pre-labeled with first confirmation data; the first confirmation data includes at least the category name.

7. The method for labeling vehicle autonomous driving scenario data according to claim 4, characterized in that, After determining that the chaos index is less than or equal to the first threshold, and before storing the sub-image information corresponding to the current segmentation element into the image set to be labeled, the following steps are also included: The sub-image information to which the current segmentation element belongs is pre-labeled with second confirmation data; the second confirmation data includes: boundary segmentation contour lines.

8. A vehicle autonomous driving scene data annotation system, implemented based on the vehicle autonomous driving scene data annotation method according to any one of claims 1 to 7, characterized in that, The vehicle autonomous driving scenario data annotation system includes: The data acquisition module (1) is configured to acquire initial image information in real time in response to autonomous driving instruction information; the initial image information includes at least: image data information and point cloud data information; Processing module (2) is configured to preprocess the initial image information to obtain fused image information and a set of sub-images corresponding to the fused image information; the set of sub-images includes: several sub-image information, and the sub-image information includes at least: segmentation elements; The sub-image set is traversed, and each segmentation element is input into the recognition model to obtain the recognition result; the recognition result includes whether it has been recognized or not. When the recognition result is determined to be unrecognized, the sub-image information corresponding to the current segmentation element is stored in the image set to be labeled; When it is determined that the set of images to be labeled has at least one sub-image information, a first warning message is issued; the first warning message is used to prompt the driver to make the label.

9. A server, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of a vehicle autonomous driving scene data annotation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the vehicle autonomous driving scene data annotation method as described in any one of claims 1 to 7.

Images