A data labeling method and device, and a storage medium
By employing self-supervised learning model training and feature extractor porting, the problems of long processing time and high cost in traditional street view data annotation are solved, achieving efficient and low-cost data annotation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Z-ONE TECH CO LTD
- Filing Date
- 2022-08-12
- Publication Date
- 2026-05-12
AI Technical Summary
Traditional street view data annotation methods are time-consuming, inefficient, and costly. When relying on neural network models, they require a large amount of labeled data, and the annotation accuracy is difficult to guarantee when the amount of data is insufficient.
A self-supervised learning model was used to train on unlabeled data. The trained feature extractor was then ported to the semantic segmentation model for street view data annotation.
It reduces reliance on labeled data, improves annotation efficiency and accuracy, and lowers annotation costs.
Smart Images
Figure CN115294336B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a data annotation method, apparatus and related equipment. Background Technology
[0002] With the development of autonomous driving technology, the demand for street scene perception algorithms is increasing. Correspondingly, the design and optimization of street scene perception algorithms often require acquiring and labeling a large amount of real-world street scene data. Traditional data labeling methods mostly rely on manual labeling or using neural network models to perform semantic segmentation on the acquired street scene data before labeling. While the former offers better labeling accuracy, it is time-consuming, inefficient, and costly. Due to the significant manpower required, it is difficult to meet the demands of handling massive amounts of data for semantic segmentation and labeling. The latter, while faster, heavily relies on the variety of labeled sample data or training data used in building and training the neural network model to ensure accuracy. When the amount of labeled sample data or training data is insufficient, the accuracy of semantic segmentation for labeling cannot be effectively guaranteed. Summary of the Invention
[0003] In view of this, embodiments of this application provide a data annotation method, apparatus, and storage medium to at least partially solve the above-mentioned problems.
[0004] In a first aspect, embodiments of this application provide a data annotation method, including:
[0005] Determine the semantic segmentation model based on the street view data to be labeled;
[0006] Based on the established semantic segmentation model, determine the self-supervised learning model;
[0007] The self-supervised learning model is trained using unlabeled street view data to obtain a well-trained self-supervised learning model.
[0008] The feature extractor of the trained self-supervised learning model is transplanted into the semantic segmentation model to obtain the transplanted semantic segmentation model.
[0009] The transplanted semantic segmentation model is used to annotate the street view data to be labeled.
[0010] Optionally, in one embodiment of this application, the self-supervised learning model includes a first front-end backbone network and a pre-task module:
[0011] The feature extractor is included in the first front-end backbone network, which is used to extract image information and generate corresponding feature maps.
[0012] The pre-task module is used for unsupervised semantic extraction to supervise the training of the first front-end backbone network.
[0013] Optionally, in one embodiment of this application, the semantic segmentation model includes a second front-end backbone network and a predictor;
[0014] The predictor is used to process the corresponding feature map to generate a semantic segmentation prediction map.
[0015] Optionally, in one embodiment of this application, determining a self-supervised learning model based on a determined semantic segmentation model includes:
[0016] Based on the structure of the second front-end backbone network of the age semantic segmentation model, the structure of the first front-end backbone network is determined; based on the structure of the first front-end backbone network, the structure of the self-supervised learning model is determined.
[0017] Optionally, in one embodiment of this application, the feature extractor of the trained self-supervised learning model is ported to the semantic segmentation model to obtain the ported semantic segmentation model, including:
[0018] Determine the weights of the feature extractors in the first front-end backbone network of the trained self-supervised learning model;
[0019] Based on the weights of the feature extractors in the first front-end backbone network, the second front-end backbone network of the semantic segmentation model is replaced, and the semantic segmentation model after the second front-end backbone network is replaced is determined as the ported semantic segmentation model.
[0020] Optionally, in one embodiment of this application, the semantic segmentation model is the semantic segmentation model SETR;
[0021] The second front-end backbone network of the semantic segmentation model SETR is a Transformer Layer feature extractor.
[0022] Optionally, in one embodiment of this application, the self-supervised learning model is a self-supervised learning model MOCO;
[0023] The first front-end backbone network of the self-supervised learning model MOCO includes a feature extractor consisting of encoders fq and fk.
[0024] Optionally, in one embodiment of this application, before annotating the street view data to be labeled using the ported semantic segmentation model, the method further includes: fine-tuning the ported semantic segmentation model.
[0025] Secondly, based on the data annotation apparatus method of the first aspect of this application, embodiments of this application also provide a data annotation apparatus, including:
[0026] The first determining module is used to determine the semantic segmentation model based on the street view data to be labeled;
[0027] The second determining module is used to determine the self-supervised learning model based on the determined semantic segmentation model;
[0028] The training module is used to train the self-supervised learning model using unlabeled street view data to obtain a trained self-supervised learning model.
[0029] The transplantation module is used to transplant the feature extractor of the trained self-supervised learning model to the semantic segmentation model, so as to obtain the transplanted semantic segmentation model.
[0030] The annotation module is used to annotate the street view data to be annotated using the ported semantic segmentation model.
[0031] Thirdly, embodiments of this application also provide a storage medium storing a computer program thereon, which, when executed by a processor, implements any of the data annotation methods of the first aspect of this application.
[0032] This application provides a data annotation method, apparatus, and storage medium. The data annotation method includes: determining a semantic segmentation model based on street view data to be annotated; determining a self-supervised learning model based on the determined semantic segmentation model; training the self-supervised learning model using unlabeled street view data to obtain a trained self-supervised learning model; porting the feature extractor of the trained self-supervised learning model to the semantic segmentation model to obtain a ported semantic segmentation model; and annotating the street view data to be annotated using the ported semantic segmentation model. The data annotation method provided in this application, by setting up a semantic segmentation model and a self-supervised learning model, and training the self-supervised learning model using unlabeled data, and porting the feature extractor of the trained self-supervised learning model to the semantic segmentation model, reduces the amount of labeled sample data used in model construction and training, thus ensuring the accuracy of data annotation using the semantic segmentation model while effectively guaranteeing annotation efficiency and reducing annotation costs, since the training process of the self-supervised learning model does not require acquiring massive amounts of labeled sample data. Attached Figure Description
[0033] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.
[0034] Figure 1 A flowchart illustrating a data annotation method provided in this application embodiment;
[0035] Figure 2 This is a schematic diagram of the structure of a data annotation device provided in an embodiment of this application. Detailed Implementation
[0036] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.
[0037] It should be understood that the steps described in the method embodiments of this application may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this application is not limited in this respect.
[0038] Example 1
[0039] This application provides a data annotation method, such as... Figure 1 As shown, Figure 1 This application provides a flowchart of a data annotation method, which includes:
[0040] S101. Determine the semantic segmentation model based on the street view data to be labeled.
[0041] Optionally, in one implementation of this application embodiment, the semantic segmentation model can be determined based on the complexity of the street scene data to be labeled. For example, when the complexity of the street scene data to be labeled is low, the UNet neural network model can be selected. This model consists of a first-part Encoder and a second-part Decoder. The first-part Encoder consists of two 3x3 convolutional layers (ReLU) plus a 2x2 maxpooling layer to form a downsampling model, providing four downsampling modules for feature extraction. The second-part Decoder consists of an upsampling convolutional layer (deconvolutional layer) + feature concatenation + two 3x3 convolutional layers (ReLU) repeatedly, for a total of four upsampling modules. This neural network has a relatively simple structure, wide applicability, and mature technology. When applied to process low-complexity street scene data in this application embodiment, it has the advantages of good accuracy, high efficiency, and ease of implementation.
[0042] When the amount of street view data to be labeled is large, and the data has relatively high complexity and processing accuracy, other semantic segmentation models such as DeepLab, PSPNet, RefinNet, and Fastnet can be selected as the semantic segmentation models in this application embodiment. Using these neural network models as semantic segmentation models for data processing ensures that the determined semantic segmentation model can provide relatively stable data processing capabilities to ensure the normal operation of the system, while reducing the hardware resource consumption requirements of the system to a certain extent, thereby saving hardware computing power.
[0043] When the amount of street view data to be labeled is very large, and the complexity and processing accuracy requirements are very high, and hardware computing power is not a concern, neural network models such as HRNet, OCRNet, SETR, and Transformer can be selected as semantic segmentation models for processing the street view data to be labeled. These neural network models have strong computing power, although they require a large amount of hardware processing resources, but they have good data processing results and high accuracy.
[0044] Specifically, in one implementation of this application, when considering the criteria for the amount, complexity, and processing accuracy of the street view data to be labeled, relevant evaluation parameter thresholds can be set. Based on these thresholds, the amount and complexity of the street view data to be labeled are determined. For example, if the amount of the street view data to be labeled is less than a first threshold and the complexity is less than a first threshold, the current complexity of the street view data to be labeled is determined to be low. Conversely, if the amount of the street view data to be labeled is greater than or equal to the first threshold but less than a second threshold, and the complexity is greater than or equal to a preset first threshold but less than a second threshold, the amount of street view data to be labeled is determined to be large and the complexity is high. And so on. Of course, other methods can also be used to determine the requirements for the amount, complexity, and processing accuracy of the street view data to be labeled; this application does not require such methods.
[0045] Specifically, this embodiment is merely an illustrative example of determining a semantic segmentation model based on street view data to be labeled, and does not imply that this application is limited to this. For example, in one implementation of this application, the semantic segmentation model can be determined according to the different types of street view data, such as classifying the street view data to be labeled into different categories of static street view data or dynamic street view data. This makes the semantic segmentation model used in this application more targeted when performing semantic segmentation and data labeling on different types of data, thereby improving the accuracy of semantic segmentation and data standards.
[0046] S102. Based on the determined semantic segmentation model, determine the self-supervised learning model.
[0047] This application first performs unsupervised training on a self-supervised model, and then further uses model porting to enable the determined untrained semantic segmentation model to directly acquire better semantic segmentation and data annotation capabilities. To facilitate this process, when determining the self-supervised learning model, the structure of the self-supervised learning model can be designed or determined based on the structure of the determined semantic segmentation model. For example, a structurally similar self-supervised learning model can be designed or determined based on the structure of the determined semantic segmentation model, thereby ensuring that the ported semantic segmentation model has good data processing capabilities while improving the ease of model porting.
[0048] Specifically, in one optional implementation scenario of this application embodiment, the self-supervised learning model includes a first front-end backbone network and a pre-task module: the first front-end backbone network includes feature extractors for extracting features from the input image, used to extract image information and generate corresponding feature maps; the pre-task module is used for unsupervised semantic extraction to supervise the training of the first front-end backbone network. This allows the self-supervised learning model used in this application embodiment to use unlabeled data as training samples, ensuring training accuracy while completing model training with a stable training process, and also reducing manual costs to a certain extent.
[0049] Correspondingly, the determined semantic segmentation model includes a second front-end backbone network and a predictor; the second front-end backbone network also includes a feature extractor, which is used to extract corresponding features from the input image, and the predictor is used to process the corresponding feature map to generate a semantic segmentation prediction map.
[0050] Preferably, in one embodiment of this application, determining the self-supervised learning model based on the determined semantic segmentation model includes: determining the structure of the first front-end backbone network based on the structure of the second front-end backbone network of the semantic segmentation model, and determining the structure of the self-supervised learning model based on the structure of the first front-end backbone network. In the application scenario of this application embodiment, since both the first and second front-end backbone networks are used to extract input image information to generate corresponding feature maps, when porting the model, this mainly involves porting this part. To facilitate porting, the structure of the first front-end backbone network in the self-supervised learning model can be determined or designed based on the structure of the second front-end backbone network of the determined semantic segmentation model. This allows for further designing or determining the structure of the self-supervised learning model to be used based on the structure of the first front-end backbone network, ensuring convenient model porting. For example, in one implementation of this application, the first front-end backbone network and the second front-end backbone network can be configured with the same encoding structure. Specifically, the feature extractors in the first and second front-end backbone networks have the same or similar encoding structures. This embodiment is merely illustrative and does not imply that this application is limited thereto.
[0051] In a practical application scenario of this application embodiment, specifically, the semantic segmentation model is SETR, a vision-based Transformer semantic segmentation model. The second front-end backbone network of SETR is a Transformer Layer feature extractor. When the model processes the input image data, it first divides the input image into blocks, then performs block embedding and adds position encoding to each image block, converting the image into a vector sequence. This sequence then passes through a Transformer block, which includes 24 Transformer layers. Each Transformer layer consists of MSA+MLP+Layer Norm+residual connections for processing. Simultaneously, when decoding the image, SETR converts the 2D encoder output vector into a 3D feature map. Through original upsampling, progressive upsampling, or multi-level feature summing, using upsampling processing similar to a feature pyramid network, data noise can be effectively eliminated, ensuring the accuracy of image data processing.
[0052] Correspondingly, when the semantic segmentation model SETR is determined to be the model for semantic segmentation data annotation, the supervised learning model can be identified as the self-supervised learning model MOCO (Momentum Contrast for Unsupervised Visual Representation Learning). The first front-end backbone network of MOCO includes encoderq and encoderrk. MOCO trains the visual representation encoder by matching the encoded query q with the dictionary of encoded keys using contrast loss, resulting in good and stable processing capabilities for image data processing.
[0053] Specifically, in the application scenarios of this application embodiment, the transformer layer in the semantic segmentation model SETR and the feature extractor, or feature extraction layer, composed of encoder q (encoder fq) and encoder rk (encoder fk) in the self-supervised model MOCO can both be referred to as the backbone network of their respective models, and the second front-end backbone network and the first front-end backbone network described in this application embodiment. In the actual use of this application embodiment, the transformer layer in the semantic segmentation model SETR is used as encoder q (encoder fq) and encoder rk (encoder fk) of the self-supervised learning model MOCO for pre-training. Since the semantic segmentation model SETR emphasizes its use of a transformer structure as a feature extraction layer, while the self-supervised learning model MOCO, as a general self-supervised learning framework, does not focus on the design of the feature extraction layer, it is directly referred to as the encoder, that is, the feature extraction layer of the self-supervised learning model MOCO, and also serves as the backbone network of the self-supervised learning model MOCO, that is, the second front-end backbone network described in this application embodiment. This pre-training process is easy to implement and has good pre-training effects.
[0054] Optionally, in other implementations of this application, the self-supervised model may also be a neural network model such as BYOL, SimCLR, or MAE, to adapt to the data processing computing power required in the process of processing the street scene data to be labeled, to ensure the stability and balance of the system operation, and to ensure the efficiency of the model training process while saving hardware resources to reduce costs.
[0055] S103. Use unlabeled street view data to train the self-supervised learning model to obtain a trained self-supervised learning model.
[0056] This application relates to semantic segmentation and data annotation application scenarios. By using unlabeled street view data as samples to train a self-supervised learning model, it can effectively reduce the cost of model training, improve efficiency, and also ensure the accuracy of model training.
[0057] S104. The feature extractor of the trained self-supervised learning model is ported to the semantic segmentation model to obtain the ported semantic segmentation model. In this embodiment, after the self-supervised learning model is trained using unlabeled data, the feature extractor of the trained self-supervised learning model also obtains the corresponding trained feature extraction capability. At this time, the feature extractor is directly ported to the determined semantic segmentation model, replacing the untrained feature extractor in the semantic segmentation model, thereby enabling the ported semantic segmentation model to obtain the corresponding feature extraction capability.
[0058] Optionally, in one embodiment of this application, the feature extractor of the trained self-supervised learning model is ported to the semantic segmentation model to obtain the ported semantic segmentation model. This includes: determining the weights of the feature extractor of the trained self-supervised learning model; replacing the second front-end backbone network of the semantic segmentation model according to the weights of the feature extractor; and determining the ported semantic segmentation model using the replaced second front-end backbone network. By replacing the weights, the porting process becomes simpler, more efficient, and easier to implement.
[0059] Optionally, in the embodiments of this application, when the structures of the first front-end backbone network and the second front-end backbone network are the same, the entire network can be replaced, such as directly using the second front-end backbone network to replace the first front-end backbone network in the semantic segmentation model, and the replaced semantic segmentation model is determined as the transplanted semantic segmentation model, thereby making the transplantation process simpler and more efficient.
[0060] S105. Data annotation is performed on the street view data to be annotated using the transplanted semantic segmentation model.
[0061] Optionally, in one embodiment of this application, before performing data annotation on the street view data to be annotated using the transplanted semantic segmentation model, the data annotation method further includes: fine-tuning the transplanted semantic segmentation model, that is, freezing some convolutional layers (usually most convolutional layers near the input) of the pre-trained model and training the remaining convolutional layers (usually some convolutional layers near the output) and fully connected layers, so as to ensure the accuracy of the data annotation results achieved by the transplanted semantic segmentation model in performing semantic segmentation on the input image data.
[0062] Optionally, in one implementation of this application embodiment, fine-tuning the ported semantic segmentation model includes: training the ported semantic segmentation model using labeled street view data, thereby fine-tuning the ported semantic segmentation model. This fine-tuning process is simple, reliable, accurate, requires relatively little labeled data, is efficient, and easy to implement.
[0063] This application provides a data annotation method, comprising: determining a semantic segmentation model based on street view data to be annotated; determining a self-supervised learning model based on the determined semantic segmentation model; training the self-supervised learning model using unlabeled street view data to obtain a trained self-supervised learning model; transplanting the feature extractor of the trained self-supervised learning model into the semantic segmentation model to obtain a transplanted semantic segmentation model; and annotating the street view data to be annotated using the transplanted semantic segmentation model. The data annotation method provided in this application, by setting up a semantic segmentation model and a self-supervised learning model, and training the self-supervised learning model using unlabeled data, and transplanting the feature extractor of the trained self-supervised learning model into the semantic segmentation model, effectively reduces the amount of labeled data used in model construction and training, as the training process of the self-supervised learning model does not require acquiring massive amounts of labeled sample data. This ensures the accuracy of data annotation using the semantic segmentation model while effectively guaranteeing annotation efficiency and reducing annotation costs.
[0064] Example 2:
[0065] Based on the data annotation method of the first aspect of this application, embodiments of this application also provide a data annotation apparatus, such as... Figure 2 As shown, Figure 2 This is a schematic diagram of the structure of a data annotation device 20 provided in Embodiment 2 of this application. The data annotation device 20 includes:
[0066] The first determining module 201 is used to determine the semantic segmentation model based on the street view data to be labeled;
[0067] The second determining module 202 is used to determine a self-supervised learning model based on the determined semantic segmentation model;
[0068] Training module 203 is used to train the self-supervised learning model using unlabeled street view data to obtain a trained self-supervised learning model.
[0069] The transplantation module 204 is used to transplant the feature extractor of the trained self-supervised learning model to the semantic segmentation model to obtain the transplanted semantic segmentation model.
[0070] The annotation module 205 is used to annotate the street view data to be annotated using the ported semantic segmentation model.
[0071] Optionally, in one implementation of this application embodiment, the self-supervised learning model includes a first front-end backbone network and a pre-task module:
[0072] The feature extractor is included in the first front-end backbone network, which is used to extract image information and generate corresponding feature maps.
[0073] The pre-task module is used for unsupervised semantic extraction to supervise the training of the first front-end backbone network.
[0074] Optionally, in one implementation of this application embodiment, the semantic segmentation model includes a second front-end backbone network and a predictor; the second front-end backbone network includes a feature extractor for extracting features from the input image, and the predictor is used to process the corresponding feature map to generate a semantic segmentation prediction map.
[0075] Optionally, in one implementation of this application embodiment, the second determining module 202 determines a self-supervised learning model based on the determined semantic segmentation model, including: determining the structure of the first front-end backbone network based on the structure of the second front-end backbone network of the semantic segmentation model; and determining the structure of the self-supervised learning model based on the structure of the first front-end backbone network, thereby determining the self-supervised model.
[0076] Optionally, in one implementation of this application embodiment, the transplantation module 204 transplants the feature extractor of the trained self-supervised learning model into the semantic segmentation model to obtain the transplanted semantic segmentation model, including:
[0077] Determine the weights of the feature extractor in the trained self-supervised learning model;
[0078] Based on the weights of the feature extractor, the second front-end backbone network of the semantic segmentation model is replaced, and the semantic segmentation model after the second front-end backbone network is replaced is determined as the ported semantic segmentation model.
[0079] Optionally, in one implementation of this application embodiment, the determined semantic segmentation model is the semantic segmentation model SETR; the second front-end backbone network of the semantic segmentation model SETR is a Transformer Layer feature extractor.
[0080] Correspondingly, in one implementation of the embodiments of this application, the self-supervised learning model is the self-supervised learning model MOCO; wherein, the first front-end backbone network of the self-supervised learning model MOCO includes a feature extractor composed of encoder q (encoder fq) and encoder rk (encoder fk).
[0081] Optionally, in one implementation of this application embodiment, the data annotation device further includes a fine-tuning module (not shown in the figures). The fine-tuning module is used to fine-tune the transplanted semantic segmentation model before performing data annotation on the street view data to be annotated using the transplanted semantic segmentation model, so as to improve the accuracy of the transplanted semantic segmentation model in performing semantic segmentation and achieving data annotation.
[0082] This application provides a data annotation device. It includes a first determining module for determining a semantic segmentation model based on street view data to be annotated; a second determining module for determining a self-supervised learning model based on the determined semantic segmentation model; a training module for training the self-supervised learning model using unlabeled street view data to obtain a trained self-supervised learning model; a transplantation module for transplanting the feature extractor of the trained self-supervised learning model into the semantic segmentation model to obtain a transplanted semantic segmentation model; and an annotation module for annotating the street view data to be annotated using the transplanted semantic segmentation model. The data annotation device provided in this application, by setting up a semantic segmentation model and a self-supervised learning model, and training the self-supervised learning model using unlabeled data, and then transplanting the feature extractor of the trained self-supervised learning model into the semantic segmentation model, reduces the amount of labeled sample data used in model construction and training, as the training process of the self-supervised learning model does not require acquiring massive amounts of labeled sample data. While ensuring the accuracy of data annotation using the semantic segmentation model, each module is highly targeted, effectively ensuring data annotation efficiency and reducing data annotation costs.
[0083] Example 3
[0084] Based on the data annotation method of Embodiment 1 of this application, this application also provides a storage medium storing a computer program thereon. When executed by a processor, the program implements the data annotation method as described in any of the above method embodiments of this application. The data annotation method includes, but is not limited to:
[0085] Determine the semantic segmentation model based on the street view data to be labeled;
[0086] Based on the established semantic segmentation model, determine the self-supervised learning model;
[0087] The self-supervised learning model is trained using unlabeled street view data to obtain a well-trained self-supervised learning model.
[0088] The feature extractor of the trained self-supervised learning model is transplanted into the semantic segmentation model to obtain the transplanted semantic segmentation model.
[0089] The transplanted semantic segmentation model is used to annotate the street view data to be labeled.
[0090] Specific embodiments of the subject matter have now been described. Other embodiments are within the scope of the appended claims. In some cases, the actions described in the claims can be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing can be advantageous.
[0091] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.
[0092] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.
[0093] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.
[0094] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.
[0095] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0096] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0097] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0098] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific transactions or implement specific abstract data types. This application can also be practiced in distributed computing environments where transactions are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.
[0099] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.
[0100] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.
Claims
1. A data annotation method, characterized in that, include: The semantic segmentation model to be selected is determined based on the quantity, complexity, processing accuracy, or type of the street view data to be labeled; the semantic segmentation model includes a second front-end backbone network and a predictor; the predictor is used to process the corresponding feature map to generate a semantic segmentation prediction map; The structure of the first front-end backbone network is determined based on the structure of the second front-end backbone network of the semantic segmentation model. Based on the structure of the first front-end backbone network, the structure of the self-supervised learning model is determined to define the self-supervised learning model. The self-supervised learning model includes a first front-end backbone network and a pre-task module. The first front-end backbone network includes a feature extractor for extracting features from the input image, used to extract image information and generate corresponding feature maps. The pre-task module is used for unsupervised semantic extraction to supervise the training of the first front-end backbone network. The semantic segmentation model also includes a second front-end backbone network and a predictor. The second front-end backbone network also includes a feature extractor for extracting corresponding features from the input image. The predictor processes the corresponding feature maps to generate a semantic segmentation prediction map. The feature extractors in the first and second front-end backbone networks have the same encoding structure. The self-supervised learning model is trained using unlabeled street view data to obtain a well-trained self-supervised learning model. The weights of the feature extractor of the trained self-supervised learning model are determined. Based on the weights of the feature extractor, the second front-end backbone network of the semantic segmentation model is replaced by weight replacement. The semantic segmentation model after the second front-end backbone network is replaced is the transplanted semantic segmentation model. The transplanted semantic segmentation model is used to annotate the street view data to be labeled.
2. The data annotation method according to claim 1, characterized in that, The semantic segmentation model is SETR. The second front-end backbone network of the semantic segmentation model SETR is a Transformer Layer feature extractor.
3. The data annotation method according to claim 1, characterized in that, The self-supervised learning model is the self-supervised learning model MOCO; The first front-end backbone network of the self-supervised learning model MOCO includes a feature extractor consisting of encoders fq and fk.
4. The data annotation method according to claim 1, characterized in that, Before labeling the street view data to be labeled using the transplanted semantic segmentation model, the method also includes fine-tuning the transplanted semantic segmentation model.
5. A data annotation device, characterized in that, include: The first determining module is used to determine the selected semantic segmentation model based on the quantity, complexity, processing accuracy, or type of the street view data to be labeled; the semantic segmentation model includes a second front-end backbone network and a predictor; the predictor is used to process the corresponding feature map to generate a semantic segmentation prediction map; The second determining module is used to determine the structure of the first front-end backbone network based on the structure of the second front-end backbone network of the semantic segmentation model. Based on the structure of the first front-end backbone network, the structure of the self-supervised learning model is determined to define the self-supervised learning model. The self-supervised learning model includes a first front-end backbone network and a pre-task module. The first front-end backbone network includes a feature extractor for extracting features from the input image, used to extract image information and generate corresponding feature maps. The pre-task module is used for unsupervised semantic extraction to supervise the training of the first front-end backbone network. The semantic segmentation model also includes a second front-end backbone network and a predictor. The second front-end backbone network also includes a feature extractor for extracting corresponding features from the input image. The predictor processes the corresponding feature maps to generate a semantic segmentation prediction map. The feature extractors in the first and second front-end backbone networks have the same encoding structure. A training module is used to train the self-supervised learning model using unlabeled street view data to obtain a trained self-supervised learning model. The transplantation module is used to determine the weights of the feature extractor of the trained self-supervised learning model. Based on the weights of the feature extractor, the second front-end backbone network of the semantic segmentation model is replaced by weight replacement. The semantic segmentation model after the second front-end backbone network is replaced is the transplanted semantic segmentation model. The annotation module is used to annotate the street view data to be annotated using the ported semantic segmentation model.
6. A storage medium having a computer program stored thereon, which, when executed by a processor, implements the data annotation method as claimed in any one of claims 1-4.