Data annotation method, device and equipment

Abstract

The invention provides a data annotation method, device and equipment, and relates to the technical field of cloud, the method comprises the following steps: receiving first to-be-annotated data sent by a user, the first to-be-annotated data being at least one of an image, a point cloud and a depth map; in a multi-modal basic model library, obtaining a target vertical domain multi-modal basic model corresponding to a business scene to which the first to-be-labeled data belongs; receiving a first downstream task selection message sent by the user, wherein the first downstream task selection message is used for indicating a first labeling type executed on the to-be-labeled data; in a downstream task model library, obtaining a target downstream task model which corresponds to a business scene to which the first to-be-labeled data belongs and is used for executing the first labeling type; and marking the first to-be-marked data based on the target vertical domain multi-modal basic model and the target downstream task model. According to the invention, automatic labeling of data can be realized.

Description

[0001] This application claims priority to Chinese patent application No. 202311334022.7, filed on October 13, 2023, and entitled “Methods, devices and apparatus for data annotation”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of cloud technology, and in particular to a method, device and equipment for data annotation. Background Art

[0003] With the development of artificial intelligence (AI) technology, AI has been widely used in various industries such as automobiles, robots, and medical care. In AI technology, various neural network models play a key role. Neural network models can realize functions such as object recognition, object segmentation, and scene understanding. In order to make the neural network model more accurate when realizing these functions, it is necessary to train the neural network model with labeled data in advance. For labeled data, it is usually obtained by manually labeling the original data. However, manual labeling is inefficient. Summary of the invention

[0004] The present application provides a method, apparatus and device for data annotation, which can automatically annotate data. The corresponding technical solutions are as follows:

[0005] In a first aspect, a method for data annotation is provided, which is applied to a cloud platform, wherein a multimodal basic model library and a downstream task model library are deployed on the cloud platform, wherein the multimodal basic model library stores multiple vertical multimodal basic models, each vertical multimodal basic model corresponds to a business scenario, and the downstream task model library stores multiple downstream task models corresponding to each business scenario for respectively executing multiple annotation types, wherein the method comprises:

[0006] Receive the first data to be annotated sent by the user, wherein the first data to be annotated is at least one of an image, a point cloud and a depth map. In the multimodal basic model library, obtain the target vertical domain multimodal basic model corresponding to the business scenario to which the first data to be annotated belongs. Receive the first downstream task selection message sent by the user, wherein the first downstream task selection message is used to indicate the first annotation type to be performed on the data to be annotated. In the downstream task model library, obtain the target downstream task model for executing the first annotation type corresponding to the business scenario to which the first data to be annotated belongs. Based on the target vertical domain multimodal basic model and the target downstream task model, annotate the first data to be annotated to obtain the annotated data corresponding to the first data to be annotated, wherein the target vertical domain multimodal basic model is used to extract features from the first data to be annotated, and the target downstream task model is used to output the annotated data corresponding to the first data to be annotated based on the features extracted by the target vertical domain multimodal basic model.

[0007] In the technical solution provided in the present application, the cloud platform is deployed with a multimodal basic model library and a downstream task model library. The multimodal basic model library stores multiple vertical domain multimodal basic models, each vertical domain multimodal basic model corresponds to a business scenario, and the downstream task model library stores multiple downstream task models corresponding to each of the above business scenarios for respectively executing multiple annotation types. The user can upload at least one of the images and point clouds to the cloud platform as the data to be annotated. The data to be annotated can be collected by the user through a real sensor or a virtual sensor. The user can also specify the annotation type, which can be two-dimensional (2-dimention, 2D) object recognition, 2D object segmentation, 2D scene understanding, 3D object recognition, 3D object segmentation, 3D scene understanding, etc. The cloud platform selects the target vertical domain multimodal basic model corresponding to the business scenario to which the data to be annotated belongs in the multimodal basic model library, and selects the target downstream task model corresponding to the business scenario to which the data to be annotated belongs in the downstream task model library for executing the user-specified annotation type. Then, the selected target multimodal basic model and target downstream task model are used to annotate the data to be annotated. This method can automatically label the data to be labeled uploaded by the user according to the user's labeling requirements, thereby improving the labeling efficiency.

[0008] In a possible implementation, before labeling the first to-be-labeled data based on the target vertical domain multimodal basic model and the target downstream task model, the method further includes:

[0009] Based on the first data to be labeled, the target vertical domain multimodal basic model is trained.

[0010] In the technical solution provided in the present application, before using the target vertical multimodal basic model, the first data to be labeled uploaded by the user can be used to label the target vertical multimodal basic model for unsupervised training. The training here can also be referred to as fine-tuning the target vertical multimodal basic model.

[0011] In a possible implementation, the method further includes:

[0012] Receive a labeled data type selection message sent by a user, perform semantic reconstruction on the first labeled data, obtain first labeled reconstructed data of the data type indicated by the labeled data type selection message, and send the first labeled reconstructed data to the user.

[0013] In the technical solution provided by the present application, the user may specify the type of annotation data, such as image, point cloud, depth map, etc. After obtaining the annotated data, the annotated reconstructed data of the annotated data type specified by the user may be obtained through semantic reconstruction and returned to the user.

[0014] In a possible implementation, the method further includes:

[0015] The first annotated data is semantically reconstructed to obtain second annotated reconstructed data, wherein the data type of the second annotated reconstructed data is the same as the data type of the annotated data. Based on the second annotated reconstructed data, the target vertical multimodal basic model and the target downstream task model are trained to obtain the trained vertical multimodal basic model and the trained downstream task model. The target vertical multimodal basic model in the multimodal basic model library is updated to the trained vertical multimodal basic model, and the target downstream task model in the downstream task model library is updated to the trained downstream task model.

[0016] In the technical solution provided in the present application, for labeled data, more labeled data can be generated through semantic reconstruction, and the generated more labeled data can be used to fine-tune and update the target vertical multimodal basic model and the target downstream task model, so that the target vertical multimodal basic model and the target downstream task model can more accurately label the business scenarios to which the labeled data belongs.

[0017] In a possible implementation, the user may specify a business scenario, and accordingly, the processing may be as follows:

[0018] A business scenario selection message sent by a user is received, and the business scenario indicated by the business scenario selection message is determined as the business scenario to which the first data to be labeled belongs.

[0019] In a possible implementation, the method further includes:

[0020] Receive second data to be annotated sent by a user, wherein the second data to be annotated is at least one of an image, a point cloud, and a depth map. Receive a second downstream task selection message sent by a user, wherein the second downstream task selection message is used to indicate a second annotation type to be performed on the second data to be annotated. Determine that the vertical multimodal basic model corresponding to the business scenario to which the second data to be annotated belongs is not stored in the multimodal basic model library. Obtain manually annotated data corresponding to part of the data in the second data to be annotated, wherein the manually annotated data is annotated according to the second annotation type. Perform semantic reconstruction on the manually annotated data to obtain third annotated reconstructed data. Send the third annotated reconstructed data to the user.

[0021] In the technical solution provided in the present application, if the cloud platform does not store the target vertical domain multimodal basic model corresponding to the business scenario to which the data to be labeled belongs, the data to be labeled can be manually labeled to obtain labeled data. The labeled data can be directly returned to the user, or more labeled data can be obtained after semantic reconstruction and returned to the user.

[0022] In a possible implementation, the multimodal basic model library further stores a comprehensive domain multimodal basic model, the downstream task model library further stores a plurality of comprehensive domain downstream task models for respectively performing a plurality of annotation types, and the method further includes:

[0023] Obtain a comprehensive domain multimodal basic model from the multimodal basic model library, and obtain a target comprehensive domain downstream task model for executing the second annotation type from the downstream task model library. Based on the third annotated reconstruction data, train the comprehensive domain multimodal basic model and the target comprehensive domain downstream task model to obtain the trained comprehensive domain multimodal basic model and the trained comprehensive domain downstream task model. The trained comprehensive domain multimodal basic model is used as the vertical domain multimodal basic model corresponding to the business scenario to which the second data to be annotated belongs, and is stored in the multimodal basic model library. The trained comprehensive domain downstream task model is used as the downstream task model for executing the second annotation type corresponding to the business scenario to which the second data to be annotated belongs.

[0024] In the technical solution provided in the present application, if the cloud platform does not store the target vertical domain multimodal basic model corresponding to the business scenario to which the data to be labeled belongs, then after obtaining the labeled data and semantically reconstructing it to obtain the labeled reconstructed data, the labeled reconstructed data can be used to train the comprehensive domain multimodal basic model and the target comprehensive domain downstream task model to obtain the vertical domain multimodal basic model corresponding to the business scenario to which the second data to be labeled belongs and the downstream task model for executing the second annotation type corresponding to the business scenario to which the second data to be labeled belongs. In this way, if a user subsequently uploads the data to be labeled for the business scenario to which the second data to be labeled belongs, the vertical domain multimodal basic model corresponding to the business scenario to which the second data to be labeled belongs and the downstream task model for executing the second annotation type corresponding to the business scenario to which the second data to be labeled belongs can be used directly for automatic annotation.

[0025] In a second aspect, a data annotation device is provided, the method is applied to a cloud platform, the cloud platform is deployed with a multimodal basic model library and a downstream task model library, the multimodal basic model library stores multiple vertical domain multimodal basic models, each vertical domain multimodal basic model corresponds to a business scenario, the downstream task model library stores multiple downstream task models corresponding to each business scenario for respectively executing multiple annotation types, the method includes:

[0026] A receiving module, configured to receive data to be annotated sent by a user, wherein the data to be annotated is at least one of an image, a point cloud, and a depth map; and receive a first downstream task selection message sent by the user, wherein the first downstream task selection message is used to indicate a first annotation type to be performed on the data to be annotated;

[0027] A labeling module, used to obtain, from the multimodal basic model library, a target vertical domain multimodal basic model corresponding to the business scenario to which the first data to be labeled belongs; and to obtain, from the downstream task model library, a target downstream task model for executing the first labeling type corresponding to the business scenario to which the first data to be labeled belongs;

[0028] The labeling module is also used to label the first data to be labeled based on the target vertical multimodal basic model and the target downstream task model to obtain labeled data corresponding to the first data to be labeled, wherein the target vertical multimodal basic model is used to extract features from the first data to be labeled, and the target downstream task model is used to output the labeled data corresponding to the first data to be labeled based on the features extracted by the target vertical multimodal basic model.

[0029] In a possible implementation, the device further includes a training module, which is used to:

[0030] Based on the first data to be labeled, the target vertical domain multimodal basic model is trained.

[0031] In a possible implementation, the receiving module is further configured to:

[0032] Receiving a marked data type selection message sent by the user;

[0033] The device also includes a reconstruction module, which is used to:

[0034] Performing semantic reconstruction on the first annotated data to obtain first annotated reconstructed data of the data type indicated by the annotated data type selection message;

[0035] The first labeled reconstruction data is sent to the user.

[0036] In a possible implementation manner, the device further includes a reconstruction module, which is used to:

[0037] Performing semantic reconstruction on the first annotated data to obtain second annotated reconstructed data, wherein a data type of the second annotated reconstructed data is the same as a data type of the annotated data;

[0038] The device also includes a training module, which is used to train the target vertical domain multimodal basic model and the target downstream task model based on the second labeled reconstruction data to obtain a trained vertical domain multimodal basic model and a trained downstream task model;

[0039] The device also includes an updating module, which is used to update the target vertical multimodal basic model in the multimodal basic model library to the trained vertical multimodal basic model; and to update the target downstream task model in the downstream task model library to the trained downstream task model.

[0040] In a possible implementation, the receiving module is further configured to:

[0041] Receive business scenario selection messages sent by users;

[0042] Determining the business scenario indicated by the business scenario selection message as the business scenario to which the first data to be labeled belongs;

[0043] The marking module is used for:

[0044] Determine whether the multimodal basic model library stores a target vertical domain multimodal basic model corresponding to the business scenario to which the first data to be labeled belongs.

[0045] In a possible implementation, the receiving module is further configured to:

[0046] Receiving second data to be annotated sent by a user, wherein the second data to be annotated is at least one of an image, a point cloud, and a depth map;

[0047] receiving a second downstream task selection message sent by the user, wherein the second downstream task selection message is used to indicate a second annotation type to be performed on the second data to be annotated;

[0048] The annotation module is also used for:

[0049] Determining that the vertical domain multimodal basic model corresponding to the business scenario to which the second data to be annotated belongs is not stored in the multimodal basic model library; obtaining manually annotated data corresponding to part of the second data to be annotated, wherein the manually annotated data is annotated according to the second annotation type;

[0050] The device also includes a reconstruction module, which is used to perform semantic reconstruction on the manually annotated data to obtain third annotated reconstructed data; and send the third annotated reconstructed data to the user.

[0051] In a possible implementation, the multimodal basic model library further stores a comprehensive domain multimodal basic model, the downstream task model library further stores a plurality of comprehensive domain downstream task models for respectively executing the plurality of annotation types, and the training module is further used to:

[0052] Acquire a comprehensive domain multimodal basic model from the multimodal basic model library;

[0053] Acquire a target comprehensive domain downstream task model for executing the second annotation type in the downstream task model library;

[0054] Based on the third annotated reconstruction data, the comprehensive domain multimodal basic model and the target comprehensive domain downstream task model are trained to obtain a trained comprehensive domain multimodal basic model and a trained comprehensive domain downstream task model;

[0055] The trained comprehensive domain multimodal basic model is used as the vertical domain multimodal basic model corresponding to the business scenario to which the second data to be annotated belongs, and is stored in the multimodal basic model library;

[0056] The trained comprehensive domain downstream task model is stored in a downstream task model library as a downstream task model for executing the second annotation type corresponding to the business scenario to which the second data to be annotated belongs.

[0057] In a third aspect, a computing device cluster is provided, the computing device cluster comprising at least one computing device, each computing device comprising a processor and a memory;

[0058] The processor of the at least one computing device is used to execute instructions stored in the memory of the device, so that the computing device cluster performs the data labeling method as described in the first aspect above.

[0059] In a fourth aspect, a computer program product comprising instructions is provided. When the instructions are executed by a computing device cluster, the computing device cluster executes the data labeling method as described in the first aspect.

[0060] In a fifth aspect, a computer-readable storage medium is provided, comprising computer program instructions. When the computer program instructions are executed by a computing device, the computing device executes the data labeling method as described in the first aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 is a schematic diagram of a data annotation system provided in an embodiment of the present application;

[0062] Figure 2 This is a flow chart of a method for data labeling provided in an embodiment of the present application;

[0063] Figure 3 is a schematic diagram of a grid structure of a multimodal basic model provided in an embodiment of the present application;

[0064] Figure 4 It is a schematic diagram of the structure of a data annotation device provided in an embodiment of the present application;

[0065] Figure 5 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0066] Figure 6 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0067] Figure 7 It is a structural diagram of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] To facilitate understanding of the embodiments of the present application, the data labeling method provided in the embodiments of the present application is described below in conjunction with the accompanying drawings. The embodiments of the present application provide a method for data labeling, which can be implemented by a cloud platform, and the cloud platform can be a computing device or a computing device cluster. The data labeling method provided in the embodiments of the present application can label images (image), point clouds (point clouds), depth maps (depth maps) and other labeled data uploaded by users to obtain labeled data. Furthermore, users can use labeled data to train neural network models used in business scenarios such as autonomous driving, robots, and inspection point marking, so that the trained neural network model has a better understanding of the environment.

[0069] See also Figure 1 , shows a schematic diagram of the system architecture of a data labeling method provided by an embodiment of the present application. Figure 1 As shown, in the method for data annotation provided in the embodiment of the present application, the cloud platform is deployed with a multimodal basic model library and a downstream task model library, and multiple vertical domain multimodal basic models are stored in the multimodal basic model library, each vertical domain multimodal basic model corresponds to a business scenario, and the downstream task model library stores multiple downstream task models corresponding to each of the above business scenarios for respectively executing multiple annotation types. The user can upload at least one of the images and point clouds to the cloud platform as the data to be annotated. The data to be annotated can be collected by the user through a real sensor or a virtual sensor, and the annotation type can also be specified. The annotation type can be two-dimensional (2-dimention, 2D) object recognition, 2D object segmentation, 2D scene understanding, 3D object recognition, 3D object segmentation, 3D scene understanding, etc. The cloud platform selects the target vertical domain multimodal basic model corresponding to the business scenario to which the data to be annotated belongs in the multimodal basic model library, and selects the target downstream task model corresponding to the business scenario to which the data to be annotated belongs in the downstream task model library for executing the user-specified annotation type. Then, the selected target multimodal basic model and target downstream task model are used to annotate the data to be annotated. In addition, the annotated data obtained by annotation can be semantically reconstructed, and the target multimodal basic model in the multimodal basic model library and the target downstream task model in the downstream task model library can be trained and updated based on the semantically reconstructed data.

[0070] See also Figure 2 , which shows a flow chart of a method for data annotation provided in an embodiment of the present application. The method can be implemented by a cloud platform, such as Figure 2 As shown, the processing of the method may include the following steps:

[0071] Step 201: The cloud platform receives the data to be labeled sent by the user.

[0072] The data to be annotated may be at least one of an image, a point cloud, and a depth map.

[0073] In implementation, users can collect at least one of the images and point clouds of target business scenes through real sensors, virtual sensors, etc. as data to be labeled according to their needs. The target business scenes can be residential areas, electricity, office buildings, open roads, ports, parks, exhibition halls, parks, etc. In addition, the data to be labeled collected by users can be time-series.

[0074] Then, the user can send the data to be labeled to the cloud platform through the user device.

[0075] Step 202: The cloud platform receives a downstream task selection message sent by a user, wherein the downstream task selection message is used to indicate a target labeling type to be performed on the data to be labeled.

[0076] In implementation, the user can specify the required annotation type through the user device according to actual business needs, such as 2D object recognition, 2D object segmentation, 2D scene understanding, 3D object recognition, 3D object segmentation, 3D scene understanding, etc. Correspondingly, the user device can send a downstream task selection message to the cloud platform, and the downstream task selection message can carry indication information of the target annotation type.

[0077] There are many ways for a user to specify the desired target annotation type through a user device, and several of the ways are described below as examples:

[0078] Method 1: The user device may display a label type list, which may include options of multiple label types, and the user may select an option of a desired target label type in the label type list.

[0079] Method 2: The user may input the desired description information of the target annotation type into the user device. The description information of the target annotation type may include description text, description voice, etc. corresponding to the target annotation type. For example, if the target annotation type is object recognition, the description information of the target annotation type may be the text "3D object recognition" or the voice corresponding to the text content of 3D object recognition.

[0080] Step 203: The cloud platform determines the target business scenario to which the data to be labeled belongs.

[0081] In implementation, in order to more accurately label the data to be labeled, the cloud platform can also determine the target business scenario to which the data to be labeled belongs. There are many ways to determine the target business scenario to which the data to be labeled belongs. The following are some examples of these methods:

[0082] Method 1: The user specifies the target business scenario to which the data to be labeled belongs.

[0083] In method 1, the user can specify the target business scenario to which the data to be annotated belongs, and accordingly, the user device can send a business scenario selection message to the cloud platform. The business scenario selection message carries the indication information of the target business scenario. After receiving the business scenario selection message, the cloud platform determines the target business scenario to which the annotated data belongs based on the indication information carried by the message.

[0084] There are many ways for users to specify the target business scenario to which the data to be labeled belongs. The following are some examples of these ways:

[0085] Method 1: The user device can display a business scenario list, in which at least one business scenario is recorded, and the at least one business scenario is a business scenario corresponding to at least one vertical multimodal basic model stored in the multimodal basic model library. If the target business scenario to which the data to be labeled belongs is in the business scenario list, the user can select the option of the target business scenario in the business scenario list. If the target business scenario to which the data to be labeled belongs is not in the business scenario list, the user can input description information of the target business scenario to which the data to be labeled belongs into the user device. The description information of the target business scenario may include descriptive text, descriptive voice, etc. corresponding to the target business scenario. For example, the target business scenario is an exhibition hall, and the information of the target business scenario can be the text "exhibition hall" or a voice with the corresponding text content being an exhibition hall.

[0086] Method 2: The user can input information of the target business scenario to which the data to be labeled belongs into the user device.

[0087] The user can input description information of the target business scenario to which the data to be labeled belongs into the user device. The specific description information can be the same as described in the above method 1, and will not be repeated here.

[0088] Method 2: The cloud platform identifies the target business scenario to which the data to be labeled belongs

[0089] In the second method, after receiving the data to be labeled uploaded by the user, the cloud platform can input the data to be labeled into the classification model, and the classification model outputs the target business scenario indication information. Then, the cloud platform uses the business scenario indicated by the target business scenario indication information as the target business scenario to which the data to be labeled belongs. Among them, the classification model can be a kind of neural network model.

[0090] Step 204: The cloud platform determines whether the target business scenario is in the business scenario list.

[0091] In implementation, the cloud platform can be deployed with a multimodal basic model library and a downstream task model library. The multimodal basic model library can store a vertical multimodal basic model corresponding to each business scenario in at least one business scenario. The vertical multimodal basic model is a type of neural network model. In contrast, the downstream task model library can store multiple downstream task models corresponding to each business scenario in the at least one business scenario. The multiple downstream task models corresponding to each business scenario are used to execute multiple types of annotations. In addition, the cloud platform can also record a business scenario list, which records at least one of the above business scenarios.

[0092] After the cloud platform determines the business scenario to which the data to be labeled belongs, it can query whether the target business scenario is recorded in the business scenario list.

[0093] Step 205: When the cloud platform determines that the target business scenario is recorded in the business scenario list, it obtains the target vertical domain multimodal basic model corresponding to the target business scenario in the multimodal basic model library, and obtains the target downstream task model for executing the target annotation type corresponding to the target business scenario in the downstream task model library.

[0094] In implementation, when the cloud platform determines that the target business scenario is recorded in the business scenario list, it can be determined that the target vertical domain multimodal basic model corresponding to the target business scenario is stored in the multimodal basic model library, and the target downstream task model for executing the target annotation type corresponding to the target business scenario is stored in the downstream task model library. Then, the cloud platform obtains the target vertical domain multimodal basic model in the multimodal basic model library and the target downstream task model in the downstream task model library.

[0095] Step 206: The cloud platform labels the data to be labeled based on the target vertical domain multimodal basic model and the target downstream task model to obtain labeled data corresponding to the data to be labeled.

[0096] In implementation, when the data to be labeled is an image, after obtaining the target vertical multimodal basic model, the cloud platform inputs the data to be labeled into the target vertical multimodal basic model, and the target vertical multimodal basic model extracts features from the data to be labeled and outputs a first feature vector. Then, the first feature vector is output to the target downstream task model, and the target downstream task model outputs the labeled data corresponding to the data to be labeled. In the case where the target downstream task model is a 2D object recognition model, the labeled data may be data obtained by identifying and labeling the categories of each object in the image. In the case where the target downstream task model is a 2D object segmentation model, the labeled data may be data obtained by pixel-level segmentation and labeling of different objects in the image. In the case where the target downstream task model is a 2D scene understanding model, the labeled data may be data obtained by scene understanding labeling of the image.

[0097] In the case where the data to be labeled is a point cloud, after obtaining the target vertical multimodal basic model, the cloud platform inputs the data to be labeled into the target vertical multimodal basic model in time sequence, and the target vertical multimodal basic model extracts features from the data to be labeled and outputs a second feature vector. Then, the second feature vector is output to the target downstream task model, and the target downstream task model outputs the labeled data corresponding to the data to be labeled. In the case where the target downstream task model is a 3D object recognition model, the labeled data may be data obtained by identifying and labeling the categories of each object in the point cloud. In the case where the target downstream task model is a 3D object segmentation model, the labeled data may be data obtained by segmenting and labeling different objects in the point cloud. In the case where the target downstream task model is a 3D scene understanding model, the labeled data may be data obtained by labeling the scene understanding in the point cloud.

[0098] In the case where the data to be annotated is an image and a point cloud, after obtaining the target vertical domain multimodal basic model, the cloud platform uses the image and the point cloud as two input data, inputs the target vertical domain multimodal basic model, and the target vertical domain multimodal basic model performs feature fusion on the image and the point cloud, and outputs the fused third feature vector. Then, the third feature vector is output to the target downstream task model, and the target downstream task model outputs the annotated data corresponding to the data to be annotated. In the case where the target downstream task model is a 2D object recognition model, the annotated data may be data obtained by identifying and annotating the categories of each object in the image. In the case where the target downstream task model is a 2D object segmentation model, the annotated data may be data obtained by pixel-level segmentation and annotation of different objects in the image. In the case where the target downstream task model is a 2D scene understanding model, the annotated data may be data obtained by scene understanding annotation of the image. In the case where the target downstream task model is a 3D object recognition model, the annotated data may be data obtained by identifying and annotating the categories of each object in the point cloud. In the case where the target downstream task model is a 3D object segmentation model, the annotated data may be data obtained by segmenting and annotating different objects in the point cloud. When the target downstream task model is a 3D scene understanding model, the labeled data may be data obtained by annotating scene understanding in the point cloud.

[0099] In addition, if the data to be labeled has a time sequence, when the data to be labeled is input into the target vertical domain multimodal basic model, the data to be labeled can also be input into the target vertical domain multimodal basic model according to the time sequence.

[0100] In a possible implementation, in addition to uploading the data to be annotated, the user can also send a target text to the cloud platform to describe the scene understanding requirements of the data to be annotated. Accordingly, the processing of the above step 206 can be: the cloud platform annotates the data to be annotated based on the target vertical domain multimodal basic model, the target downstream task model and the target text, and obtains the annotated data corresponding to the data to be annotated.

[0101] In this possible implementation, when the data to be annotated is an image, after acquiring the target vertical domain multimodal basic model, the cloud platform uses the data to be annotated and the target text as two input data, and inputs them into the target vertical domain multimodal basic model. The target vertical domain multimodal basic model performs feature extraction on the data to be annotated, outputs the fourth feature vector, and performs feature extraction on the target text, and outputs the fifth feature vector. Then, the fourth feature vector and the fifth feature vector are output to the target downstream task model, and the target downstream task model outputs the annotated data corresponding to the data to be annotated. When the target downstream task model is a 2D scene understanding model, the annotated data can be scene understanding annotations of the image according to the requirements indicated by the target text, for example, the target text is "what buildings are in the scene".

[0102] In the case where the data to be annotated is a point cloud, after obtaining the target vertical domain multimodal basic model, the cloud platform uses the data to be annotated and the target text as two input data, and inputs them into the target vertical domain multimodal basic model. The target vertical domain multimodal basic model performs feature extraction on the data to be annotated, outputs the sixth feature vector, and performs feature extraction on the target text, and outputs the seventh feature vector. Then, the sixth feature vector and the seventh feature vector are output to the target downstream task model, and the target downstream task model outputs the annotated data corresponding to the data to be annotated. In the case where the target downstream task model is a 3D scene understanding model, the annotated data can be scene understanding annotations of the point cloud according to the requirements indicated by the target text, for example, the target text is "what buildings are in the scene".

[0103] In the case where the data to be annotated are images and point clouds, after obtaining the target vertical domain multimodal basic model, the cloud platform uses the images, point clouds and target texts in the data to be annotated as three-way input data, and inputs them into the target vertical domain multimodal basic model. The target vertical domain multimodal basic model performs feature fusion on the image and point cloud, outputs the eighth feature vector, performs feature extraction on the target text, and outputs the ninth feature vector. Then, the eighth feature vector and the ninth feature vector are output to the target downstream task model, and the target downstream task model outputs the annotated data corresponding to the data to be annotated. In the case where the target downstream task model is a 2D scene understanding model, the annotated data can be the scene understanding annotation of the image according to the requirements indicated by the target text, for example, the target text is "what buildings are in the scene". In the case where the target downstream task model is a 2D scene understanding model, the annotated data can be the scene understanding annotation of the point cloud according to the requirements indicated by the target text, for example, the target text is "what buildings are in the scene".

[0104] In addition, the case where the data to be labeled is a depth map is processed in the same way as the case where the data to be labeled is a point cloud, and the case where the data to be labeled is an image and a depth map is processed in the same way as the case where the data to be labeled is an image and a point cloud, which will not be repeated here.

[0105] In a possible implementation, before using the target vertical domain multimodal basic model for labeling, the target vertical domain multimodal basic model can be unsupervisedly trained using the data to be labeled uploaded by the user. Accordingly, the target vertical domain multimodal basic model used for labeling in the above step 206 is unsupervisedly trained here.

[0106] In a possible implementation, the cloud platform may return the annotated data obtained after the annotation in step 206 to the user. Alternatively, the cloud platform may return the annotated data obtained after the annotation in step 206 to the user after manual annotation correction.

[0107] In a possible implementation, the user can also specify the annotation data type, and accordingly, the user device can send an annotation data type selection message to the cloud platform. After obtaining the annotated data, the cloud platform can first manually perform annotation correction on the annotated data, and then perform semantic reconstruction on the annotated data after annotation correction to obtain the second annotated reconstructed data of the annotated data type indicated by the annotated data type selection message. The annotated data type can be an annotated dense point cloud, an annotated single-frame point cloud, an annotated depth map, an annotated semantic grid, an annotated picture of any perspective, etc. After obtaining the second annotated reconstructed data, the cloud platform can also return the second annotated reconstructed data to the user.

[0108] In one possible implementation, after obtaining the labeled data, the cloud platform can first manually correct the annotations of the labeled data, and then semantically reconstruct the labeled data after the annotation correction to obtain third labeled reconstructed data of the same data type as the labeled data. Furthermore, the cloud platform can use the third labeled reconstructed data to train the target vertical multimodal basic model and the target downstream task model to obtain the trained vertical multimodal basic model and the trained downstream task model. Then, the cloud platform can update the target vertical multimodal basic model stored in the multimodal basic model library to the vertical multimodal basic model trained here, and update the target downstream task model stored in the downstream task model library to the downstream task model trained here.

[0109] Step 207: When the cloud platform determines that the target business scenario is not recorded in the business scenario list, it obtains the comprehensive domain multimodal basic model from the multimodal basic model library and obtains the comprehensive domain downstream task model for executing the target annotation type from the downstream task model library.

[0110] In implementation, a comprehensive domain downstream task model may also be stored in the multimodal basic model library, and a comprehensive domain downstream task model for executing each annotation type may also be stored in the downstream task model library.

[0111] When the cloud platform determines that the target business scenario is not recorded in the business scenario list, it can obtain the comprehensive domain multimodal basic model from the multimodal basic model library and obtain the target comprehensive domain downstream task model for executing the target annotation type from the downstream task model library.

[0112] In a possible implementation, after the comprehensive domain multimodal basic model is obtained, unsupervised training can be performed on the comprehensive domain multimodal basic model based on the data to be annotated uploaded by the user.

[0113] Step 208: The cloud platform obtains manually annotated data of all or part of the data to be annotated, and semantically reconstructs the manually annotated data to obtain first annotated reconstructed data.

[0114] In implementation, when the target vertical multimodal basic model corresponding to the target business scenario is not stored in the multimodal basic model library, the cloud platform can manually annotate all or part of the data to be annotated according to the user's annotation requirements to obtain manually annotated data, and then semantically reconstruct the manually annotated data to obtain the first annotated reconstructed data. The data type of the first annotated reconstructed data is the same as the data type of the annotated data output by the target downstream task model. For example, if the data type output by the downstream task model is an annotated image, the first annotated reconstructed data is also an annotated image. If the data type output by the downstream task model is an annotated point cloud, the first annotated reconstructed data is also an annotated point cloud.

[0115] In addition to manual annotation, in step 208, a 2D annotation model may be used to perform 2D annotation on all or part of the data to be annotated, and semantic reconstruction of the data obtained by 2D annotation.

[0116] After obtaining the first annotated reconstructed data, the cloud platform may return the obtained first annotated reconstructed data to the user.

[0117] In one possible implementation, when the user specifies the annotation data type, the cloud platform also performs semantic reconstruction on the 2D annotated data to obtain four annotated reconstructed data of the annotation data type indicated by the annotation data type selection message, and can return the fourth annotated reconstructed data to the user.

[0118] Step 209: The cloud platform trains the comprehensive domain multimodal basic model and the target comprehensive domain downstream task model based on the first annotated reconstruction data.

[0119] In implementation, after obtaining the first annotated reconstruction data in step 208, the first annotated reconstruction data can be used to train the comprehensive domain multimodal base model and the target downstream task model to obtain the trained multimodal base model and the trained comprehensive domain downstream task model.

[0120] In addition, if after the comprehensive domain multimodal basic model is obtained, the comprehensive domain multimodal basic model is unsupervisedly trained based on the data to be labeled uploaded by the user, then the comprehensive domain multimodal basic model trained in step 209 can be the one after the comprehensive domain multimodal basic model is unsupervisedly trained based on the data to be labeled uploaded by the user.

[0121] Step 210: The cloud platform stores the trained multimodal basic model as the vertical multimodal basic model corresponding to the target business scenario in the multimodal basic model library, and stores the trained comprehensive domain downstream task model as the downstream task model for executing the target labeling type corresponding to the target business scenario in the downstream task model library.

[0122] In implementation, after the comprehensive domain basic model and the target comprehensive domain downstream task model are trained using the first annotated reconstruction data, a save query message for the trained multimodal basic model and the trained comprehensive domain downstream task model can be sent to the user. If the user allows the trained multimodal basic model and the trained comprehensive domain downstream task model to be saved, a save permission message can be returned to the cloud platform through the user device. After receiving the save permission message sent by the user device, the cloud platform stores the trained multimodal basic model as the vertical domain multimodal basic model corresponding to the target business scenario in the multimodal basic model library, and stores the trained comprehensive domain downstream task model as the downstream task model for executing the target annotation type corresponding to the target business scenario in the downstream task model library.

[0123] In one possible implementation, see Figure 3 , shows a possible network architecture of a multimodal basic model, where the multimodal basic model can be a vertical domain multimodal basic model or a comprehensive domain multimodal basic model. In this network architecture, it includes a mask module, mask tokens, image encoder, voxel encoder, fuse attention (fuse attn.), spatial queries, tokenizer, and embedding layer. The network structure can have four inputs, which can be input as images, depth maps, point clouds, and texts respectively. When no text is input, the output of the multimodal basic model is the output of the tokenizer. When text is input, the output of the multimodal basic model is the output of fuseattn. and embedding. The output of this network structure is used as the input of the downstream task model.

[0124] The present application also provides a data annotation device, which may be a cloud platform, such as Figure 4 As shown, the device may include a receiving module 410 and a marking module 420, wherein:

[0125] The receiving module 410 is configured to receive data to be annotated sent by a user, wherein the data to be annotated is at least one of an image, a point cloud, and a depth map; and receive a first downstream task selection message sent by the user, wherein the first downstream task selection message is used to indicate a first annotation type to be performed on the data to be annotated.

[0126] The annotation module 420 is used to obtain, from the multimodal basic model library, a target vertical domain multimodal basic model corresponding to the business scenario to which the first data to be annotated belongs; and to obtain, from the downstream task model library, a target downstream task model for executing the first annotation type corresponding to the business scenario to which the first data to be annotated belongs;

[0127] The labeling module 420 is also used to label the first data to be labeled based on the target vertical multimodal basic model and the target downstream task model to obtain labeled data corresponding to the first data to be labeled, wherein the target vertical multimodal basic model is used to extract features from the first data to be labeled, and the target downstream task model is used to output the labeled data corresponding to the first data to be labeled based on the features extracted by the target vertical multimodal basic model.

[0128] In a possible implementation, the device further includes a training module, which is used to:

[0129] Based on the first data to be labeled, the target vertical domain multimodal basic model is trained.

[0130] In a possible implementation, the receiving module 410 is further configured to:

[0131] Receiving a marked data type selection message sent by the user;

[0132] The device also includes a reconstruction module, which is used to:

[0133] Performing semantic reconstruction on the first annotated data to obtain first annotated reconstructed data of the data type indicated by the annotated data type selection message;

[0134] The first labeled reconstruction data is sent to the user.

[0135] In a possible implementation manner, the device further includes a reconstruction module, which is used to:

[0136] Performing semantic reconstruction on the first annotated data to obtain second annotated reconstructed data, wherein a data type of the second annotated reconstructed data is the same as a data type of the annotated data;

[0137] The device also includes a training module, which is used to train the target vertical domain multimodal basic model and the target downstream task model based on the second labeled reconstruction data to obtain a trained vertical domain multimodal basic model and a trained downstream task model;

[0138] The device also includes an updating module, which is used to update the target vertical multimodal basic model in the multimodal basic model library to the trained vertical multimodal basic model; and to update the target downstream task model in the downstream task model library to the trained downstream task model.

[0139] In a possible implementation, the receiving module 410 is further configured to:

[0140] Receive business scenario selection messages sent by users;

[0141] Determining the business scenario indicated by the business scenario selection message as the business scenario to which the first data to be labeled belongs;

[0142] The marking module 420 is used to:

[0143] Determine whether the multimodal basic model library stores a target vertical domain multimodal basic model corresponding to the business scenario to which the first data to be labeled belongs.

[0144] In a possible implementation, the receiving module 410 is further configured to:

[0145] Receiving second data to be annotated sent by a user, wherein the second data to be annotated is at least one of an image, a point cloud, and a depth map;

[0146] receiving a second downstream task selection message sent by the user, wherein the second downstream task selection message is used to indicate a second annotation type to be performed on the second data to be annotated;

[0147] The marking module 420 is further used for:

[0148] Determining that the vertical domain multimodal basic model corresponding to the business scenario to which the second data to be annotated belongs is not stored in the multimodal basic model library; obtaining manually annotated data corresponding to part of the second data to be annotated, wherein the manually annotated data is annotated according to the second annotation type;

[0149] The device also includes a reconstruction module, which is used to perform semantic reconstruction on the manually annotated data to obtain third annotated reconstructed data; and send the third annotated reconstructed data to the user.

[0150] In a possible implementation, the multimodal basic model library further stores a comprehensive domain multimodal basic model, the downstream task model library further stores a plurality of comprehensive domain downstream task models for respectively executing the plurality of annotation types, and the training module is further used to:

[0151] Acquire a comprehensive domain multimodal basic model from the multimodal basic model library;

[0152] Acquire a target comprehensive domain downstream task model for executing the second annotation type in the downstream task model library;

[0153] Based on the third annotated reconstruction data, the comprehensive domain multimodal basic model and the target comprehensive domain downstream task model are trained to obtain a trained comprehensive domain multimodal basic model and a trained comprehensive domain downstream task model;

[0154] The trained comprehensive domain multimodal basic model is used as the vertical domain multimodal basic model corresponding to the business scenario to which the second data to be annotated belongs, and is stored in the multimodal basic model library;

[0155] The trained comprehensive domain downstream task model is stored in a downstream task model library as a downstream task model for executing the second annotation type corresponding to the business scenario to which the second data to be annotated belongs.

[0156] The above-mentioned receiving module 410 and marking module 420 can be implemented by software or hardware. As an example, the implementation of the receiving module 410 is introduced below. Similarly, the implementation of the marking module 420 can refer to the implementation of the receiving module 410.

[0157] As an example of a software functional unit, the receiving module 410 may include code running on a computing instance. The computing instance may be at least one of a physical host (computing device), a virtual machine, a container, and other computing devices. Furthermore, the computing device may be one or more. For example, the receiving module 410 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application may be distributed in the same region or in different regions. The multiple hosts / virtual machines / containers used to run the code may be distributed in the same AZ or in different AZs, each AZ including a data center or multiple data centers with close geographical locations. Typically, a region may include multiple AZs.

[0158] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same VPC or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway must be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0159] As an example of a hardware functional unit, the receiving module 410 may include at least one computing device, such as a server, etc. Alternatively, the receiving module 410 may also be a device implemented by ASIC or PLD, etc. The PLD may be implemented by CPLD, FPGA, GAL or any combination thereof.

[0160] The multiple computing devices included in the receiving module 410 can be distributed in the same region or in different regions. The multiple computing devices included in the receiving module 4100 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the management module 410 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0161] The present application also provides a computing device 100. Figure 5 As shown, the computing device 100 includes: a bus 102, a processor 104, a memory 106, and a communication interface 108. The processor 104, the memory 106, and the communication interface 108 communicate with each other through the bus 102. The computing device 100 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 100.

[0162] The bus 102 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The bus 102 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 102 may include a path for transmitting information between various components of the computing device 100 (eg, the memory 106, the processor 104, and the communication interface 108).

[0163] The processor 104 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0164] The memory 106 may include a volatile memory, such as a random access memory (RAM). The memory 106 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0165] The memory 106 stores executable codes, and the processor 104 executes the executable codes to respectively implement the functions of the aforementioned receiving module 410 and the labeling module 420, thereby implementing the data labeling method. That is, the memory 106 stores instructions for executing the data labeling method.

[0166] The communication interface 108 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 100 and other devices or a communication network.

[0167] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0168] like Figure 6 As shown, the computing device cluster includes at least one computing device 100. The memory 106 in one or more computing devices 100 in the computing device cluster may store the same instructions for executing the data labeling method.

[0169] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing the data labeling method. In other words, the combination of one or more computing devices 100 may jointly execute the instructions for executing the data labeling method.

[0170] It should be noted that the memory 106 in different computing devices 100 in the computing device cluster may store different instructions, which are respectively used to execute part of the functions of the data labeling method.

[0171] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing the data labeling method. In other words, the combination of one or more computing devices 100 may jointly execute the instructions for executing the data labeling method.

[0172] It should be noted that the memory 106 in different computing devices 100 in the computing device cluster may store different instructions for executing part of the functions of the data labeling method. That is, the instructions stored in the memory 106 in different computing devices 100 may implement the functions of one or more nodes in the receiving module 410 and the labeling module 420.

[0173] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Figure 7 A possible implementation is shown. Figure 7 As shown, two computing devices 100A and 100B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 106 in the computing device 100A stores instructions for executing the functions of the receiving module 410. At the same time, the memory 106 in the computing device 100B stores instructions for executing the functions of the marking module 420. It should be understood that Figure 7 The functions of the computing device 100A shown in FIG. 1 may also be completed by multiple computing devices 100. Similarly, the functions of the computing device 100B may also be completed by multiple computing devices 100.

[0174] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Figure 6 and Figure 7 The connection mode of the computing device cluster is different in that the memory 106 in one or more computing devices 100 in the computing device cluster may store the same instructions for executing the data labeling method.

[0175] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing the data labeling method. In other words, the combination of one or more computing devices 100 can jointly execute instructions for executing the method based on data labeling.

[0176] It should be noted that the memory 106 in different computing devices 100 in the computing device cluster can store different instructions for executing part of the functions of the access management device based on cloud computing technology. That is, the instructions stored in the memory 106 in different computing devices 100 can implement the functions of one or more nodes in the receiving module 410 and the marking module 420.

[0177] The present application also provides a computer program product including instructions. The computer program product may be a software or program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on a computing device, at least one computing device executes the data annotation method provided in the present application.

[0178] The present application embodiment also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the data labeling method provided in the present application embodiment.

[0179] In this application, the words such as "first", "second" and the like are used to distinguish the same or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first" and "second", nor is the quantity and execution order limited. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of various examples, the first eigenvector can be referred to as the second eigenvector, and similarly, the second eigenvector can be referred to as the first eigenvector. The first eigenvector and the second eigenvector can both be collectively referred to as eigenvectors, and in some cases, can be separate and different eigenvectors.

[0180] The term "at least one" in this application means one or more, and the term "plurality" in this application means two or more.

[0181] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for data annotation, characterized in that: The method is applied to a cloud platform, and the cloud platform is deployed with a multimodal basic model library and a downstream task model library. The multimodal basic model library stores multiple vertical domain multimodal basic models, each vertical domain multimodal basic model corresponds to a business scenario, and the downstream task model library stores multiple downstream task models corresponding to each business scenario for respectively executing multiple annotation types. The method includes: Receiving first data to be annotated sent by a user, wherein the first data to be annotated is at least one of an image, a point cloud, and a depth map; In the multimodal basic model library, a target vertical domain multimodal basic model corresponding to the business scenario to which the first data to be labeled belongs is obtained; receiving a first downstream task selection message sent by the user, wherein the first downstream task selection message is used to indicate a first annotation type to be performed on the data to be annotated; In the downstream task model library, a target downstream task model for executing the first annotation type corresponding to the business scenario to which the first to-be-annotated data belongs is obtained; Based on the target vertical multimodal basic model and the target downstream task model, the first data to be labeled is labeled to obtain labeled data corresponding to the first data to be labeled, wherein the target vertical multimodal basic model is used to extract features from the first data to be labeled, and the target downstream task model is used to output the labeled data corresponding to the first data to be labeled based on the features extracted by the target vertical multimodal basic model.

2. The method according to claim 1, characterized in that: The method further comprises: Receiving a marked data type selection message sent by the user; Performing semantic reconstruction on the first annotated data to obtain first annotated reconstructed data of the data type indicated by the annotated data type selection message; The first labeled reconstruction data is sent to the user.

3. The method according to claim 1, characterized in that The method further comprises: Performing semantic reconstruction on the first labeled data to obtain second labeled reconstructed data, wherein a data type of the second labeled reconstructed data is the same as a data type of the labeled data; Based on the second annotated reconstruction data, the target vertical domain multimodal basic model and the target downstream task model are trained to obtain a trained vertical domain multimodal basic model and a trained downstream task model; Updating the target vertical domain multimodal basic model in the multimodal basic model library to the trained vertical domain multimodal basic model; The target downstream task model in the downstream task model library is updated to the trained downstream task model.

4. The method according to any one of claims 1 to 3, characterized in that Before obtaining the target vertical domain multimodal basic model corresponding to the business scenario to which the first to-be-annotated data belongs, the method further includes: Receive business scenario selection messages sent by users; Determining the business scenario indicated by the business scenario selection message as the business scenario to which the first data to be labeled belongs; Determine whether the multimodal basic model library stores a target vertical domain multimodal basic model corresponding to the business scenario to which the first data to be labeled belongs.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Receiving second data to be annotated sent by a user, wherein the second data to be annotated is at least one of an image, a point cloud, and a depth map; receiving a second downstream task selection message sent by the user, wherein the second downstream task selection message is used to indicate a second annotation type to be performed on the second data to be annotated; Determining that the vertical domain multimodal basic model corresponding to the business scenario to which the second data to be annotated belongs is not stored in the multimodal basic model library; Acquire manually annotated data corresponding to part of the second data to be annotated, wherein the manually annotated data is annotated according to the second annotation type; Performing semantic reconstruction on the manually annotated data to obtain third annotated reconstructed data; The third labeled reconstruction data is sent to the user.

6. The method according to claim 5, characterized in that The multimodal basic model library also stores a comprehensive domain multimodal basic model, and the downstream task model library also stores a plurality of comprehensive domain downstream task models for respectively executing the plurality of annotation types. The method further includes: Acquire a comprehensive domain multimodal basic model from the multimodal basic model library; Acquire a target comprehensive domain downstream task model for executing the second annotation type in the downstream task model library; Based on the third annotated reconstruction data, the comprehensive domain multimodal basic model and the target comprehensive domain downstream task model are trained to obtain a trained comprehensive domain multimodal basic model and a trained comprehensive domain downstream task model; The trained comprehensive domain multimodal basic model is used as the vertical domain multimodal basic model corresponding to the business scenario to which the second data to be annotated belongs, and is stored in the multimodal basic model library; The trained comprehensive domain downstream task model is stored in the downstream task model library as a downstream task model for executing the second annotation type corresponding to the business scenario to which the second data to be annotated belongs.

7. A data labeling device, characterized in that: The method is applied to a cloud platform, and the cloud platform is deployed with a multimodal basic model library and a downstream task model library. The multimodal basic model library stores multiple vertical domain multimodal basic models, each vertical domain multimodal basic model corresponds to a business scenario, and the downstream task model library stores multiple downstream task models corresponding to each business scenario for respectively executing multiple annotation types. The method includes: A receiving module, configured to receive data to be annotated sent by a user, wherein the data to be annotated is at least one of an image, a point cloud, and a depth map; and receive a first downstream task selection message sent by the user, wherein the first downstream task selection message is used to indicate a first annotation type to be performed on the data to be annotated; A labeling module, used to obtain, from the multimodal basic model library, a target vertical domain multimodal basic model corresponding to the business scenario to which the first data to be labeled belongs; and to obtain, from the downstream task model library, a target downstream task model for executing the first labeling type corresponding to the business scenario to which the first data to be labeled belongs; The labeling module is also used to label the first data to be labeled based on the target vertical multimodal basic model and the target downstream task model to obtain labeled data corresponding to the first data to be labeled, wherein the target vertical multimodal basic model is used to extract features from the first data to be labeled, and the target downstream task model is used to output the labeled data corresponding to the first data to be labeled based on the features extracted by the target vertical multimodal basic model.

8. The device according to claim 7, characterized in that The receiving module is further used for: Receiving a marked data type selection message sent by the user; The device also includes a reconstruction module, which is used to: Performing semantic reconstruction on the first annotated data to obtain first annotated reconstructed data of the data type indicated by the annotated data type selection message; The first labeled reconstruction data is sent to the user.

9. The device according to claim 7, characterized in that The device also includes a reconstruction module, which is used to: Performing semantic reconstruction on the first labeled data to obtain second labeled reconstructed data, wherein a data type of the second labeled reconstructed data is the same as a data type of the labeled data; The device also includes a training module, which is used to train the target vertical domain multimodal basic model and the target downstream task model based on the second labeled reconstruction data to obtain a trained vertical domain multimodal basic model and a trained downstream task model; The device also includes an updating module, which is used to update the target vertical multimodal basic model in the multimodal basic model library to the trained vertical multimodal basic model; and to update the target downstream task model in the downstream task model library to the trained downstream task model.

10. The device according to any one of claims 7 to 9, characterized in that: The receiving module is further used for: Receive business scenario selection messages sent by users; Determining the business scenario indicated by the business scenario selection message as the business scenario to which the first data to be labeled belongs; The marking module is used for: Determine whether the multimodal basic model library stores a target vertical domain multimodal basic model corresponding to the business scenario to which the first data to be labeled belongs.

11. The device according to any one of claims 7 to 10, characterized in that The receiving module is further used for: Receiving second data to be annotated sent by a user, wherein the second data to be annotated is at least one of an image, a point cloud, and a depth map; receiving a second downstream task selection message sent by the user, wherein the second downstream task selection message is used to indicate a second annotation type to be performed on the second data to be annotated; The annotation module is also used for: Determining that the vertical domain multimodal basic model corresponding to the business scenario to which the second data to be annotated belongs is not stored in the multimodal basic model library; obtaining manually annotated data corresponding to part of the second data to be annotated, wherein the manually annotated data is annotated according to the second annotation type; The device also includes a reconstruction module, which is used to perform semantic reconstruction on the manually annotated data to obtain third annotated reconstructed data; and send the third annotated reconstructed data to the user.

12. The device according to claim 11, characterized in that The multimodal basic model library also stores a comprehensive domain multimodal basic model, the downstream task model library also stores a plurality of comprehensive domain downstream task models for respectively executing the plurality of annotation types, and the training module is further used to: Acquire a comprehensive domain multimodal basic model from the multimodal basic model library; Acquire a target comprehensive domain downstream task model for executing the second annotation type in the downstream task model library; Based on the third annotated reconstruction data, the comprehensive domain multimodal basic model and the target comprehensive domain downstream task model are trained to obtain a trained comprehensive domain multimodal basic model and a trained comprehensive domain downstream task model; The trained comprehensive domain multimodal basic model is used as the vertical domain multimodal basic model corresponding to the business scenario to which the second data to be annotated belongs, and is stored in the multimodal basic model library; The trained comprehensive domain downstream task model is stored in the downstream task model library as a downstream task model for executing the second annotation type corresponding to the business scenario to which the second data to be annotated belongs.

13. A computing device cluster, characterized in that: The computing device cluster includes at least one computing device, each computing device including a processor and a memory; The processor of the at least one computing device is used to execute instructions stored in the memory of the device, so that the computing device cluster executes the data labeling method according to any one of claims 1 to 6.

14. A computer program product comprising instructions, characterized in that When the instruction is executed by a computing device cluster, the computing device cluster executes the data labeling method according to any one of claims 1 to 6.

15. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by an edge, the edge performs the data labeling method as described in any one of claims 1 to 6.

Images