A data labeling method and system, an electronic device, and a storage medium

Abstract

The application provides a data labeling method and system, an electronic device and a storage medium. First, the original data is obtained, and the original data is manually labeled to obtain a first labeled data set. Then, the first labeled data set is input into a preset neural network model for training, and after the training parameters converge, a trained neural network model is output. Finally, the trained neural network model is used to automatically label the original data to obtain a second labeled data set. The application can realize training model feedback automatic labeling, and can integrate different recognition training models into a multi-type automatic labeling model through a fusion matrix, so as to realize one-time labeling of multiple types. The application can start model training through small batches of data, thereby saving time and resource costs. The application can also realize forward circulation iteration of the training model to the automatic labeling model, and the automatic labeling model can be updated synchronously after the training model is iterated.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and autonomous driving technology, specifically to a data annotation method, system, electronic device, and storage medium. Background Technology

[0002] The development of autonomous driving technology requires a massive amount of data, and this amount increases exponentially with the advancement of autonomous driving levels. Autonomous driving algorithm development necessitates multiple incremental training iterations to improve the performance of the on-vehicle algorithms. This training utilizes data from various sources, including speech, NLP (Natural Language Processing) text, video, images, and radar point cloud data. After the raw data is collected and cleaned, it requires thorough annotation. The development of autonomous driving algorithms relies heavily on feeding these labeled, high-quality data into the algorithms to achieve optimal training results.

[0003] Traditional methods for automated annotation of autonomous driving data primarily rely on self-built annotation models. This approach demands sophisticated algorithms, which most companies cannot afford to invest significant resources in developing. Furthermore, these automated annotation models require extensive training on large datasets to achieve optimal results, further increasing the training workload. For autonomous driving R&D companies, substantial training resources are prioritized for driving perception and control algorithms, leaving little room for dedicated training of automated annotation algorithms. Secondly, autonomous driving algorithms require diverse data types for annotation. Automated annotation necessitates feeding different types of data into these algorithms to ensure comprehensive training data coverage; otherwise, low recognition rates and extensive manual rework will occur. Therefore, it is unlikely that dedicated research into automated annotation algorithms by autonomous driving R&D companies will yield the desired results in the short term. Long-term algorithm development and validation with large datasets are essential; otherwise, the automated annotation results from the model will fail to improve annotation efficiency.

[0004] Therefore, how to balance data security and annotation efficiency, that is, how to quickly annotate data in a relatively safe and controllable environment to meet the data volume requirements of algorithm iteration, has become the core demand of autonomous driving algorithm R&D companies. Summary of the Invention

[0005] In view of the shortcomings of the prior art described above, this application provides a data annotation method, system, electronic device and storage medium to solve the above technical problems.

[0006] This application provides a data annotation method, which includes the following steps:

[0007] Obtain the raw data and manually annotate the raw data to obtain the first annotated dataset;

[0008] The first labeled dataset is input into a preset neural network model for training, and the trained neural network model is output after the training parameters converge.

[0009] The original data is automatically labeled using the trained neural network model to obtain a second labeled dataset.

[0010] In one embodiment of this application, the process of automatically labeling the original data using a trained neural network model includes:

[0011] The trained neural network model is designated as the first data labeling model.

[0012] Obtain the vector of the first data annotation model, and fuse the vector of the first data annotation model with a preset fusion matrix to obtain the second data annotation model;

[0013] The original data is automatically labeled using the second data labeling model to obtain the second labeled dataset.

[0014] In one embodiment of this application, after obtaining the second data annotation model, the method further includes:

[0015] The second data annotation model is iteratively trained using the first labeled dataset, and the data annotation iterative model is output after the training parameters converge.

[0016] The original data is automatically labeled using the data labeling iterative model to obtain the second labeled dataset.

[0017] In one embodiment of this application, the raw data includes at least one of the following: images, videos, radar point clouds, text, and voice.

[0018] This application also provides a data annotation system, the system comprising:

[0019] The data acquisition module is used to acquire raw data;

[0020] The manual annotation module is used to manually annotate the original data to obtain the first annotated dataset;

[0021] The model training module is used to input the first labeled dataset into a preset neural network model for training, and output the trained neural network model after the training parameters converge.

[0022] An automatic annotation module is used to automatically annotate the original data using a trained neural network model to obtain a second annotated dataset.

[0023] In one embodiment of this application, the automatic annotation module automatically annotates the original data using a trained neural network model, comprising:

[0024] The trained neural network model is designated as the first data labeling model.

[0025] Obtain the vector of the first data annotation model, and fuse the vector of the first data annotation model with a preset fusion matrix to obtain the second data annotation model;

[0026] The original data is automatically labeled using the second data labeling model to obtain the second labeled dataset.

[0027] In one embodiment of this application, after obtaining the second data annotation model, the system further includes:

[0028] The second data annotation model is iteratively trained using the first labeled dataset, and the data annotation iterative model is output after the training parameters converge.

[0029] The original data is automatically labeled using the data labeling iterative model to obtain the second labeled dataset.

[0030] In one embodiment of this application, the raw data includes at least one of the following: images, videos, radar point clouds, text, and voice.

[0031] This application also provides an electronic device, the electronic device comprising:

[0032] One or more processors;

[0033] A storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to implement the data annotation method as described in any of the preceding descriptions.

[0034] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer's processor, causes the computer to perform the data annotation method as described in any of the above-described methods.

[0035] As described above, this application provides a data annotation method, system, electronic device, and storage medium, which have the following beneficial effects:

[0036] This application first acquires raw data and manually annotates it to obtain a first labeled dataset. Then, it inputs this first labeled dataset into a pre-set neural network model for training. After the training parameters converge, it outputs the trained neural network model. Finally, it uses the trained neural network model to automatically annotate the raw data, obtaining a second labeled dataset. Therefore, this application can achieve automatic annotation by feeding the trained model back into the automatic annotation process. Through a fusion matrix, different targeted recognition training models can be integrated into a multi-type automatic annotation model, achieving the goal of annotating multiple types of data at once. Compared to traditional automatic annotation methods, this application can start model training with a small batch of data, and release automatic annotations once the model parameters are stable, saving time and resource costs. Secondly, this application, through a fusion matrix, can integrate different recognition algorithm models into a multi-type annotation model, annotating multiple targets or format types at once without repeating the automatic annotation process multiple times. Furthermore, this application can also achieve a positive loop iteration from the training model to the automatic annotation model; as the training model iterates, the automatic annotation model is updated synchronously. Furthermore, this application develops a model for target recognition, where the labeled object is the perceived target. The recognition result of the perception model is used in reverse as the labeling result of the annotation model, achieving multiple uses for a single model. Simultaneously, in this application, models whose accuracy does not meet the requirements of the training model can also be integrated into the automatic annotation system. The automatic annotation model iterates along with the training model, and after each iteration, it is automatically pushed to the annotation model. This eliminates the need for separate training of the annotation model, enabling parameter iteration of the annotation model.

[0037] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0038] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0039] Figure 1 This is a schematic diagram illustrating an exemplary system architecture that applies the technical solutions in one or more embodiments of this application;

[0040] Figure 2 A schematic flowchart illustrating a data annotation method provided in one embodiment of this application;

[0041] Figure 3 A logical diagram illustrating automatic data annotation provided in one embodiment of this application;

[0042] Figure 4 A flowchart illustrating a data annotation method provided in another embodiment of this application;

[0043] Figure 5 This is a schematic diagram of the hardware structure of a data annotation system provided in one embodiment of this application;

[0044] Figure 6 This is a schematic diagram of the hardware structure of an electronic device suitable for implementing one or more embodiments of this application. Detailed Implementation

[0045] The embodiments of this application will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be understood that the preferred embodiments are only for illustrating this application and are not intended to limit the scope of protection of this application.

[0046] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0047] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the present application. However, it will be apparent to those skilled in the art that embodiments of the present application may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the present application.

[0048] Figure 1 A schematic diagram of an exemplary system architecture that can apply the technical solutions of one or more embodiments of this application is shown. Figure 1 As shown, the system architecture 100 may include terminal device 110, network 120, and server 130. Terminal device 110 may include various electronic devices such as smartphones, tablets, laptops, and desktop computers. Server 130 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. Network 120 may be a communication medium of various connection types capable of providing a communication link between terminal device 110 and server 130, such as a wired communication link or a wireless communication link.

[0049] Depending on the implementation requirements, the system architecture in this application embodiment can have any number of terminal devices, networks, and servers. For example, server 130 can be a server group composed of multiple server devices. In addition, the technical solutions provided in this application embodiment can be applied to terminal device 110, or to server 130, or can be implemented jointly by terminal device 110 and server 130. This application does not impose any special limitations on this.

[0050] In one embodiment of this application, the terminal device 110 or server 130 can first acquire the raw data and manually annotate it to obtain a first annotated dataset; then, the first annotated dataset is input into a preset neural network model for training, and after the training parameters converge, the trained neural network model is output; finally, the trained neural network model is used to automatically annotate the raw data to obtain a second annotated dataset. By using the terminal device 110 or server 130 to execute the data annotation method, the training model can be fed back into the automatic annotation. Through a fusion matrix, different targeted recognition training models can be integrated into a multi-type automatic annotation model, achieving the goal of annotating multiple types of data at once. Compared to traditional automatic annotation modes, model training can be started with a small batch of data, and automatic annotation can be released after the model parameters stabilize, saving time and resource costs. Secondly, through the fusion matrix, different recognition algorithm models can be integrated into a multi-type annotation model, annotating multiple targets or format types at once without repeating the automatic annotation process multiple times. Furthermore, it also enables a positive loop iteration from the training model to the automatic annotation model; as the training model iterates, the automatic annotation model is updated synchronously.

[0051] The above sections introduced an exemplary system architecture that applies the technical solution of this application. Next, we will continue to introduce the data annotation method of this application.

[0052] Figure 2 A schematic flowchart of a data annotation method provided in an embodiment of this application is shown. Specifically, in an exemplary embodiment, as follows... Figure 2 As shown, this embodiment provides a data annotation method, which includes the following steps:

[0053] S210, Obtain the raw data and manually annotate the raw data to obtain the first annotated dataset. As an example, in this embodiment, the raw data includes, but is not limited to, data types such as images, videos, radar point clouds, text, and voice.

[0054] S220, the first labeled dataset is input into a preset neural network model for training, and the trained neural network model is output after the training parameters converge.

[0055] S230, The original data is automatically labeled using the trained neural network model to obtain the second labeled dataset.

[0056] Therefore, this embodiment can achieve automatic annotation by feeding back the training model. Through the fusion matrix, different targeted recognition training models can be integrated into a multi-type automatic annotation model, achieving the goal of annotating multiple types of data simultaneously. Compared to traditional automatic annotation methods, this embodiment can start model training with small batches of data, and automatically annotate once the model parameters are stable, saving time and resource costs. Secondly, this embodiment can integrate different recognition algorithm models into a multi-type annotation model through the fusion matrix, annotating multiple targets or format types at once without repeating the automatic annotation process multiple times. Furthermore, this embodiment can also achieve a positive loop iteration from the training model to the automatic annotation model; as the training model iterates, the automatic annotation model is updated synchronously.

[0057] In an exemplary embodiment, the process of automatically labeling the original data using a trained neural network model includes: designating the trained neural network model as a first data labeling model; obtaining the vector of the first data labeling model and fusing the vector of the first data labeling model with a preset fusion matrix to obtain a second data labeling model; and automatically labeling the original data using the second data labeling model to obtain a second labeled dataset. Specifically, the published trained models f(x) and g(x) need to generate a labeling model consistent with the labeling type requirements through the fusion matrix F to achieve data matching consistent with the original data labeling format; then:

[0058] Result=Ψ[f(x),g(x),...]·F (1);

[0059] Q*=R→Ψ[f(x), g(x),...]·F (2);

[0060] Where Ψ[f(x), g(x), ...] represents the training model vector, F represents the fusion matrix, and Q* is the automatic labeling result after inputting the original data R.

[0061] In an exemplary embodiment, after obtaining the second data annotation model, the method further includes: iteratively training the second data annotation model using the first annotation dataset, and outputting a data annotation iterative model after the training parameters converge; and automatically annotating the original data using the data annotation iterative model to obtain the second annotation dataset. Specifically, the second annotation model can perform preliminary automatic annotation of the original data after integration. However, due to the low accuracy of the second annotation model, manual annotation adjustment may still be necessary. That is, the second data annotation model can be iteratively trained one or more times using the original data until the training parameters converge after the last iteration, outputting a data annotation iterative model with the required accuracy. Then, the output data annotation model is used to automatically annotate the original data to obtain a dataset with the required annotation accuracy. In this embodiment, the accuracy of the automatic annotation model improves with each iteration, and the purpose of this improvement is to maximize the annotation capacity Q*.

[0062] like Figure 3 As shown in another exemplary embodiment, this application also provides a method for automatically labeled data based on feedback from a trained model, comprising the following steps:

[0063] Input of data to be labeled: The raw data R is directly input for manual labeling. The raw data includes data types such as images, videos, radar point clouds, text, and voice.

[0064] Manual annotation: The input data enters the manual annotation stage. Humans perform preliminary cleaning of the elements in the original data, remove static and duplicate data, and then annotate them. The annotation types include images, videos, radar point clouds, text, and voice. After the annotation is submitted, an annotated dataset is formed.

[0065] The labeled valuable data is stored in the database to form a small dataset (Dataset(q1, q2, ...)) for model training.

[0066] Data feeding algorithm training: After receiving the dataset Dataset(q1, q2, ...), start model training, determine whether the training parameters are stable, if not, repeat the first two steps of training, and after the parameters are basically stable, they can be packaged and output; there are: Dataset(q1) → f(x), Dataset(q2) → g(x), ...;

[0067] Model deployment: Once the model training is stable, it can be deployed. The deployed model can be stored in the model management system as a model file package. The model management system can automatically load the trained models f(x), g(x), etc., into the automatic annotation module.

[0068] Automatic annotation integration: After the training models f(x), g(x), etc. are released, they need to generate an annotation model that is consistent with the annotation type requirements through the fusion matrix F, so as to achieve data matching with the original data annotation format.

[0069] Result=Ψ[f(x),g(x),...]·F (1);

[0070] Q*=R→Ψ[f(x), g(x),...]·F (2);

[0071] Where Ψ[f(x), g(x), ...] represents the training model vector, F represents the fusion matrix, and Q* is the automatic labeling result after inputting the original data R.

[0072] Therefore, this embodiment integrates a training model with stable parameters into the automatic annotation module, enabling automatic annotation of corresponding target types. Specifically, this embodiment first manually annotates valuable training data, forming a labeled dataset Dataset(q1, q2, ...). Then, it starts model training using Dataset(q1, q2, ...). After the training parameters stabilize, it outputs training model files f(x), g(x), etc. These files are automatically converted and integrated into the automatic annotation module, which then automatically annotates the original data for targets of the same type, ultimately efficiently producing a labeled dataset Q*. Essentially, this embodiment allows the training model to feed back into the automatic annotation process. Through a fusion matrix, different targeted recognition training models can be integrated into a multi-type automatic annotation model, achieving the goal of simultaneous annotation of multiple types of data.

[0073] like Figure 4 As shown in an exemplary embodiment of this application, a data annotation method is also provided, including the following steps:

[0074] S1: Collect 1000 original front-view images and label the car and pedestrian elements based on the collected original images. The purpose of manual labeling is only to inject the original training set into the algorithm to obtain stable model parameters. As the accuracy of the automatic labeling model increases, the proportion of manual labeling will gradually decrease, and may even be completely replaced.

[0075] S2: After manual annotation is completed, the data is classified and filtered to select the dataset that meets the conditions. The filtering process involves cleaning and deleting duplicate scene data, such as car target (sedan, SUV, truck, bus, construction vehicle, etc.) image q1 and pedestrian sign image q2 at intersection, forming the training dataset Dataset(q1, q2).

[0076] S3: Inject the manually annotated 2D bounding box JSON results of the manually labeled car and pedestrian dataset images into the perception algorithm for pedestrian and car detection and recognition to start training. If the parameters are unstable, the amount of labeled data can be increased to ensure that the parameters converge.

[0077] S4: The trained model after parameter convergence can be packaged and released, q1—>f(x1), q2—>g(x1), where f(x1) and g(x1) are the output models after the first training.

[0078] S5: The first training model is automatically integrated. The integration process requires fusing the pedestrian and car detection and recognition models. The fusion result is Result1 = Ψ[f(x1), g(x1)]·F. When adding a third training model, the fusion matrix needs to be expanded. After fusion, the result is integrated into the automatic annotation module to achieve the automatic annotation function.

[0079] After the automatic annotation model is integrated, preliminary automatic annotation can be achieved. However, due to the low accuracy of the model, manual annotation and adjustment may still be necessary. After adjustment, the S2 process is repeated to form q1' and q2', the valuable labeled data, which are used to train the model iteratively. The S4 process is repeated: q1' → f(x1'), q2' → g(x2'). The iterative automatic annotation model, i.e., the fusion result, is Result2 = Ψ[f(x1', q2', g(x2')]. 1’ ), g(x 1’ Since no new training model was introduced, F remains unchanged. Adding a new model requires adjusting the fusion matrix F. Therefore, the automatic annotation model iterated from Result1 to Result2. Subsequent iterations follow the same logic; the accuracy of the automatic annotation model improves with each iteration, aiming to maximize the annotation output Q*.

[0080] Therefore, this embodiment integrates a training model with stable parameters into the automatic annotation module, enabling automatic annotation of corresponding target types. Specifically, this embodiment first manually annotates valuable training data, forming a labeled dataset Dataset(q1, q2, ...). Then, it starts model training using Dataset(q1, q2, ...). After the training parameters stabilize, it outputs training model files f(x), g(x), etc. These files are automatically converted and integrated into the automatic annotation module, which then automatically annotates the original data for targets of the same type, ultimately efficiently producing a labeled dataset Q*. Essentially, this embodiment allows the training model to feed back into the automatic annotation process. Through a fusion matrix, different targeted recognition training models can be integrated into a multi-type automatic annotation model, achieving the goal of simultaneous annotation of multiple types of data.

[0081] In summary, this application provides a data annotation method. First, raw data is acquired and manually annotated to obtain a first annotated dataset. Then, the first annotated dataset is input into a pre-set neural network model for training. After the training parameters converge, the trained neural network model is output. Finally, the trained neural network model is used to automatically annotate the raw data to obtain a second annotated dataset. Therefore, this method can achieve automatic annotation by feeding the trained model back into the annotation process. Through a fusion matrix, different targeted recognition training models can be integrated into a multi-type automatic annotation model, achieving the goal of annotating multiple types of data at once. Compared to traditional automatic annotation methods, this method can start model training with a small batch of data, and automatically annotate the data once the model parameters are stable, saving time and resource costs. Secondly, this method, through a fusion matrix, can integrate different recognition algorithm models into a multi-type annotation model, annotating multiple targets or format types at once without repeating the automatic annotation process multiple times. Furthermore, this method can also achieve a positive loop iteration from the training model to the automatic annotation model; as the training model iterates, the automatic annotation model is updated synchronously. Furthermore, this method develops models primarily for target recognition, where the labeled objects are the perceived targets. The recognition results of the perception model are then used as the labeling results of the annotation model, enabling a single model to serve multiple purposes. Additionally, models whose accuracy does not meet the requirements of the training model can be integrated into the automatic annotation system. The automatic annotation model iterates alongside the training model, and after each iteration, it is automatically pushed to the annotation model, achieving parameter iteration of the annotation model without the need for dedicated training.

[0082] like Figure 5 As shown, this application also provides a data annotation system, the system comprising:

[0083] The data acquisition module 510 is used to acquire raw data; as an example, in this embodiment, the raw data includes, but is not limited to, data types such as images, videos, radar point clouds, text, and voice.

[0084] The manual annotation module 520 is used to manually annotate the original data to obtain the first annotated dataset;

[0085] The model training module 530 is used to input the first labeled dataset into a preset neural network model for training, and output the trained neural network model after the training parameters converge.

[0086] The automatic annotation module 540 is used to automatically annotate the original data using a trained neural network model to obtain a second annotated dataset.

[0087] Therefore, this embodiment can achieve automatic annotation by feeding back the training model. Through the fusion matrix, different targeted recognition training models can be integrated into a multi-type automatic annotation model, achieving the goal of annotating multiple types of data simultaneously. Compared to traditional automatic annotation methods, this embodiment can start model training with small batches of data, and automatically annotate once the model parameters are stable, saving time and resource costs. Secondly, this embodiment can integrate different recognition algorithm models into a multi-type annotation model through the fusion matrix, annotating multiple targets or format types at once without repeating the automatic annotation process multiple times. Furthermore, this embodiment can also achieve a positive loop iteration from the training model to the automatic annotation model; as the training model iterates, the automatic annotation model is updated synchronously.

[0088] In an exemplary embodiment, the automatic annotation module 540 automatically annotates the original data using a trained neural network model, comprising: designating the trained neural network model as a first data annotation model; obtaining the vector of the first data annotation model and fusing the vector of the first data annotation model with a preset fusion matrix to obtain a second data annotation model; and automatically annotating the original data using the second data annotation model to obtain the second annotated dataset.

[0089] Specifically, the published trained models f(x) and g(x) need to generate a labeled model consistent with the labeling type requirements through the fusion matrix F, achieving data matching consistent with the original data labeling format; therefore:

[0090] Result=Ψ[f(x),g(x),...]·F (1);

[0091] Q*=R→Ψ[f(x), g(x),...]·F (2);

[0092] Where Ψ[f(x), g(x), ...] represents the training model vector, F represents the fusion matrix, and Q* is the automatic labeling result after inputting the original data R.

[0093] In an exemplary embodiment, after obtaining the second data annotation model, the system further includes: iteratively training the second data annotation model using the first annotation dataset, and outputting a data annotation iterative model after the training parameters converge; and automatically annotating the original data using the data annotation iterative model to obtain the second annotation dataset. Specifically, the second annotation model can perform preliminary automatic annotation of the original data after integration. However, due to the low accuracy of the second annotation model, manual annotation adjustments may still be necessary. That is, the second data annotation model can be iteratively trained one or more times using the original data until the training parameters converge after the last iteration, outputting a data annotation iterative model with the required accuracy. Then, the output data annotation model is used to automatically annotate the original data, thus obtaining a dataset with the required annotation accuracy. In this embodiment, the accuracy of the automatic annotation model improves with each iteration, and the purpose of this improvement is to maximize the annotation capacity Q*.

[0094] In another exemplary embodiment, this application also provides a system for automatically labeled data based on feedback from a trained model, for performing the following steps:

[0095] Input of data to be labeled: The raw data R is directly input for manual labeling. The raw data includes data types such as images, videos, radar point clouds, text, and voice.

[0096] Manual annotation: The input data enters the manual annotation stage. Humans perform preliminary cleaning of the elements in the original data, remove static and duplicate data, and then annotate them. The annotation types include images, videos, radar point clouds, text, and voice. After the annotation is submitted, an annotated dataset is formed.

[0097] The labeled valuable data is stored in the database to form a small dataset (Dataset(q1, q2, ...)) for model training.

[0098] Data feeding algorithm training: After receiving the dataset Dataset(q1, q2, ...), start model training, determine whether the training parameters are stable, if not, repeat the first two steps of training, and after the parameters are basically stable, they can be packaged and output; there are: Dataset(q1) → f(x), Dataset(q2) → g(x), ...;

[0099] Model deployment: Once the model training is stable, it can be deployed. The deployed model can be stored in the model management system as a model file package. The model management system can automatically load the trained models f(x), g(x), etc., into the automatic annotation module.

[0100] Automatic annotation integration: After the training models f(x), g(x), etc. are released, they need to generate an annotation model that is consistent with the annotation type requirements through the fusion matrix F, so as to achieve data matching with the original data annotation format.

[0101] Result=Ψ[f(x),g(x),...]·F (1);

[0102] Q*=R→Ψ[f(x), g(x),...]·F (2);

[0103] Where Ψ[f(x), g(x), ...] represents the training model vector, F represents the fusion matrix, and Q* is the automatic labeling result after inputting the original data R.

[0104] Therefore, this embodiment integrates a training model with stable parameters into the automatic annotation module, enabling automatic annotation of corresponding target types. Specifically, this embodiment first manually annotates valuable training data, forming a labeled dataset Dataset(q1, q2, ...). Then, it starts model training using Dataset(q1, q2, ...). After the training parameters stabilize, it outputs training model files f(x), g(x), etc. These files are automatically converted and integrated into the automatic annotation module, which then automatically annotates the original data for targets of the same type, ultimately efficiently producing a labeled dataset Q*. Essentially, this embodiment allows the training model to feed back into the automatic annotation process. Through a fusion matrix, different targeted recognition training models can be integrated into a multi-type automatic annotation model, achieving the goal of simultaneous annotation of multiple types of data.

[0105] In an exemplary embodiment of this application, a data annotation system is also provided for performing the following steps:

[0106] S1: Collect 1000 original front-view images and label the car and pedestrian elements based on the collected original images. The purpose of manual labeling is only to inject the original training set into the algorithm to obtain stable model parameters. As the accuracy of the automatic labeling model increases, the proportion of manual labeling will gradually decrease, and may even be completely replaced.

[0107] S2: After manual annotation is completed, the data is classified and filtered to select the dataset that meets the conditions. The filtering process involves cleaning and deleting duplicate scene data, such as car target (sedan, SUV, truck, bus, construction vehicle, etc.) image q1 and pedestrian sign image q2 at intersection, forming the training dataset Dataset(q1, q2).

[0108] S3: Inject the manually annotated 2D bounding box JSON results of the manually labeled car and pedestrian dataset images into the perception algorithm for pedestrian and car detection and recognition to start training. If the parameters are unstable, the amount of labeled data can be increased to ensure that the parameters converge.

[0109] S4: The trained model after parameter convergence can be packaged and released, q1—>f(x1), q2—>g(x1), where f(x1) and g(x1) are the output models after the first training.

[0110] S5: The first training model is automatically integrated. The integration process requires fusing the pedestrian and car detection and recognition models. The fusion result is Result1 = Ψ[f(x1), g(x1)]·F. When adding a third training model, the fusion matrix needs to be expanded. After fusion, the result is integrated into the automatic annotation module to achieve the automatic annotation function.

[0111] After the automatic annotation model is integrated, preliminary automatic annotation can be achieved. However, due to the low accuracy of the model, manual annotation and adjustment may still be necessary. After adjustment, the S2 process is repeated to form q1' and q2', the valuable labeled data, which are used to train the model iteratively. The S4 process is repeated: q1' → f(x1'), q2' → g(x2'). The iterative automatic annotation model, i.e., the fusion result, is Result2 = Ψ[f(x1', q2', g(x2')]. 1’ ), g(x 1’ Since no new training model was introduced, F remains unchanged. Adding a new model requires adjusting the fusion matrix F. Therefore, the automatic annotation model iterated from Result1 to Result2. Subsequent iterations follow the same logic; the accuracy of the automatic annotation model improves with each iteration, aiming to maximize the annotation output Q*.

[0112] Therefore, this embodiment integrates a training model with stable parameters into the automatic annotation module, enabling automatic annotation of corresponding target types. Specifically, this embodiment first manually annotates valuable training data, forming a labeled dataset Dataset(q1, q2, ...). Then, it starts model training using Dataset(q1, q2, ...). After the training parameters stabilize, it outputs training model files f(x), g(x), etc. These files are automatically converted and integrated into the automatic annotation module, which then automatically annotates the original data for targets of the same type, ultimately efficiently producing a labeled dataset Q*. Essentially, this embodiment allows the training model to feed back into the automatic annotation process. Through a fusion matrix, different targeted recognition training models can be integrated into a multi-type automatic annotation model, achieving the goal of simultaneous annotation of multiple types of data.

[0113] In summary, this application provides a data annotation system. First, raw data is acquired and manually annotated to obtain a first annotated dataset. Then, the first annotated dataset is input into a pre-set neural network model for training. After the training parameters converge, the trained neural network model is output. Finally, the trained neural network model is used to automatically annotate the raw data to obtain a second annotated dataset. Therefore, this system can achieve automatic annotation by feeding back the training model. Through a fusion matrix, different targeted recognition training models can be integrated into a multi-type automatic annotation model, achieving the goal of annotating multiple types of data at once. Compared to traditional automatic annotation methods, this system can start model training with a small batch of data, and automatically annotate once the model parameters are stable, saving time and resource costs. Secondly, this system can integrate different recognition algorithm models into a multi-type annotation model through a fusion matrix, annotating multiple targets or format types at once without repeating the automatic annotation process multiple times. Furthermore, this system can also achieve a positive loop iteration from the training model to the automatic annotation model; as the training model iterates, the automatic annotation model is updated synchronously. Furthermore, this system develops models primarily for target recognition. The labeled objects are the perceived targets, and the recognition results of the perception model are used in reverse as the labeling results of the annotation model, enabling a single model to serve multiple purposes. Simultaneously, models whose accuracy does not meet the requirements of the training model can also be integrated into the automatic annotation system. The automatic annotation model iterates along with the training model, and after each iteration, it is automatically pushed to the annotation model, achieving parameter iteration of the annotation model without the need for separate training.

[0114] It should be noted that the data annotation system provided in the above embodiments and the data annotation method provided in the above embodiments belong to the same concept. The specific ways in which each module and unit performs operations have been described in detail in the method embodiments, and will not be repeated here. In practical applications, the data annotation system provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above, and this is not a limitation.

[0115] Embodiments of this application also provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the data annotation methods provided in the above embodiments.

[0116] Figure 6 A schematic diagram of a computer system suitable for implementing the embodiments of this application is shown. It should be noted that... Figure 6The computer system 1000 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0117] like Figure 6 As shown, the computer system 1000 includes a Central Processing Unit (CPU) 1001, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 1002 or programs loaded from storage portion 1008 into Random Access Memory (RAM) 1003, such as performing the methods described in the above embodiments. The RAM 1003 also stores various programs and data required for system operation. The CPU 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An Input / Output (I / O) interface 1005 is also connected to the bus 1004.

[0118] The following components are connected to I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to I / O interface 1005 as needed. Removable media 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1010 as needed so that computer programs read from them can be installed into storage section 1008 as needed.

[0119] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1009, and / or installed from removable medium 1011. When the computer program is executed by central processing unit (CPU) 1001, it performs various functions defined in the system of this application.

[0120] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0121] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0122] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0123] Another aspect of this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer's processor, causes the computer to perform the data annotation method as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0124] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data annotation methods provided in the various embodiments described above.

[0125] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A data annotation method, characterized in that, The method includes the following steps: Obtain the raw data and manually annotate it to obtain a first annotated dataset; wherein the first annotated dataset includes the training dataset. Dataset(q 1 ，q 2 ) , q 1 represents a car image. q 2 represents an image of a pedestrian sign at an intersection; wherein, the original data includes images and / or videos; The first labeled dataset is input into a preset neural network model for training, and after the training parameters converge, the trained neural network model is output, including: packaging and publishing the neural network model after the training parameters have converged, to obtain a model based on... q The neural network model output after the first training is denoted as . f(x 1 ) and based on q 2. The neural network model output after the first training is denoted as . g(x 1 ) ; The original data is automatically labeled using a trained neural network model to obtain a second labeled dataset, including: the trained neural network model... f(x) and g(x) Through the fusion matrix F Generate a labeled model that matches the label type requirements, denoted as Result1, where Result1 = Ψ [ f(x 1 ), g(x 1 ) ]· F ,in, Ψ [ f(x 1 ), g(x 1 ) ] represents a neural network model f(x) and g(x) The vector, F This represents the fusion matrix; then, the original data is automatically labeled using the annotation model Result1, resulting in: Q*=R→Ψ [ f(x 1 ), g(x 1 ) ]· F ,in, Q* The automatic annotation results after inputting the original data R will be... Q* As the second labeled dataset.

2. The data annotation method according to claim 1, characterized in that, The process of automatically labeling the raw data using a trained neural network model includes: The trained neural network model is designated as the first data labeling model. Obtain the vector of the first data annotation model, and fuse the vector of the first data annotation model with a preset fusion matrix to obtain the second data annotation model; The original data is automatically labeled using the second data labeling model to obtain the second labeled dataset.

3. The data annotation method according to claim 2, characterized in that, After obtaining the second data annotation model, the method further includes: The second data annotation model is iteratively trained using the first labeled dataset, and the data annotation iterative model is output after the training parameters converge. The original data is automatically labeled using the data labeling iterative model to obtain the second labeled dataset.

4. A data annotation system, characterized in that, The system includes: A data acquisition module is used to acquire raw data; wherein the raw data includes images and / or videos; The manual annotation module is used to manually annotate the original data to obtain a first annotated dataset; wherein, the first annotated dataset includes the training dataset. Dataset(q 1 ，q 2 ) , q 1 represents a car image. q 2 represents a pedestrian sign image at an intersection; The model training module is used to input the first labeled dataset into a preset neural network model for training, and output the trained neural network model after the training parameters converge. This includes: packaging and publishing the neural network model after the training parameters have converged, to obtain a model based on... q The neural network model output after the first training is denoted as . f(x 1 ) and based on q 2. The neural network model output after the first training is denoted as . g(x 1 ) ; An automatic annotation module is used to automatically annotate the original data using a trained neural network model to obtain a second labeled dataset, including: the trained neural network model... f(x) and g(x) Through the fusion matrix F Generate a labeled model that matches the label type requirements, denoted as Result1, where Result1 = Ψ [ f(x 1 ), g(x 1 ) ]· F ,in, Ψ [ f (x 1 ), g(x 1 ) ] represents a neural network model f(x) and g(x) The vector, F This represents the fusion matrix; then, the original data is automatically labeled using the annotation model Result1, resulting in: Q*=R→Ψ [ f(x 1 ), g(x 1 ) ]· F ,in, Q* The automatic annotation results after inputting the original data R will be... Q* As the second labeled dataset.

5. The data annotation system according to claim 4, characterized in that, The automatic annotation module uses a trained neural network model to automatically annotate the original data, including the following steps: The trained neural network model is designated as the first data labeling model. Obtain the vector of the first data annotation model, and fuse the vector of the first data annotation model with a preset fusion matrix to obtain the second data annotation model; The original data is automatically labeled using the second data labeling model to obtain the second labeled dataset.

6. The data annotation system according to claim 5, characterized in that, After obtaining the second data annotation model, the system further includes: The second data annotation model is iteratively trained using the first labeled dataset, and the data annotation iterative model is output after the training parameters converge. The original data is automatically labeled using the data labeling iterative model to obtain the second labeled dataset.

7. An electronic device, characterized in that, The electronic device includes: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to implement the data annotation method as described in any one of claims 1 to 3.

8. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by the computer's processor, causes the computer to perform the data annotation method as described in any one of claims 1 to 3.

Images