Integrated machine learning model pre-labeling method and system

Abstract

The invention discloses an integrated machine learning model pre-labeling method and system. The method comprises the following steps: S1) importing data; s2) model selection and matching; s202) multi-model selection and automatic optimization: comprehensively selecting an optimal model based on a selection strategy of data features and task complexity; s3) automatic configuration of the model; S4) pre-labeling of the model; s5) performing manual auditing and correction; and S6) outputting a labeling result. S7) performing model performance evaluation, and automatically adjusting hyper-parameters; s8) updating a model recommendation strategy through a random forest algorithm; the system comprises a user module, a data import module, an annotation type selection module, a model configuration module, an annotation module, an auditing and adjusting module, an annotation result output module and a model adjusting module. According to the method, the machine learning model is integrated to realize pre-annotation of the data, meanwhile, the recommendation algorithm based on the annotation type is designed, the machine learning model is recommended according to the annotation type selected by the user, and the annotation accuracy and speed are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data annotation, and in particular relates to an integrated machine learning model pre-annotation method and system. Background Art

[0002] In the current annotation field, with the advent of the big data era, data annotation has become an indispensable part of machine learning model training. Whether it is text classification, text recognition, named entity recognition, or image, audio, video and time series data processing, a large amount of labeled data is required to support model training and optimization.

[0003] Traditional data labeling methods mainly rely on manual operations, which have many limitations. First, manual labeling is a labor-intensive process that requires a lot of human resources, which not only increases costs but also makes labeling work time-consuming and labor-intensive. Second, due to the differences in professional backgrounds and subjective judgments of different labelers, manual labeling is difficult to ensure the consistency and accuracy of labeling results, which will limit the effect of model training. In addition, the efficiency of manual labeling is relatively low and it is difficult to meet the current demand for large-scale data labeling.

[0004] Therefore, it is urgent to design a new integrated machine learning method and system to solve the above problems. Summary of the invention

[0005] Purpose of the invention: In order to overcome the above shortcomings, the purpose of the present invention is to provide an integrated machine learning model pre-labeling method and system, which realizes data pre-labeling by integrating machine learning models, reduces the workload of manual labeling, improves the accuracy and speed of labeling, and can more efficiently complete the labeling task when processing large-scale and multi-type data; designs a recommendation algorithm based on the labeling type, which can intelligently recommend the most suitable machine learning model according to the labeling type selected by the user; uses historical data to train the random forest algorithm model to dynamically adjust the best model recommendation; supports the fusion labeling of 2D images and 3D point cloud data, and improves the labeling efficiency.

[0006] Technical solution: In order to achieve the above object, the present invention provides an integrated machine learning model pre-labeling method, comprising the following steps: S1): Import data, import various types of data that need to be labeled; S2): Model selection and matching, selecting machine learning models based on data types and features; S201): Match the corresponding model type according to the data type; S202): Multi-model selection and automatic optimization. For the same annotation type, there may be multiple applicable pre-trained models. The optimal model is selected based on the selection strategy of data characteristics and task complexity. S3): Automatic model configuration: for pre-trained models used for various types of data, the model uses its own hyperparameter optimization algorithm to automatically set default hyperparameters; S4): Model pre-annotation; S5): Manual review and correction; S6): Output the labeling results, and obtain the labeling results, labeling time and labeling accuracy; S7): Model performance evaluation, cross-validation, evaluation of its accuracy, recall and F1 value on the labeling task, and automatic adjustment of hyperparameters based on the above performance indicators; S8): Update the model recommendation strategy through the random forest algorithm.

[0007] Furthermore, by integrating machine learning models to achieve pre-labeling of data, the workload of manual labeling can be reduced, and the accuracy and speed of labeling can be improved, especially when processing large-scale, multi-type data, labeling tasks can be completed more efficiently; Further, the S202) includes: S20201): Selection strategy based on task complexity. Task complexity refers to the difficulty of the annotation task, including but not limited to text length, image resolution, audio duration, etc. Task complexity score complexity The calculation formula is as follows: Text data: score complexity = Text length (number of characters or words) / 1000 Image data: score complexity = Image resolution (number of pixels) / 1000000 Audio data: score complexity = audio duration (seconds) / 60; S20202): Selection strategy based on data features. Data features refer to the specific attributes of the labeled data, such as entity types in text, object categories in images, etc. Data feature scores feature The calculation formula is as follows: Text data: score feature = Number of entity types / 10 Image data: score feature = Number of object categories / 10 Audio data: score feature = number of spectrum peaks / 100; S20203): Comprehensively select the optimal model.

[0008] Furthermore, the selection strategy based on task complexity selects models according to the complexity of different tasks, which can ensure the adaptability of the model; the selection strategy based on data features will give priority to models that can handle specific data features, which can enhance the generalization ability of the model.

[0009] Further, the S20203) includes: S2020301): Weight setting: weights are set based on expert experience and historical data, and the weights of task complexity and data features are set as w 1 and w 2, and w 1 and w 2 and 1 are equal; S2020302): Model selection score calculation, for each candidate model, its task complexity score complexity and data feature score feature The scores are weighted averaged and the formula is as follows: score total = w 1×score complexity + w 2×score feature Get the final model selection score total , select the model with the highest selection score. Evaluating model selection from the perspectives of task complexity and data characteristics can select the most appropriate model for tasks of different complexity, thereby improving the efficiency and accuracy of task processing; at the same time, model selection will be dynamically adjusted according to the complexity of the task, and has strong adaptability.

[0010] Further, the S8) includes: S801): Collect data from past annotation tasks, including task complexity, data features, selected models, their annotation time, and annotation accuracy; S802): Extract text length, keyword frequency, number of entity types, and annotation time to create a training set based on historical data; S803): Select the random forest algorithm, use the training set to train the model, take the task complexity and data features as input, take the selection result of the best model as output, and obtain the trained random forest algorithm model; S804): When there is a new labeling task, extract the task complexity and data features, input the trained random forest algorithm model, and obtain the prediction selection result of the best model. Use historical data to train the random forest algorithm model, and predict the best model based on the trained random forest algorithm model, and dynamically adjust the model recommendation strategy to ensure the scientific nature of the model recommendation strategy.

[0011] Furthermore, the data import in S1) also supports the simultaneous import of 2D images and 3D point cloud data, and fusion annotations, including: S101): Coordinate system alignment: the 2D image coordinate system is the pixel coordinate system, which indicates the position of each pixel in the image; the 3D point cloud coordinate system is the camera coordinate system, which indicates the position of each point in the point cloud in the three-dimensional space; the camera's internal and external parameter matrices are used for conversion to complete the coordinate system alignment; S102): Use the camera internal parameter matrix K and external parameter matrix [R|t] to project the 3D point P onto the 2D image plane p. The formula is as follows: p = K [ R | t ] P Among them, the camera intrinsic parameter matrix K includes camera parameters such as focal length and principal point offset, R and t are the rotation and translation parameters of the camera, respectively, which are obtained by the calibration process; S103): According to the projection result in S102), the points in the 3D point cloud are matched with the pixels in the 2D image. The 2D image and 3D point cloud data are imported at the same time and fused with annotations, which improves the readability of the annotations, improves the efficiency and accuracy of the annotations, and improves the robustness of the model in complex environments.

[0012] Furthermore, in the data imported in S1), for image data, the image quality is compressed by using the Thumbnailator compression algorithm for uploading. Compressing the image quality reduces the transmission flow and the waiting period while ensuring the image features as much as possible.

[0013] The present invention also provides an integrated machine learning model pre-annotation system, comprising: a user module, a data import module, an annotation type selection module, a model configuration module, an annotation module, a review and adjustment module, an annotation result output module and a model adjustment module; When a user accesses the system, he or she first assigns different permissions through the user module, and then completes the labeling task through the data import module, labeling type selection module, model configuration module, labeling module, review and adjustment module, and labeling result output module in sequence; after the labeling result output module is called, the system automatically calls the model adjustment module to adjust the model hyperparameters and model recommendation selections.

[0014] Furthermore, the user module includes roles with five levels of permissions, namely Owner, Administrator, Manager, Reviewer and Annotator; Owner has the highest permissions; Administrator is set by the owner and can add, delete and assign users as managers, reviewers and annotators; Administrators and managers can create workspaces to organize projects; Administrators can view, add or remove any workspace and members; Managers can view any workspace and members, add or remove their own workspace and members; Reviewers can view tasks in their workspace, and review and adjust annotations on tasks; Annotators only have the permission to view and annotate tasks assigned to them in their workspace. Different users are graded by assigning roles to ensure efficient task management and assignment.

[0015] Furthermore, after the workspace is created, members are added, and each member is divided into a sandbox space in the workspace; the sandbox space is used to complete tasks independently; the members can also update the tasks in the sandbox space to the workspace for sharing and collaboration. The sandbox space ensures that each task is completed in an independent and controlled environment, preventing interference between characters and protecting the security of tasks and data.

[0016] It can be seen from the above technical solution that the present invention has the following beneficial effects: 1. The present invention provides an integrated machine learning model pre-labeling method and system, which realizes data pre-labeling by integrating a machine learning model, reduces the workload of manual labeling, improves the accuracy and speed of labeling, and can more efficiently complete the labeling task when processing large-scale and multi-type data; 2. The present invention provides an integrated machine learning model pre-annotation method and system, designs a recommendation algorithm based on annotation type, and can intelligently recommend the most suitable machine learning model according to the annotation type selected by the user; 3. The present invention provides an integrated machine learning model pre-labeling method and system, which uses historical data to train a random forest algorithm model to dynamically adjust the optimal model recommendation; 4. The present invention provides an integrated machine learning model pre-annotation method and system, which supports the fusion annotation of 2D images and 3D point cloud data, and improves the annotation efficiency; 5. The present invention provides an integrated machine learning model pre-annotation method and system, which uses the Thumbnailator compression algorithm to compress the image quality for uploading, thereby reducing the transmission traffic and the waiting period; 6. The present invention provides an integrated machine learning model pre-labeling method and system, with user authority classification, to ensure efficient task management and allocation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A step diagram of a method and system for pre-labeling an integrated machine learning model according to the present invention; Figure 2 A flowchart of a method for pre-labeling an integrated machine learning model and a method in the system according to the present invention; Figure 3 A schematic diagram of the structure of a system in an integrated machine learning model pre-labeling method and system according to the present invention; Figure 4 This is an example of the format of the random forest algorithm model training data in the integrated machine learning model pre-labeling method and system described in the present invention. DETAILED DESCRIPTION

[0018] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention. Example

[0019] In this embodiment, Figure 1 and Figure 2 The present invention discloses a pre-labeling method for an integrated machine learning model, comprising the following steps: S1): Import data, import various types of data that need to be labeled; S2): Model selection and matching, selecting machine learning models based on data types and features; S201): Match the corresponding model type according to the data type; S202): Multi-model selection and automatic optimization. For the same annotation type, there may be multiple applicable pre-trained models. The optimal model is selected based on the selection strategy of data characteristics and task complexity. S3): Automatic model configuration: for pre-trained models used for various types of data, the model uses its own hyperparameter optimization algorithm to automatically set default hyperparameters; S4): Model pre-annotation; S5): Manual review and correction; S6): Output the labeling results, and obtain the labeling results, labeling time and labeling accuracy; S7): Model performance evaluation, cross-validation, evaluation of its accuracy, recall and F1 value on the labeling task, and automatic adjustment of hyperparameters based on the above performance indicators; S8): Update the model recommendation strategy through the random forest algorithm.

[0020] In this embodiment, Figure 1 and Figure 2 , the S202) includes: S20201): Selection strategy based on task complexity. Task complexity refers to the difficulty of the annotation task, including but not limited to text length, image resolution, audio duration, etc. Task complexity score complexity The calculation formula is as follows: Text data: score complexity = Text length (number of characters or words) / 1000 Image data: score complexity = Image resolution (number of pixels) / 1000000 Audio data: score complexity = audio duration (seconds) / 60; S20202): Selection strategy based on data features. Data features refer to the specific attributes of the labeled data, such as entity types in text, object categories in images, etc. Data feature scores feature The calculation formula is as follows: Text data: score feature = Number of entity types / 10 Image data: score feature = Number of object categories / 10 Audio data: score feature = number of spectrum peaks / 100; S20203): Comprehensively select the optimal model.

[0021] Specifically, S20202): In the selection strategy based on data features, the data feature score feature The calculation formula has the following alternatives: Text data: score feature = TF-IDF value (text diversity) / 10 Image data: score feature = Number of edges after edge detection (image complexity) / 1000.

[0022] In this embodiment, Figure 1 and Figure 2 , the S20203) includes: S2020301): Weight setting: weights are set based on expert experience and historical data, and the weights of task complexity and data features are set as w 1 and w 2, and w 1 and w2 and 1 are equal; S2020302): Model selection score calculation, for each candidate model, its task complexity score complexity and data feature score feature The scores are weighted averaged and the formula is as follows: score total = w 1×score complexity + w 2×score feature Get the final model selection score total , the model with the highest selection score is selected.

[0023] Specifically, for the text annotation task, suppose there is a text annotation task with a text length of 1000 characters and containing 5 entity types; Task complexity score: score complexity = 1000 / 1000 = 1.0 Data feature score: score feature = 5 / 10 = 0.5 Given weight w 1=0.6, w 20.4, Model Selection Score total The calculation process is as follows: score total = 0.6×1.0+0.4×0.5=0.6+0.2=0.8 The calculation of the model selection score for the image annotation task is the same as above, and the model with the highest model selection score is finally selected.

[0024] In this embodiment, Figure 1 , Figure 2 and Figure 4 , the S8) comprises: S801): Collect data from past annotation tasks, including task complexity, data features, selected models, their annotation time, and annotation accuracy; S802): Extract text length, keyword frequency, number of entity types, and annotation time to create a training set based on historical data; S803): Select the random forest algorithm, use the training set to train the model, take the task complexity and data features as input, take the selection result of the best model as output, and obtain the trained random forest algorithm model; S804): When there is a new labeling task, extract the task complexity and data features, input the trained random forest algorithm model, and obtain the prediction selection result of the best model.

[0025] Specifically, an A / B test was used to verify the effect. An A / B test was designed to divide users into two groups: Group A: continued to use the old model selection strategy; Group B: used the new random forest model selection strategy. The labeling results, feedback, and labeling time of the two groups of users were recorded. The labeling accuracy was obtained: the average accuracy of Group A was 75%, and the average accuracy of Group B was 85%. At the same time, in terms of labeling time, the average labeling time of Group A was 60 seconds, and the average labeling time of Group B was 45 seconds. It can be found that using the trained random forest algorithm model for model selection significantly improved the labeling quality and efficiency.

[0026] In this embodiment, Figure 1 and Figure 2 , the data imported in S1) also supports the simultaneous import of 2D images and 3D point cloud data, and fusion annotation, including: S101): Coordinate system alignment: the 2D image coordinate system is the pixel coordinate system, which indicates the position of each pixel in the image; the 3D point cloud coordinate system is the camera coordinate system, which indicates the position of each point in the point cloud in the three-dimensional space; the camera's internal and external parameter matrices are used for conversion to complete the coordinate system alignment; S102): Use the camera internal parameter matrix K and external parameter matrix [R|t] to project the 3D point P onto the 2D image plane p. The formula is as follows: p = K [ R | t ] P Among them, the camera intrinsic parameter matrix K includes camera parameters such as focal length and principal point offset, R and t are the rotation and translation parameters of the camera, respectively, which are obtained by the calibration process; S103): According to the result of the projection in S102), the points in the 3D point cloud are matched with the pixels in the 2D image.

[0027] Specifically, data import supports the import of multiple point cloud file formats, such as PCD and BIN formats.

[0028] Specifically, when annotating 3D data, the corresponding box can be seen in the RGB data; the annotation results of the 3D point cloud data are projected onto the RGB data after calculating the camera parameters.

[0029] Specifically, the 2D box and the 3D box are linked to ensure that when one box is modified, the other box is also updated accordingly.

[0030] In this embodiment, Figure 1 and Figure 2 , the data is imported in S1), and for image data, the Thumbnailator compression algorithm is used to compress the image quality for uploading.

[0031] Specifically, the JPEG format compression algorithm is used by default, which is suitable for image compression in most cases; the JPEG format can effectively reduce the file size while maintaining high image quality.

[0032] In this embodiment, Figure 3 , the present invention also discloses an integrated machine learning model pre-annotation system, including: a user module, a data import module, an annotation type selection module, a model configuration module, an annotation module, a review and adjustment module, an annotation result output module and a model adjustment module; When a user accesses the system, he or she first assigns different permissions through the user module, and then completes the labeling task through the data import module, labeling type selection module, model configuration module, labeling module, review and adjustment module, and labeling result output module in sequence; after the labeling result output module is called, the system automatically calls the model adjustment module to adjust the model hyperparameters and model recommendation selections.

[0033] In this embodiment, Figure 3 The user module includes five levels of roles, namely Owner, Administrator, Manager, Reviewer and Annotator; The owner has the highest authority; the administrator is set by the owner and can add, delete, and assign users as managers, reviewers, and annotators; both administrators and managers can create workspaces to organize projects; administrators can view, add, or remove any workspace and members; managers can view any workspace and members, and add or remove their own workspace and members; reviewers can view tasks in their workspaces, as well as review and adjust comments on tasks; annotators only have the authority to view and annotate tasks assigned to them in their workspaces.

[0034] Specifically, after creating a project, you can update the project to the workspace so that others can collaborate on the project.

[0035] Specifically, you can use the API endpoints to programmatically activate and assign user roles by sending a POST request to the / api / organizations / {id} / memberships endpoint to assign a role to a specific user ID and organization ID.

[0036] In particular, administrators can generate an invitation link, copy it, and share it with users who need to join the organization.

[0037] In this embodiment, Figure 3After creating the workspace, members are added, and each member will be assigned a sandbox space in the workspace; the sandbox space is used to complete tasks independently; the members can also update the tasks in the sandbox space to the workspace for sharing and collaboration.

[0038] Specifically, each user has a personal sandbox workspace that can be used to experiment and familiarize themselves with tasks.

[0039] Specifically, MinIO is used to store and manage a large number of labeling tasks.

[0040] The above description is only a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements without departing from the principle of the present invention, and these improvements should also be regarded as within the protection scope of the present invention.

Claims

1. A pre-labeling method for an integrated machine learning model, characterized by: The steps include: S1): Import data, import various types of data that need to be labeled; S2): Model selection and matching, selecting machine learning models based on data types and features; S201): Match the corresponding model type according to the data type; S202): Multi-model selection and automatic optimization. For the same annotation type, there may be multiple applicable pre-trained models. The optimal model is selected based on the selection strategy of data characteristics and task complexity. S3): Automatic model configuration: for pre-trained models used for various types of data, the model uses its own hyperparameter optimization algorithm to automatically set default hyperparameters; S4): Model pre-annotation; S5): Manual review and correction; S6): Output the labeling results, and obtain the labeling results, labeling time and labeling accuracy; S7): Model performance evaluation, cross-validation, evaluation of its accuracy, recall and F1 value on the labeling task, and automatic adjustment of hyperparameters based on the above performance indicators; S8): Update the model recommendation strategy through the random forest algorithm.

2. The method for pre-labeling an integrated machine learning model according to claim 1, characterized in that: The S202) includes: S20201): Selection strategy based on task complexity. Task complexity refers to the difficulty of the annotation task, including but not limited to text length, image resolution, audio duration, etc. Task complexity score complexity The calculation formula is as follows: Text data: score complexity = Text length (number of characters or words) / 1000 Image data: score complexity = Image resolution (number of pixels) / 1000000 Audio data: score complexity = audio duration (seconds) / 60; S20202): Selection strategy based on data features. Data features refer to the specific attributes of the labeled data, such as entity types in text, object categories in images, etc. Data feature scores feature The calculation formula is as follows: Text data: score feature = Number of entity types / 10 Image data: score feature = Number of object categories / 10 Audio data: score feature = number of spectrum peaks / 100; S20203): Comprehensively select the optimal model.

3. The method for pre-labeling an integrated machine learning model according to claim 2, characterized in that: The S20203) includes: S2020301): Weight setting: weights are set based on expert experience and historical data, and the weights of task complexity and data features are set as w 1 and w 2, and w 1 and w 2 and 1 are equal; S2020302): Model selection score calculation, for each candidate model, its task complexity score complexity and data feature score feature The scores are weighted averaged and the formula is as follows: score total = w 1×score complexity + w 2×score feature Get the final model selection score total , the model with the highest selection score is selected.

4. The method for pre-labeling an integrated machine learning model according to claim 1, characterized in that: The S8) includes: S801): Collect data from past annotation tasks, including task complexity, data features, selected models, their annotation time, and annotation accuracy; S802): Extract text length, keyword frequency, number of entity types, and annotation time to create a training set based on historical data; S803): Select the random forest algorithm, use the training set to train the model, take the task complexity and data features as input, take the selection result of the best model as output, and obtain the trained random forest algorithm model; S804): When there is a new labeling task, extract the task complexity and data features, input the trained random forest algorithm model, and obtain the prediction selection result of the best model.

5. The method for pre-labeling an integrated machine learning model according to claim 1, characterized in that: The data imported in S1) also supports the simultaneous import of 2D images and 3D point cloud data, and fusion annotation, including: S101): Coordinate system alignment: the 2D image coordinate system is the pixel coordinate system, which indicates the position of each pixel in the image; the 3D point cloud coordinate system is the camera coordinate system, which indicates the position of each point in the point cloud in the three-dimensional space; the camera's internal and external parameter matrices are used for conversion to complete the coordinate system alignment; S102): Use the camera internal parameter matrix K and external parameter matrix [R|t] to project the 3D point P onto the 2D image plane p. The formula is as follows: p = K [ R | t ] P Among them, the camera intrinsic parameter matrix K includes camera parameters such as focal length and principal point offset, R and t are the rotation and translation parameters of the camera, which are obtained by the calibration process; S103): According to the result of the projection in S102), the points in the 3D point cloud are matched with the pixels in the 2D image.

6. The method for pre-labeling an integrated machine learning model according to claim 1, characterized in that: In the step S1), data is imported. For image data, the image quality is compressed by using the Thumbnailator compression algorithm for uploading.

7. An integrated machine learning model pre-labeling system, used to implement an integrated machine learning model pre-labeling method according to claims 1 to 6, characterized in that: include: User module, data import module, annotation type selection module, model configuration module, annotation module, review and adjustment module, annotation result output module and model adjustment module; When a user accesses the system, he or she first assigns different permissions through the user module, and then completes the labeling task through the data import module, labeling type selection module, model configuration module, labeling module, review and adjustment module, and labeling result output module in sequence; after the labeling result output module is called, the system automatically calls the model adjustment module to adjust the model hyperparameters and model recommendation selections.

8. The integrated machine learning model pre-labeling system according to claim 7, characterized in that: The user module includes five roles with different levels of permissions, namely Owner, Administrator, Manager, Reviewer and Annotator; The owner has the highest authority; the administrator is set by the owner and can add, delete, and assign users as managers, reviewers, and annotators; both administrators and managers can create workspaces to organize projects; administrators can view, add, or remove any workspace and members; managers can view any workspace and members, and add or remove their own workspace and members; reviewers can view tasks in their workspaces, as well as review and adjust comments on tasks; annotators only have the authority to view and annotate tasks assigned to them in their workspaces.

9. The integrated machine learning model pre-labeling system according to claim 8, characterized in that: After the workspace is created, members are added, and each member is assigned a sandbox space in the workspace; the sandbox space is used to complete tasks independently; the members can also update the tasks in the sandbox space to the workspace for sharing and collaboration.

Images