Semi-automated annotation method and device based on an open-world large model

Abstract

This application relates to a semi-automated annotation method and device based on an open-world large model. The method includes: obtaining target domain data to be annotated corresponding to a target domain and data annotation requirement information; inputting the target domain data to be annotated and the data annotation requirement information into the open-world large model to determine a target model annotation category and a target model annotation logic; annotating the target domain data to be annotated according to the target model annotation category and the target model annotation logic to obtain pre-annotated target domain data; obtaining annotation data modification information, and adjusting the annotations in the pre-annotated target domain data according to the annotation data modification information to obtain annotated target domain data. Using this method can improve the efficiency of image annotation for image information in a target image.

Description

[0001] Technical scenario

[0002] This application relates to the field of artificial intelligence technology, and particularly to a semi-automated annotation method, device, computer device, storage medium, and computer program product based on an open-world large model. Background technique

[0003] With the development of artificial intelligence technology, image annotation technology has emerged. This technology is a method of associating text descriptions with images, aiming to enable computers to understand and interpret image content. This technology plays an important role in computer vision, artificial intelligence, and natural language processing scenarios. Image annotation is usually used to add descriptions, labels, or keywords to images so that computers can better understand the images and convert them into processable data.

[0004] In traditional technologies, rules defined manually are used to describe the possible objects, scenes, and relationships in images, and then image features are matched to generate image annotations. However, the image annotation process requires manual participation, which is time-consuming and laborious, limiting the utilization and application of image data and resulting in low efficiency in image annotation of image information in target images. Summary of the invention

[0005] Based on this, in view of the above technical problems, it is necessary to provide a method, device, computer device, computer-readable storage medium, and computer program product that can improve the efficiency of image annotation of image information in target images.

[0006] In a first aspect, this application provides a semi-automated annotation method based on an open-world large model. The method includes: obtaining target scene data to be annotated corresponding to a target scene and data annotation requirement information; inputting the target scene data to be annotated and the data annotation requirement information into an open-world large model to determine a target model annotation category and a target model annotation logic; annotating the target scene data to be annotated according to the target model annotation category and the target model annotation logic to obtain pre-annotated target scene data; obtaining annotation data modification information, and adjusting the annotations in the pre-annotated target scene data according to the annotation data modification information to obtain annotated target scene data.

[0007] In a second aspect, the present application also provides a semi-automatic annotation device based on an open-world large model. The device includes: a data acquisition module, configured to acquire target scene data to be annotated corresponding to a target scene and data annotation requirement information; a data analysis module, configured to input the target scene data to be annotated and the data annotation requirement information into the open-world large model to determine a target model annotation category and a target model annotation logic; a data annotation module, configured to annotate the target scene data to be annotated according to the target model annotation category and the target model annotation logic to obtain pre-annotated target scene data; and an annotation adjustment module, configured to acquire annotation data modification information and adjust the annotations in the pre-annotated target scene data according to the annotation data modification information to obtain annotated target scene data.

[0008] In a third aspect, the present application also provides a computer device. The computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the following steps are implemented: acquiring target scene data to be annotated corresponding to a target scene and data annotation requirement information; inputting the target scene data to be annotated and the data annotation requirement information into the open-world large model to determine a target model annotation category and a target model annotation logic; annotating the target scene data to be annotated according to the target model annotation category and the target model annotation logic to obtain pre-annotated target scene data; acquiring annotation data modification information and adjusting the annotations in the pre-annotated target scene data according to the annotation data modification information to obtain annotated target scene data.

[0009] In a fourth aspect, the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, and when the computer program is executed by a processor, the following steps are implemented: acquiring target scene data to be annotated corresponding to a target scene and data annotation requirement information; inputting the target scene data to be annotated and the data annotation requirement information into the open-world large model to determine a target model annotation category and a target model annotation logic; annotating the target scene data to be annotated according to the target model annotation category and the target model annotation logic to obtain pre-annotated target scene data; acquiring annotation data modification information and adjusting the annotations in the pre-annotated target scene data according to the annotation data modification information to obtain annotated target scene data.

[0010] Fifth aspect, the present application also provides a computer program product. The computer program product includes a computer program which, when executed by a processor, implements the following steps: obtaining target scene data to be labeled corresponding to a target scene and data annotation requirement information; inputting the target scene data to be labeled and the data annotation requirement information into an open-world large model to determine a target model annotation category and a target model annotation logic; labeling the target scene data to be labeled according to the target model annotation category and the target model annotation logic to obtain pre-labeled target scene data; obtaining annotation data modification information, and adjusting the annotations in the pre-labeled target scene data according to the annotation data modification information to obtain labeled target scene data.

[0011] The above semi-automated annotation method, device, computer device, storage medium and computer program product based on an open-world large model obtain target scene data to be labeled corresponding to a target scene and data annotation requirement information; input the target scene data to be labeled and the data annotation requirement information into an open-world large model to determine a target model annotation category and a target model annotation logic; label the target scene data to be labeled according to the target model annotation category and the target model annotation logic to obtain pre-labeled target scene data; obtain annotation data modification information, and adjust the annotations in the pre-labeled target scene data according to the annotation data modification information to obtain labeled target scene data.

[0012] By combining an open-world large model and data annotation requirement information, image annotation is performed on the target scene data to be labeled corresponding to the target scene, and further, the image annotation result is revised by using the annotation data modification information input by the user. Combining with the automatic feedback loop mechanism of the system, the open-world large model can be adjusted by using the user's modification, which is beneficial to improving the efficiency of image annotation for image information in the target image. Description of the Drawings

[0013] Figure 1 It is an application environment diagram of a semi-automated annotation method based on an open-world large model in an embodiment;

[0014] Figure 2 It is a flowchart of a semi-automated annotation method based on an open-world large model in an embodiment;

[0015] Figure 3 It is a flowchart of a method for identifying a target model annotation category and a target model annotation logic in an embodiment;

[0016] Figure 4 It is a flowchart of a method for identifying a target model annotation category and a target model annotation logic in another embodiment;

[0017] Figure 5 It is a schematic flowchart of a method for pre-labeling target scenario data in an embodiment;

[0018] Figure 6 It is a schematic flowchart of a method for obtaining labeled target scenario data in an embodiment;

[0019] Figure 7 It is a schematic flowchart of a method for training an open-world large model in an embodiment;

[0020] Figure 8 It is a schematic flowchart of a method for optimizing an open-world large model in an embodiment;

[0021] Figure 9 It is a schematic diagram of the overall architecture of a semi-automated annotation system based on an open-world large model and the method of using it in an embodiment;

[0022] Figure 10 It is a schematic diagram of the implementation process of an incremental learning mechanism in an embodiment;

[0023] Figure 11 It is a structural block diagram of a semi-automated annotation device based on an open-world large model in an embodiment;

[0024] Figure 12 It is an internal structure diagram of a computer device in an embodiment. Detailed implementation manners

[0025] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0026] A semi-automated annotation method based on an open-world large model provided by an embodiment of the present application can be applied to, for example Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or can be placed on the cloud or other network servers. The server 104 obtains the to-be-annotated target scenario data corresponding to the target scenario and the data annotation requirement information from the terminal 102; inputs the to-be-annotated target scenario data and the data annotation requirement information into the open-world large model to determine the target model annotation category and the target model annotation logic; according to the target model annotation category and the target model annotation logic, annotate the to-be-annotated target scenario data to obtain the pre-annotated target scenario data; obtain the annotation data modification information, and according to the annotation data modification information, adjust the annotation in the pre-annotated target scenario data to obtain the annotated target scenario data. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. The portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0027] In one embodiment, as Figure 2 shown, a semi-automated annotation method based on an open-world large model is provided. Taking the server in Figure 1 as an example for illustration, the method includes the following steps:

[0028] Step 202, obtain the to-be-annotated target scenario data corresponding to the target scenario and the data annotation requirement information.

[0029] Among them, the target scenario can be the scenario for annotation, for example: medical image annotation scenario, agricultural image annotation scenario, urban image annotation scenario, cultural image annotation scenario, etc.

[0030] Among them, the to-be-annotated target scenario data can be the data obtained in the target scenario, which can be one or more of text data, image data, or video data.

[0031] Among them, the data annotation requirement information can be the requirements and rules for annotating the to-be-annotated target scenario data.

[0032] Specifically, the server 104 responds to the instructions of the terminal 102, obtains the target scene data to be annotated corresponding to the target scene and the data annotation requirement information from the terminal 102, and stores the obtained target scene data to be annotated and the data annotation requirement information in the storage unit. When the server needs to process any data record in the target scene data to be annotated and the data annotation requirement information, it is retrieved from the storage unit to the volatile storage resource for the central processing unit to calculate. Among them, any data record can be a single data input to the central processing unit, or multiple data can be input to the central processing unit at the same time.

[0033] For example, the server 104 responds to the instructions of the terminal 102, obtains the target scene data to be annotated corresponding to the target scene and the data annotation requirement information from the terminal 102, and stores them in the storage unit in the server 104. Among them, there are 10 data records corresponding to the target scene data to be annotated and the data annotation requirement information obtained by the server 104, and multiple data records can be input to the central processing unit at the same time.

[0034] Step 204, input the target scene data to be annotated and the data annotation requirement information into the open-world large model to determine the target model annotation category and the target model annotation logic.

[0035] Among them, the open-world large model can be an artificial intelligence model that automatically learns and understands various natural language data and uses an incremental learning mechanism to learn new categories without repeated training. Among them, the open-world large model includes the open-world large language model.

[0036] Among them, the target model annotation category can be the category that needs to use the open-world large model to annotate the target scene data to be annotated.

[0037] Among them, the target model annotation logic can be the context reasoning logic that needs to use the open-world large model to annotate the target scene data to be annotated.

[0038] Specifically, the to-be-annotated target scenario data and data annotation requirement information are input into the open-world large model. The open-world large model extracts text features and image features from the to-be-annotated target scenario data respectively, obtaining the to-be-annotated data text features and the to-be-annotated data image features. Then, the type to which the text information of the to-be-annotated target scenario data belongs is identified using the to-be-annotated data text features, and the type to which the image information of the to-be-annotated target scenario data belongs is identified using the to-be-annotated data image features. The above two types are fused as the initial target model annotation categories. Then, the one with the smallest difference value between the initial target model annotation categories and multiple preset model annotation categories in the open-world large model is selected as the target model annotation category for image annotation. At the same time, with the selected target model annotation category as the constraint condition, the model annotation logic that best matches the target model annotation category is selected from multiple preset model annotation logics in the open-world large model as the target model annotation logic.

[0039] Step 206: According to the target model annotation category and the target model annotation logic, annotate the to-be-annotated target scenario data to obtain the pre-annotated target scenario data.

[0040] Among them, the pre-annotated target scenario data can be data obtained by using the open-world large model for annotation but without confirmation.

[0041] Specifically, due to the self-learning feature of the open-world large model, during the process of annotating the to-be-annotated target scenario data, the open-world large model will continuously optimize its target model annotation logic while annotating each data information. The specific process is as follows: Randomly select any to-be-annotated data information from the to-be-annotated target scenario data as the current to-be-annotated data information, and annotate the current to-be-annotated data information according to the target model annotation category and the target model annotation logic to obtain the annotated data information corresponding to the current to-be-annotated data information. Then, the open-world large model adjusts its own target model annotation logic according to the annotated data information to obtain the adjusted target model annotation logic. Then, the server 104 repeats the steps of "randomly select any to-be-annotated data information from the to-be-annotated target scenario data as the current to-be-annotated data information, and annotate the current to-be-annotated data information according to the target model annotation category and the target model annotation logic to obtain the annotated data information corresponding to the current to-be-annotated data information" for the new target model annotation logic and the next to-be-annotated data information until each to-be-annotated data information is annotated to obtain the pre-annotated target scenario data.

[0042] Step 208: Obtain the annotation data modification information, and adjust the annotations in the pre-annotated target scene data according to the annotation data modification information to obtain the annotated target scene data.

[0043] Among them, the annotation data modification information can be information for adjusting the image annotation of the pre-annotated target scene data.

[0044] Among them, the annotated target scene data can be data that has been adjusted and correctly annotated in the image of the target scene.

[0045] Specifically, since there are multiple unadjusted annotation information in the pre-annotated target scene data, before adjustment, the weights of all unadjusted annotation information are 1. Obtain the annotation data modification information through the annotation interface, where the annotation interface interacts with the human annotator through a graphical user interface (GUI) or a command-line interface (CLI). According to the annotation data modification information, reduce the weights of each unadjusted annotation information, while increase the weights of the modified annotation information corresponding to each unadjusted annotation information, but the sum of the weights of each unadjusted annotation information and the corresponding modified annotation information is 1. According to the ratio of the weight of each unadjusted annotation information to the weight of the corresponding modified annotation information, fuse each unadjusted annotation information with the corresponding modified annotation information to obtain the annotated target scene data. The annotated target scene data can receive corrections or confirmations from the human annotator, and feedback the confirmation results to the open-world large model for optimizing and updating the knowledge base of the open-world large model. As Figure 9 shown is a system corresponding to a semi-automated annotation method based on an open-world large model and its overall method architecture.

[0046] In the above semi-automated annotation method based on an open-world large model, obtain the to-be-annotated target scene data corresponding to the target scene and the data annotation requirement information; input the to-be-annotated target scene data and the data annotation requirement information into the open-world large model to determine the target model annotation category and the target model annotation logic; according to the target model annotation category and the target model annotation logic, annotate the to-be-annotated target scene data to obtain the pre-annotated target scene data; obtain the annotation data modification information, and adjust the annotations in the pre-annotated target scene data according to the annotation data modification information to obtain the annotated target scene data.

[0047] By combining the open-world large model and the data annotation requirement information, perform image annotation on the to-be-annotated target scene data corresponding to the target scene, and further revise the image annotation result using the annotation data modification information input by the user. Combining the automatic feedback loop mechanism of the system, it is possible to use the user's modifications to adjust the open-world large model, which is beneficial to improving the efficiency of image annotation for image information in the target image.

[0048] In one embodiment, as Figure 3 shown, input the target scene data to be labeled and the data annotation requirement information into the open-world large model to determine the target model annotation category and the target model annotation logic, including:

[0049] Step 302: Extract text features from the target scene data to be labeled to obtain the text features of the data to be labeled.

[0050] Among them, the text features of the data to be labeled can be the result of extracting features from the text information in the target scene data to be labeled.

[0051] Specifically, use the target scene data to be labeled as the input of the open-world large model. The target scene data to be labeled can be information in different forms such as text, images, and videos. In the input stage of the target scene data to be labeled, it is necessary to preprocess the text information in the target scene data to be labeled to meet the input format requirements of the open-world large model.

[0052] Utilize the intermediate layer of the open-world large model to extract text features from the preprocessed target scene data to be labeled to obtain the text features of the data to be labeled corresponding to the target scene data to be labeled.

[0053] Step 304: Extract image features from the target scene data to be labeled to obtain the image features of the data to be labeled.

[0054] Among them, the image features of the data to be labeled can be the result of extracting features from the image information in the target scene data to be labeled.

[0055] Specifically, use the target scene data to be labeled as the input of the open-world large model. The target scene data to be labeled can be information in different forms such as text, images, and videos. In the input stage of the target scene data to be labeled, it is necessary to preprocess the image information in the target scene data to be labeled to meet the input format requirements of the open-world large model.

[0056] Utilize the intermediate layer of the open-world large model to extract image features from the preprocessed target scene data to be labeled to obtain the image features of the data to be labeled corresponding to the target scene data to be labeled.

[0057] Step 306: Identify the target model annotation category and the target model annotation logic according to the data annotation requirement information, the text features of the data to be labeled, and the image features of the data to be labeled.

[0058] Specifically, using the data annotation requirement information as the selection constraint conditions for the model annotation categories and model annotation logics, based on the characteristics of the data text to be annotated and the characteristics of the data image to be annotated, identify the initial target model annotation categories that can be used to annotate the data of the target scene to be annotated. However, there are multiple preset model annotation categories in the open-world large model. Therefore, calculate the difference values between the initial target model annotation categories and the multiple preset model annotation categories, and select the preset model annotation category with the smallest difference value as the target model annotation category for annotating the data of the target scene to be annotated. Finally, using the selected target model annotation category as the constraint condition, select the model annotation logic that best matches the target model annotation category from the multiple preset model annotation logics of the open-world large model as the target model annotation logic.

[0059] In this embodiment, by separately extracting the features of the text and the image in the data of the target scene to be annotated, and under the constraint of the data annotation requirement information, identifying the target model annotation category and the target model annotation logic, it is possible to utilize the semantic reasoning and context understanding capabilities of the open-world large model to improve the accuracy of subsequent image marking.

[0060] In one embodiment, as Figure 4 shown, based on the data annotation requirement information, the characteristics of the data text to be annotated, and the characteristics of the data image to be annotated, identifying the target model annotation category and the target model annotation logic includes:

[0061] Step 402, using the data annotation requirement information as the constraint condition, based on the characteristics of the data text to be annotated and the characteristics of the data image to be annotated, identify the initial target model annotation categories.

[0062] Among them, the initial target model annotation categories can be categories suitable for annotating the data of the target scene to be annotated, but it has not been verified whether such categories exist in the open-world large model.

[0063] Specifically, use the data annotation requirement information as the selection constraint conditions for the model annotation categories and model annotation logics, input the characteristics of the data text to be annotated and the characteristics of the data image to be annotated into the model annotation category recognition module in the open-world large model, and through the recognition of the model annotation category recognition module, identify the initial target model annotation categories that can be used to annotate the data of the target scene to be annotated.

[0064] Step 404, calculate the difference between the initial target model annotation categories and the preset model annotation categories of the open-world large model.

[0065] Among them, the preset model annotation categories can be the preset model annotation categories for the data to be annotated in the open-world large model.

[0066] Specifically, a differential calculation is performed between the labeled categories of the initial target model and the preset model labeled categories of the open-world large model to obtain the difference value between the labeled categories of the initial target model and the preset model labeled categories of the open-world large model, which is used as the result of the model labeled category difference.

[0067] Step 406: Determine the target model labeled category according to the result of the model labeled category difference.

[0068] Among them, the result of the model labeled category difference can be the degree of difference between the labeled category of the initial target model and the preset model labeled category.

[0069] Specifically, a preset model labeled category difference threshold for restricting the result of the model labeled category difference is set in advance. If multiple results of the model labeled category difference are less than the model labeled category difference threshold, then the preset model labeled category with the smallest difference value is selected from the multiple results of the model labeled category difference as the target model labeled category for labeling the target scene data to be labeled. If multiple results of the model labeled category difference are greater than the model labeled category difference threshold, then the labeled category of the initial target model is used as the target model labeled category, and the target model labeled category is added to the open-world large model, so that the open-world large model can have the ability to label new categories, and at the same time, it also maintains the ability to label the original categories.

[0070] Step 408: Identify the target model labeling logic according to the target model labeled category.

[0071] Specifically, taking the selected target model labeled category as a constraint condition, calculate the matching value between the target model labeled categories in multiple preset model labeling logics of the open-world large model, and then select the model labeling logic with the highest matching degree with the target model labeled category from the multiple preset model labeling logics of the open-world large model as the target model labeling logic.

[0072] In this embodiment, by using the text features of the data to be labeled and the image features of the data to be labeled, the target model labeled category and the target model labeling logic that match the target scene data to be labeled are identified, which enables the open-world large model to have the ability to learn new categories without repeated training and improves the efficiency of image labeling.

[0073] In one embodiment, as Figure 5 shown, label the target scene data to be labeled according to the target model labeled category and the target model labeling logic to obtain the pre-labeled target scene data, including:

[0074] Step 502: Label the current data information to be labeled according to the target model labeled category and the target model labeling logic to obtain the labeled data information.

[0075] Among them, the labeled data information can be the data information that has been image-labeled.

[0076] Specifically, traverse each unlabeled data information in the target scene data to be labeled, and randomly select any one of the unlabeled data information as the current unlabeled data information. Alternatively, according to a fixed selection rule, select any unlabeled data information that meets the fixed selection rule from each unlabeled data information as the current unlabeled data information. Further, according to the target model annotation category and the target model annotation logic, use the semantic reasoning and context understanding capabilities of the deep neural network of the open-world large model to annotate the current unlabeled data information, and obtain the labeled data information corresponding to the current unlabeled data information.

[0077] Step 504, according to the labeled data information, adjust the target model annotation logic to obtain the adjusted model annotation logic.

[0078] Among them, the adjusted model annotation logic can be the model annotation logic obtained by adjusting the target model annotation logic using the labeled data information.

[0079] Specifically, since the open-world large model has the characteristic of self-learning, during the process of annotating the target scene data to be labeled, the open-world large model will continuously optimize its target model annotation logic while annotating each data information. Use the semantic reasoning and context understanding capabilities of the deep neural network of the open-world large model, and combine the labeled data information to adjust the target model annotation logic to obtain the adjusted target model annotation logic.

[0080] Step 506, take the next unlabeled data information as the current unlabeled data information, and take the adjusted model annotation logic as the target model annotation logic, and return to execute the step of annotating the current unlabeled data information according to the target model annotation category and the target model annotation logic to obtain the labeled data information, until the pre-labeled target scene data is obtained.

[0081] Specifically, the server 104 uses the next data information to be labeled as the current data information to be labeled, and also uses the adjusted model annotation logic as the target model annotation logic, and returns to execute the step of "traversing each data information to be labeled in the target scenario data to be labeled, and randomly selecting any one of the data information to be labeled as the current data information to be labeled, or selecting any one of the data information to be labeled that meets a fixed selection rule as the current data information to be labeled according to a fixed selection rule. Further, according to the target model annotation category and the target model annotation logic, use the semantic reasoning and context understanding capabilities of the deep neural network of the open-world large model to annotate the current data information to be labeled, and obtain the labeled data information corresponding to the current data information to be labeled" until the image annotation of each data information to be labeled is completed, and the pre-labeled target scenario data is obtained.

[0082] In this embodiment, by continuously iterating the target model annotation logic using the labeled data information and using the context information between the labeled data information for image annotation, the neural network of the open-world large model can be made more perfect, and the semantic reasoning and context understanding can be adjusted according to the current situation during the image annotation process, improving the accuracy and efficiency of the image annotation.

[0083] In one embodiment, as Figure 6 shown, according to the annotation data modification information, adjust the annotations in the pre-labeled target scenario data to obtain the labeled target scenario data, including:

[0084] Step 602, according to the annotation data modification information, determine the modification data weight of the annotation to be modified in the pre-labeled target scenario data.

[0085] Among them, the modification data weight may be the weight of the modification data of the annotation information to be modified in the pre-labeled target scenario data.

[0086] Specifically, since there are multiple unadjusted annotation information in the pre-labeled target scenario data, before adjustment, the weight of each unadjusted annotation information is 1. According to the annotation data modification information, reduce the weight of each unadjusted annotation information, and the modification data weight of the corresponding modified annotation information of each unadjusted annotation information increases correspondingly, that is, the amount of reduction in the weight of the unadjusted annotation information is equal to the amount of increase in the modification data weight of the corresponding modified annotation information, and the sum of the weight of each unadjusted annotation information and the weight of the corresponding modified annotation information is 1, thereby determining the modification data weight of the annotation to be modified in the pre-labeled target scenario data.

[0087] Step 604: Adjust the annotations in the pre-annotated target scenario data according to the annotation data modification information and the modified data weight to obtain the annotated target scenario data.

[0088] Specifically, for each annotation data modification information, the modified data weight is used to fuse with the corresponding annotation data modification information to obtain each weighted annotation data modification information; similarly, for each unadjusted annotation information, the difference between 1 and the modified data weight is used as the weight, and they are respectively fused with the corresponding unadjusted annotation information to obtain each weighted unadjusted annotation information. Since the weighted annotation data modification information and the weighted unadjusted annotation information have a corresponding relationship, for any weighted unadjusted annotation information and weighted annotation data modification information, the annotated target scenario data is obtained through the fusion method.

[0089] In this embodiment, by giving the weight of the annotation data modification information and the weight of the unadjusted annotation information and fusing the two, it is possible to avoid errors in the image annotation of the open-world large model caused by additional influences and improve the stability of the image annotation.

[0090] In one embodiment, as Figure 7 shown, before the step of obtaining the target scenario data to be annotated corresponding to the target scenario and the data annotation requirement information, the method further includes:

[0091] Step 702: Obtain the training label space data.

[0092] Among them, the training label space data can be a set of all possible labels used to classify, group, or mark data. Among them, the training label space data can be discrete or continuous, specifically depending on the nature of the task.

[0093] Among them, the training label space data can be that two or more data sources or data modalities (such as images, texts, sounds, etc.) share the same label set. That is, the labels between different data sources are semantically consistent, although they may represent different data types or features.

[0094] Among them, heterogeneous training label space data can be a situation where multiple data sources or data modalities (such as images, texts, sounds, etc.) have different label sets. That is, the labels between different data sources may be semantically inconsistent because they may describe different features, attributes, or categories.

[0095] Specifically, the server 104 responds to the instructions of the annotation interface and obtains the training label space data from the annotation interface. Among them, the training label space data includes the training label space data and the heterogeneous training label space data; and stores the obtained training label space data and the heterogeneous training label space data in the storage unit. When the server needs to process any data record in the training label space data and the heterogeneous training label space data, it is retrieved from the storage unit to the volatile storage resource for the central processing unit to perform calculations. Among them, any data record can be a single data input to the central processing unit, or multiple data input to the central processing unit at the same time.

[0096] Step 704: Train the to-be-trained open-world large model according to the training label space data to obtain a first training result.

[0097] Among them, the first training result can be a preliminary open-world large model obtained by training the to-be-trained open-world large model using the training label space data.

[0098] Specifically, input the training label space data into the to-be-trained open-world large model, and through the calculation of the to-be-trained open-world large model, obtain a first loss value for the training label space data. Adjust the to-be-trained open-world large model according to the first loss value until the first loss value is less than the first preset threshold to obtain the first training result.

[0099] Step 706: Train the to-be-trained open-world large model according to the heterogeneous training label space data to obtain a second training result.

[0100] Among them, the second training result can be a preliminary open-world large model obtained by training the to-be-trained open-world large model using the heterogeneous training label space data.

[0101] Specifically, input the heterogeneous training label space data into the to-be-trained open-world large model, and through the decoupled calculation of the to-be-trained open-world large model, obtain a second loss value for the heterogeneous training label space data. Adjust the to-be-trained open-world large model according to the second loss value until the second loss value is less than the second preset threshold to obtain the second training result.

[0102] Step 708: Fuse the first training result and the second training result to obtain an open-world large model.

[0103] Specifically, combine the to-be-trained open-world large model with the first training result and the to-be-trained open-world large model with the second training result to obtain an open-world large model.

[0104] In this embodiment, by training the open-world large model to be trained using the training label space data and the heterogeneous training label space data respectively, the open-world large model can directly use the class embeddings of the test labels during testing to avoid label conflicts, improving the stability of the open-world large model.

[0105] In one embodiment, as Figure 8 shown, after obtaining the annotation data modification information and adjusting the annotations in the pre-annotated target scene data according to the annotation data modification information to obtain the annotated target scene data, the method further includes:

[0106] Step 802, generating adaptive feature information of the open-world large model according to the annotated target scene data.

[0107] Among them, the adaptive feature information can be that in machine learning and deep learning, the model can automatically adapt to learn and extract the most useful feature information according to the characteristics of different tasks or different data. This way of feature learning information allows the model to autonomously adjust its representation ability to better capture the key features of the data, thereby improving the performance of the model.

[0108] Specifically, input the annotated target scene data into the open-world large model, and the feature extraction layer of the open-world large model automatically adapts to learn and extract the most useful feature information according to the characteristics of different tasks or different data to obtain the adaptive feature information.

[0109] Step 804, generating model optimization data of the open-world large model according to the adaptive feature information.

[0110] Among them, the model optimization data can be data that increases the preset model annotation categories in the open-world large model.

[0111] Specifically, integrate the respective adaptive feature information to obtain a feature map, adjust the weights of the respective adaptive feature information in the feature map, and use the knowledge distillation method of the incremental learning mechanism of the open-world large model to generate the model optimization data of the open-world large model, where the model optimization data is used to increase the preset model annotation categories in the open-world large model.

[0112] Among them, the incremental learning mechanism therein is as Figure 10 shown, M of the open-world large model T-1 represents the model after training based on the original categories, and the dynamic structure combination uses structure expansion to optimize new categories. Specifically, a residual adapter is inserted into each convolutional block of the fixed feature extractor in the previous stage. Calculate the difference loss through the extracted features, specifically as follows:

[0113]

[0114] Among them, B represents the minimum batch-size of the input, F represents the feature map, L represents the number of layers of the feature map, and α represents the attention weights extracted from the original large model M T-1 and thus can obtain the important features learned in the original old categories.

[0115] The classification loss is used to optimize the network parameters in the feature extraction process, and its expression is as follows:

[0116]

[0117] Among them, η is a learnable scaling parameter, δ is a constant used to encourage greater separation between classes, [·]+ represents the ReLU activation function, y represents the category, and g represents the category label.

[0118] In this embodiment, by using the adaptive feature information generated from the labeled target scene data to optimize the open-world large model, it is possible to achieve a balance between the adaptability to new data and the robustness to old data.

[0119] It should be understood that although the steps in the flowcharts involved in the above embodiments are shown in sequence according to the arrows, these steps do not necessarily need to be executed in the order indicated by the arrows. Unless otherwise clearly stated in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages. These steps or stages do not necessarily need to be executed at the same time, but can be executed at different times. The execution order of these steps or stages does not necessarily need to be sequential, but can be executed alternately or in turn with at least some of the steps or stages in other steps or other steps.

[0120] Based on the same inventive concept, the embodiments of the present application also provide a semi-automated annotation device based on an open-world large model for implementing the above-mentioned semi-automated annotation method based on an open-world large model. The implementation solutions provided by this device to solve problems are similar to the implementation solutions described in the above method. Therefore, the specific limitations in one or more embodiments of the semi-automated annotation device based on an open-world large model provided below can refer to the limitations on a semi-automated annotation method based on an open-world large model in the above text, and will not be repeated here.

[0121] In one embodiment, as Figure 11As shown, a semi-automated annotation device based on an open-world large model is provided, including: a data acquisition module 1102, a data analysis module 1104, a data annotation module 1106, and an annotation adjustment module 1108, where:

[0122] The data acquisition module 1102 is used to acquire the target scene data to be annotated corresponding to the target scene and the data annotation requirement information;

[0123] The data analysis module 1104 is used to input the target scene data to be annotated and the data annotation requirement information into the open-world large model to determine the target model annotation category and the target model annotation logic;

[0124] The data annotation module 1106 is used to annotate the target scene data to be annotated according to the target model annotation category and the target model annotation logic to obtain the pre-annotated target scene data;

[0125] The annotation adjustment module 1108 is used to obtain the annotation data modification information and adjust the annotations in the pre-annotated target scene data according to the annotation data modification information to obtain the annotated target scene data.

[0126] In one embodiment, the data analysis module 1104 is further used to extract text features from the target scene data to be annotated to obtain the text features of the data to be annotated; extract image features from the target scene data to be annotated to obtain the image features of the data to be annotated; and identify the target model annotation category and the target model annotation logic according to the data annotation requirement information, the text features of the data to be annotated, and the image features of the data to be annotated.

[0127] In one embodiment, the data analysis module 1104 is further used to use the data annotation requirement information as a constraint condition, identify the initial target model annotation category according to the text features of the data to be annotated and the image features of the data to be annotated; calculate the difference between the initial target model annotation category and the preset model annotation category of the open-world large model; determine the target model annotation category according to the model annotation category difference result; and identify the target model annotation logic according to the target model annotation category.

[0128] In one embodiment, the data annotation module 1106 is further configured to annotate the current data information to be annotated according to the target model annotation category and the target model annotation logic to obtain the annotated data information; adjust the target model annotation logic according to the annotated data information to obtain the adjusted model annotation logic; use the next data information to be annotated as the current data information to be annotated, and use the adjusted model annotation logic as the target model annotation logic, and return to execute the step of annotating the current data information to be annotated according to the target model annotation category and the target model annotation logic to obtain the annotated data information until the pre-annotated target scenario data is obtained.

[0129] In one embodiment, the annotation adjustment module 1108 is further configured to determine the modification data weight of the annotation to be modified in the pre-annotated target scenario data according to the annotation data modification information; adjust the annotation in the pre-annotated target scenario data according to the annotation data modification information and the modification data weight to obtain the annotated target scenario data.

[0130] In one embodiment, the data acquisition module 1102 is further configured to acquire the training label space data; the training label space data includes heterogeneous training label space data; train the open-world large model to be trained according to the training label space data to obtain the first training result; train the open-world large model to be trained according to the heterogeneous training label space data to obtain the second training result; fuse the first training result and the second training result to obtain the open-world large model.

[0131] In one embodiment, the annotation adjustment module 1108 is further configured to generate the adaptive feature information of the open-world large model according to the annotated target scenario data; generate the model optimization data of the open-world large model according to the adaptive feature information; the model optimization data is used to increase the preset model annotation category in the open-world large model.

[0132] Each module in the above semi-automated annotation device based on the open-world large model can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0133] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as Figure 12As shown. The computer device includes a processor, a memory, and a network interface connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store server data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, it implements a semi-automated annotation method based on an open-world large model.

[0134] Those skilled in the art of this scenario can understand that Figure 12 the structure shown in is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0135] In one embodiment, a computer device is further provided, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, the steps in the above method embodiments are implemented.

[0136] In one embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0137] In one embodiment, a computer program product or a computer program is provided. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps in the above method embodiments.

[0138] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0139] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memories can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memories can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0140] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as within the scope described in this specification.

[0141] The above-described embodiments merely represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A semi-automated annotation method based on an open-world large model, characterized in that The method includes: Obtaining the target scene data to be labeled corresponding to the target scene and the data annotation requirement information; Performing text feature extraction on the target scene data to be labeled to obtain the text features of the data to be labeled; Performing image feature extraction on the target scene data to be labeled to obtain the image features of the data to be labeled; Using the data annotation requirement information as the selection constraint conditions for the model annotation category and the model annotation logic, and identifying the initial target model annotation category that can be used to label the target scene data to be labeled according to the text features of the data to be labeled and the image features of the data to be labeled; Calculating the difference values between the initial target model annotation category and multiple preset model annotation categories in the open-world large model, and selecting the preset model annotation category with the smallest difference value as the target model annotation category for labeling the target scene data to be labeled; Taking the selected target model annotation category as the constraint condition, and selecting the model annotation logic that best matches the target model annotation category from multiple preset model annotation logics in the open-world large model as the target model annotation logic; the target model annotation logic is the context reasoning logic that needs to use the open-world large model to label the target scene data to be labeled; Labeling the target scene data to be labeled according to the target model annotation category and the target model annotation logic to obtain the pre-labeled target scene data; Obtaining the annotation data modification information, and adjusting the annotations in the pre-labeled target scene data according to the annotation data modification information to obtain the labeled target scene data.

2. The method according to claim 1, wherein The target scene data to be labeled includes multiple pieces of data information to be labeled; the step of labeling the target scene data to be labeled according to the target model annotation category and the target model annotation logic to obtain the pre-labeled target scene data includes: Labeling the current data information to be labeled according to the target model annotation category and the target model annotation logic to obtain the labeled data information; Adjusting the target model annotation logic according to the labeled data information to obtain the adjusted model annotation logic; Taking the next data information to be labeled as the current data information to be labeled, and taking the adjusted model annotation logic as the target model annotation logic, and returning to execute the step of labeling the current data information to be labeled according to the target model annotation category and the target model annotation logic to obtain the labeled data information until the pre-labeled target scene data is obtained.

3. The method according to claim 1, wherein The step of adjusting the annotations in the pre-labeled target scene data according to the annotation data modification information to obtain the labeled target scene data includes: Determining the modification data weight of the annotations that need to be modified in the pre-labeled target scene data according to the annotation data modification information; Adjusting the annotations in the pre-labeled target scene data according to the annotation data modification information and the modification data weight to obtain the labeled target scene data.

4. The method according to claim 1, wherein Before the step of obtaining the to-be-annotated target scenario data corresponding to the target scenario and the data annotation requirement information, the method further includes: Obtaining training label space data; the training label space data includes heterogeneous training label space data; Training the to-be-trained open-world large model according to the training label space data to obtain a first training result; Training the to-be-trained open-world large model according to the heterogeneous training label space data to obtain a second training result; Fusing the first training result and the second training result to obtain the open-world large model.

5. The method according to claim 1, characterized in that, After the step of obtaining the annotation data modification information and adjusting the annotation in the pre-annotated target scenario data according to the annotation data modification information to obtain the annotated target scenario data, the method further includes: Generating adaptive feature information of the open-world large model according to the annotated target scenario data; Generating model optimization data of the open-world large model according to the adaptive feature information; the model optimization data is used to increase the preset model annotation categories in the open-world large model.

6. A semi-automated annotation device based on an open-world large model, characterized in that, The apparatus includes: A data acquisition module, configured to acquire to-be-annotated target scenario data corresponding to a target scenario and data annotation requirement information; A data analysis module, configured to extract text features of the to-be-annotated target scenario data to obtain to-be-annotated data text features; extract image features of the to-be-annotated target scenario data to obtain to-be-annotated data image features; The data analysis module is further configured to use the data annotation requirement information as a selection constraint condition for model annotation categories and model annotation logics, and identify initial target model annotation categories that can be used to annotate the to-be-annotated target scenario data according to the to-be-annotated data text features and the to-be-annotated data image features; calculate the difference values between the initial target model annotation categories and multiple preset model annotation categories in the open-world large model, and select the preset model annotation category with the smallest difference value as the target model annotation category for annotating the to-be-annotated target scenario data; select the model annotation logic that best matches the target model annotation category from multiple preset model annotation logics of the open-world large model with the selected target model annotation category as a constraint condition as the target model annotation logic; the target model annotation logic is the context reasoning logic for using the open-world large model to annotate the to-be-annotated target scenario data; A data annotation module, configured to annotate the to-be-annotated target scenario data according to the target model annotation category and the target model annotation logic to obtain pre-annotated target scenario data; An annotation adjustment module, configured to obtain annotation data modification information and adjust the annotation in the pre-annotated target scenario data according to the annotation data modification information to obtain annotated target scenario data.

7. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images