Data annotation method, medium and electronic equipment

Abstract

The invention is applied to the field of artificial intelligence, and provides a data annotation method, a medium and electronic device.The method comprises the steps that at least one keyword in material data within preset time is obtained from a material data set; determining the weight of at least one keyword; and updating the tags contained in the tag library according to the at least one keyword and the weight of the at least one keyword. The method further includes identifying a plurality of modal feature vectors of the material data, the material data including at least one modal. Determining a target feature vector according to the plurality of modal feature vectors; and determining a feature vector of at least one tag in a tag library, and determining a target tag in the tag library according to the target feature vector and the feature vector of the at least one tag. Furthermore, the method can automatically and regularly update the label library, supplement labels of new hot topics and eliminate outdated labels. The label library does not need to be manually updated, the labor cost is saved, the feature information of each mode can be integrated, and the data labeling accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a data labeling method, a medium and an electronic device. Background Art

[0002] Scenarios such as news recommendation and advertisement recommendation all involve data annotation. The data annotation model can annotate material data with labels, and then recommend material data to users based on the annotated labels. New material data will continue to be generated, forming a data stream that continuously flows into the data annotation model. The data annotation model has several categories of labels preset. The data annotation model usually understands the content of the material data and determines at least one label similar to the material data from several preset labels as the label of the material data. Furthermore, the data annotation model can also combine user preferences to recommend material data corresponding to the labels that the user is interested in to the user.

[0003] However, hot topics are generated quickly, and the content of new material data is also changing quickly. Newly generated topics do not have corresponding labels in the data annotation model. If the material data corresponding to the new topic flows into the data annotation model, the data annotation model cannot determine the labels with high similarity from the label library, resulting in the data annotation model being unable to output label recognition results. Alternatively, the data annotation model can only determine labels with relatively high similarity from the existing labels as output, but the matching between the labels and the material data is not high.

[0004] That is to say, when material data on new topics appears, the data annotation model cannot give accurate recognition results, which in turn makes it impossible to accurately push material data that users are interested in to users. Summary of the invention

[0005] The present application provides a data labeling method, medium and electronic device to solve the problem that the current data labeling model cannot determine accurate labels for material data of new topics.

[0006] In a first aspect, a data labeling method is provided, which is applied to a first device and a second device, and the method includes: the first device obtains at least one keyword in the material data within a preset time from the material data set; the first device determines the weight of at least one keyword; the first device updates the label contained in the label library according to the at least one keyword and the weight of at least one keyword; the second device determines a target feature vector of the first material data, the first material data includes at least one mode, and the target feature vector is used to represent the feature information contained in the at least one mode; the second device determines the target label corresponding to the first material data in the label library according to the target feature vector.

[0007] In the above scheme, the first device is a training device, and the second device is an execution device. The first device can regularly update the label library based on recent material data, so that the labels in the label library can adapt to new hot topics, and then the second device can output labels based on the label library that are more in line with current hot topics. There is no need to manually update the label library, saving labor costs. In addition, when material data related to new topics flows in, the labels corresponding to the material data can be accurately identified based on the updated label library, and the material data that the user is interested in can be accurately pushed to the user, improving the accuracy of data recommendation.

[0008] In combination with the first aspect, in some implementations, the method also includes: the first device obtains the weight of at least one tag in the tag library; the first device is also used to determine the correspondence between at least one keyword and at least one tag based on the matching degree between at least one keyword and at least one tag; the first device updates the weight of at least one tag based on the weight of at least one keyword and the weight of the tag corresponding to the at least one keyword.

[0009] In the above scheme, the first device can also determine whether the keywords and tags match based on whether they are the same. It can also determine whether the keywords and tags are synonyms based on a preset synonym thesaurus, and determine the tags in the tag library that are synonyms of the keywords as tags that match the keywords. Furthermore, the tag library can be updated based on recent material data, so that the tags in the tag library can adapt to new hot topics, and there is no need to manually update the tag library, saving labor costs.

[0010] In combination with the first aspect, in some implementations, the method further includes: the first device weights the weight of a first keyword in at least one keyword and the weight of a first tag corresponding to the first keyword in the tag library and adds them together to obtain an updated weight of the first tag.

[0011] In the above scheme, the weight of the keyword can be determined according to the frequency of the keyword appearing in the near-line material data. In this way, the newly appeared hot keywords can be added to the tag library, so that the tags in the tag library can adapt to the new hot topics without the need to manually update the tag library, saving labor costs.

[0012] In combination with the first aspect, in some implementations, the method further includes: the first device adds the second keyword as a second tag to the tag library based on the absence of a tag corresponding to the second keyword in the tag library.

[0013] In the above scheme, the first device will be able to add the recently emerging hot keywords to the tag library, so that the tags in the tag library can adapt to the new hot topics, and there is no need to manually update the tag library, saving labor costs.

[0014] In combination with the first aspect, in some implementations, the weight of the second keyword is greater than the first threshold.

[0015] In the above scheme, the first device can also simply add keywords that have appeared frequently recently, that is, keywords with higher weights, to the tag library. In this way, the tags in the tag library can introduce new hot keywords, so that the tags in the tag library can adapt to new hot topics, and there is no need to manually update the tag library, saving labor costs. In addition, when material data related to new topics flows in, the tags corresponding to the material data can be accurately identified based on the updated tag library, and the material data that the user is interested in can be accurately pushed to the user, improving the accuracy of data recommendations.

[0016] In combination with the first aspect, in some implementations, the method further includes: the first device deleting the third tag from the tag library based on the weight of the third tag in the at least one updated tag being less than a second threshold.

[0017] In the above scheme, if the weight of a tag is low, it means that the discussion about the tag is low recently, so the tag will be deleted. In this way, the number of tags in the tag library can be controlled within a certain range, and when matching material data and tags, the matching time will not be long due to too many tags.

[0018] In combination with the first aspect, in some implementations, the second device includes a target model, and the method includes: the second device obtains multiple modal feature vectors of the first material data through the target model recognition, and the modal feature vector is used to represent the feature information of the corresponding modality; the second device determines the target feature vector based on the multiple modal feature vectors through the target model; the second device determines the feature vector of at least one tag in the tag library through the target model; the second device determines the target tag in the tag library based on the target feature vector and the feature vector of at least one tag through the target model.

[0019] In the above scheme, the second device may be a label that determines the material data matches through the target model, and since the label library is updated, the target model can accurately push the material data that the user is interested in to the user, thereby improving the accuracy of data recommendation.

[0020] In combination with the first aspect, in some implementations, the first device obtains material data belonging to a label that meets preset conditions from a material data set, and the preset conditions include that the increase in the weight of the label after the update is greater than a third threshold, or that the label does not belong to the label library before the update; the first device trains a target model based on the material data of the material data belonging to the label that meets the preset conditions.

[0021] In the above scheme, the first device can also re-train the target model with the material data of labels that appear frequently in recent material data according to the update of the label library, so that the target model can further obtain accurate recognition results based on the updated label library, thereby improving the accuracy of data recommendation.

[0022] In combination with the first aspect, in some implementations, at least one modality includes one or more of video, image, text, text recognized through image, and text recognized through voice.

[0023] In the above scheme, when the second device identifies the feature vector of multimodal material data, it can also integrate the feature information of each modality, avoiding the extraction of each modality data separately, which will lead to the inability to comprehensively understand the multimodal characteristics and the correlation between the contents of each modality is not fully utilized.

[0024] In a second aspect, the present application provides a data labeling method, which obtains at least one keyword in material data within a preset time from a material data set; determines the weight of at least one keyword; and updates the labels contained in a label library based on at least one keyword and the weight of at least one keyword. The label library is used for a target model to determine a target label in the label library based on a target feature vector of first material data. The first material data includes at least one modality, and the target feature vector is used to represent feature information contained in at least one modality.

[0025] In combination with the second aspect, in some implementations, the method includes: obtaining the weight of at least one tag in the tag library; determining the correspondence between at least one keyword and at least one tag based on the matching degree between at least one keyword and at least one tag; and updating the weight of at least one tag based on the weight of at least one keyword and the weight of the tag corresponding to the at least one keyword.

[0026] In combination with the second aspect, in some implementations, the method includes: adding the weight of a first keyword in at least one keyword and the weight of a first tag corresponding to the first keyword in a tag library to obtain an updated weight of the first tag.

[0027] In combination with the second aspect, in some implementations, the method includes: based on the absence of a tag corresponding to a second keyword in the tag library, adding the second keyword as a second tag to the tag library.

[0028] In conjunction with the second aspect, in some implementations, the weight of the second keyword is greater than the first threshold

[0029] In combination with the second aspect, in some implementations, based on the weight of a third tag in the at least one updated tag being less than a second threshold, the third tag is deleted from the tag library.

[0030] In combination with the second aspect, in some implementations, the target model is also used to identify multiple modal feature vectors based on the first material data, and the modal feature vectors are used to represent feature information of the corresponding modality; the target model is also used to determine a target feature vector based on the multiple modal feature vectors, and the target feature vector is used to represent feature information contained in the multiple modal feature vectors; the target model is also used to determine a feature vector of at least one tag in the tag library; the target model is also used to determine a target tag in at least one tag based on the target feature vector and the feature vector of at least one tag.

[0031] In combination with the second aspect, in some implementations, material data belonging to labels that meet preset conditions are obtained from a material data set, and the preset conditions include that the increase in the weight of the label after the update is greater than a third threshold, or that the label does not belong to the label library before the update; and the target model is trained based on the material data of the material data belonging to labels that meet the preset conditions.

[0032] In combination with the second aspect, in some implementations, at least one modality includes one or more of video, image, text, text recognized by image, and text recognized by voice.

[0033] In a third aspect, the present application provides an electronic device, characterized in that it includes a processor and a memory, the memory is used to store instructions, and the processor is used to execute instructions. When the processor executes the instructions, it executes the method described in the first aspect or the second aspect.

[0034] In a fourth aspect, the present application provides a computer-readable storage medium, characterized in that instructions are stored in the computer-readable storage medium, and when the instructions are executed on a computing device, the method described in the first aspect or the second aspect is executed.

[0035] In a fifth aspect, the present application provides a computer program product, characterized in that the computer program product includes computer instructions, and when executed by a computing device, the computing device executes the method described in the first aspect or the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments.

[0037] Figure 1A It is a structural diagram of a data annotation model for generating keywords provided in an embodiment of the present application;

[0038] Figure 1B This is a structural diagram of a tag generation module generating keywords provided by an embodiment of the present application;

[0039] Figure 2 This is a schematic diagram of a news interface provided by an embodiment of the present application;

[0040] Figure 3 It is a structural diagram of a data annotation model training and execution provided in an embodiment of the present application;

[0041] Figure 4 It is a structural diagram of a data annotation model provided in an embodiment of the present application running on a cloud server;

[0042] Figure 5 It is a schematic diagram of the architecture of a data annotation system provided in an embodiment of the present application;

[0043] Figure 6 It is a schematic diagram of determining a matching tag by material data features provided by an embodiment of the present application;

[0044] Figure 7 is a schematic diagram of masking a default mode provided by an embodiment of the present application;

[0045] Figure 8 It is a schematic diagram of determining an outbound label and an inbound label provided in an embodiment of the present application;

[0046] Fig. 9 It is a schematic diagram of updating a label library through outbound labels and inbound labels provided by an embodiment of the present application;

[0047] Fig.10 It is a flowchart of a training method for a training target model provided in an embodiment of the present application;

[0048] Fig.11 It is a flowchart of the reasoning process of a data annotation model provided in an embodiment of the present application;

[0049] Fig.12 This is a schematic diagram of a change in keyword tag coverage provided by an embodiment of the present application;

[0050] Fig.13 It is a schematic diagram of changes in the training effect of a training method for a training target model provided in an embodiment of the present application;

[0051] Fig.14 It is a structural schematic diagram of a training device for training a target model provided in an embodiment of the present application;

[0052] Fig.15 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] Illustrative embodiments of the present application include, but are not limited to, data annotation methods, media, and electronic devices.

[0054] As mentioned above, due to the rapid generation of hot topics, the current data annotation model cannot accurately identify the topic category corresponding to the material data of new topics. It can only determine the labels of similar categories for the material data of new topics based on the existing label library, making it impossible to accurately push the material data that users are interested in to users.

[0055] In order to solve the problem that the current data annotation model cannot determine accurate labels for the material data of new topics, the present application provides a data annotation method. Among them, the target model can extract the characteristics of the material data, and determine one or more labels in the label library as the labels of the material data according to its characteristics. For example, the target model can be a data annotation model, a data recommendation model, etc. The data annotation model is used to identify the label corresponding to the material data, and its label is the category corresponding to the material data. After the data annotation model, the data recommendation model also includes, according to the label of the material data, recommending material data with corresponding labels to users. This application does not specifically limit the type of target model, and the following is an example of the target model being a data annotation model.

[0056] The method is applied to the electronic device 100, which will obtain the material data that has recently flowed in, i.e., near-line data, at a preset time. For example, the electronic device can obtain the near-line data of the past week every other week. The electronic device 100 will also obtain keywords of the near-line data and update the tags in the tag library according to the keywords of the near-line data.

[0057] The situation of updating the tags in the tag library includes adding new tags to the tag library. For example, if the keyword appearing in the near-line data does not exist in the tag library, then the keyword is added as a new tag in the tag library.

[0058] Furthermore, the electronic device 100 may also update the weights of the tags in the tag library according to the keywords of the near-line data. For example, the electronic device may count the word frequencies of the keywords in the near-line data and normalize the word frequencies as the weights of the keywords. Then, the weights of the keywords obtained from the near-line data and the weights of the tags in the tag library after the last update are weighted and fused, and then the keywords with higher weights and no corresponding tags in the tag library are saved as new tags.

[0059] Furthermore, optionally, the electronic device 100 may also periodically delete tags with lower weights from the tag library.

[0060] In this way, the label library is regularly updated according to recent material data, so that the labels in the label library can adapt to new hot topics, and the types of labels that can be output by the data annotation model are more in line with current hot topics. Furthermore, when material data related to new topics flow in, the labels corresponding to the material data can be accurately identified according to the updated label library, and the material data that users are interested in can be accurately pushed to users.

[0061] Optionally, after updating the label library, the electronic device 100 will also obtain material data corresponding to labels with large updates, such as material data corresponding to newly added labels and / or material data corresponding to labels with weight increases greater than a growth threshold, and use these material data to incrementally train the data annotation model and adjust the model parameters in the data annotation model. Then, the data annotation model can perform the next inference based on the updated model weights and label library, so that the data annotation model can obtain a recognition result with higher accuracy based on the material data corresponding to the newly added labels and labels with large weight increases.

[0062] Optionally, the extraction of keywords for material data may be performed by the electronic device 100 that trains the data annotation model, or may be performed by the electronic device 200 that executes the data annotation model. In other words, when the data annotation model of the electronic device 200 identifies the material data label, it may synchronously generate keywords for the material data, and then store the keywords and labels of the material data together in the database. Furthermore, the near-line data acquired by the electronic device that trains the data annotation model also includes keywords for each material data, and the electronic device may directly count the keywords and update the label library. Alternatively, the extraction of keywords for material data may also be performed by other devices, and this application does not impose any specific restrictions on this.

[0063] For example, Figure 1A As shown, Figure 1A The data annotation model is used to generate keywords as an example for explanation. The data annotation model 110 can be executed by the electronic device 200, and the label generation module 130 and the training module 140 can be executed by the electronic device 100. The data annotation model 110 can generate labels and keywords corresponding to each material data according to the data stream, and will also store the material data, labels and keywords in the storage module 120. The label generation module 130 will regularly pull keywords from near-line data and can determine the labels to be updated. After updating the label library, the label generation module 130 will also obtain the material data corresponding to the labels with large updates, so that the training module 140 can use these material data to train the data annotation model 110 again, that is, use the material data corresponding to the labels with large updates to perform incremental training on the data annotation model 110. The training module 140 will also send the model weights that have been updated after training to the data annotation model 110.

[0064] For example, Figure 1B As shown, Figure 1B The tag generation module 130 generates keywords as an example. The data annotation model 110 can generate a label corresponding to each material data according to the data stream, and store the material data and its label in the storage module 120. The tag generation module 130 will periodically pull near-line data, identify keywords based on the near-line data, and then determine the labels to be updated. Similarly, the training module 140 can also use the material data corresponding to the labels with large updates to train the data annotation model 110.

[0065] That is to say, if Figure 1A and Figure 1B As shown, the keywords can be synchronously generated and stored in the storage module 120 when the data stream flows into the data annotation model 110, or can be obtained by the label generation module 130 uniformly extracting keywords from the near-line data after a preset time. It should be understood that the present application does not specifically limit the module for generating keywords and the time for generating keywords.

[0066] Optionally, since material data is usually multimodal data, for example, data including multiple modes such as video, image, and text. In order to accurately identify the labels corresponding to the multimodal data, the electronic device will also extract the feature vectors of the multimodal data in each mode respectively. The feature vectors of each mode are then processed, such as input into a transformer model (transformer) for processing and average pooling, and finally a target feature vector that can represent the characteristics of each modality is output. Furthermore, when the electronic device matches the target feature vector with each label in the label library, it can identify labels that are more in line with the multimodal data, which can improve the accuracy of the data annotation model.

[0067] In addition, there may be a modality default in the material data, for example, the material data includes two modalities, video and text, but not images. For material data with modality default, the electronic device first extracts the feature vectors of each modality respectively, and performs mask processing on the default modality, that is, the feature vector of the default modality is represented by a preset mask value. Then, according to the feature vectors of each modality, including the feature vector of the preset mask value, the target feature vector is output.

[0068] For example, if Figure 2As shown, the mobile phone 10 displays a news page on the interface 203, which includes the text title "14:00 on sale! Grab beach volleyball tickets", and the video content 204 includes "Asian Games Beach Volleyball Event", "Tickets will be officially on sale at 14:00 on X / X / 202X", and "Beach Volleyball Competition will be held in Xiangshan". In addition, the mobile phone 10 also displays a display box 201, a search bar 202, and an input box 205. The display box 201 may include one or more signal strength indicators of mobile communication signals, one or more signal strength indicators of WiFi signals, a battery status indicator, and a time indicator. The search bar 202 can be used by users to search other pages, and the input box 205 can be used by users to enter comments.

[0069] When Figure 2 When the news page shown flows into the electronic device as material data, the electronic device will extract the keyword content of the text and video respectively. For example, the text keywords are "beach volleyball" and "tickets", and the video keywords are "Asian Games" and "beach volleyball". If the electronic device does not fuse the features of each modality, it may cause the news page to be labeled with the wrong label. For example, according to the text keywords, the news page is labeled as "beach volleyball" and "sports", etc., while ignoring the information of "Asian Games" in the video. The data labeling method provided by the present application can integrate the feature information of each modality. For example, based on the video keyword "Asian Games" and the text keyword "sand volleyball", it can be comprehensively understood that the user is concerned about topics related to the Asian Games, not just the sport of "beach volleyball". Furthermore, news data about the "Asian Games" can be accurately recommended to users.

[0070] In summary, the data labeling method provided by the present application can regularly update the label library, supplement the labels of new hot topics, and eliminate outdated labels. There is no need to manually update the label library, which saves labor costs. After updating the label library, the corresponding material data incremental training model will also be obtained. Furthermore, when there is a flow of material data related to a new topic, the label corresponding to the material data can also be accurately identified according to the updated label library. In addition, when identifying the feature vector of multimodal material data, the feature information of each mode can also be integrated, avoiding the problem that only extracting each modal data separately will lead to the inability to comprehensively understand the multimodal features and the correlation between the contents of each mode is not fully utilized. Furthermore, after matching the target feature vector after comprehensive understanding with each label in the updated label library, the label corresponding to the material data can be more accurately determined, thereby improving the accuracy of data labeling.

[0071] The following is an explanation of the hardware architecture of a data recommendation system provided by this application. Figure 3 As shown, Figure 3The data annotation system 1000 includes a diagram of the architecture of the training and execution processes. In addition, the data annotation model 110-2 in the electronic device 200 is used as an example to perform keyword extraction. It should be understood that, as mentioned above Figure 1B As described above, the tag generation module 130 may also perform keyword extraction, which will not be described in detail here.

[0072] like Figure 3 As shown, the system 1000 includes: electronic device 100, electronic device 200, data collection device 11, database 12 and client device 13, electronic device 100 is a training device, and electronic device 200 is an execution device. The various components in the system 100 can be connected to each other through a network, where the network can be a wired network or a wireless network, or a mixture of the two.

[0073] The data acquisition device 11 is used to continuously acquire new material data in the application, for example, new news pages in news applications, video content uploaded in short video applications, and newly added discussion posts in forum applications. The newly added material data forms a data stream, and the data acquisition device 11 is also used to input the acquired data stream into the database 12 and store it as a material data set. The material data sets in the database 12 do not necessarily all come from the data acquisition device 11, but may also be received from other devices. The database 12 can be a local database, or a database in the cloud or other third-party platform, which is not specifically limited in this application.

[0074] The database 12 will input the newly acquired material data into the data annotation model 110 - 2 of the electronic device 200 . The data annotation model 110 - 2 will identify the keywords and tags of each material data, and then store the material data and its corresponding keywords in the material data set in the database 12 .

[0075] The database 12 also stores a label library. The multiple labels in the label library can be obtained by the electronic device 100 through regular identification of near-line data by the label generation module 130. The database 12 is also used to input material data to the electronic device 100 so that the electronic device 100 can train the data annotation model 110-1. In addition, if the database 12 obtains that there is a label update in the label library, the database 12 will also obtain and send the material data corresponding to the updated label to the electronic device 100, so that the electronic device 100 can retrain the data annotation model 110-1 to obtain the data annotation model 110-2. In this way, the data annotation model 110-2 can adapt to the changes in the label library and identify the labels corresponding to the material data of the new hot topic.

[0076] The electronic device 100 is used to train the data annotation model 110-1 using the data stream in the database 12, obtain the trained data annotation model 110-2, and send it to the electronic device 200. As mentioned above, after the electronic device 100 obtains the material data corresponding to the label with a large update sent by the database 12, it also incrementally trains the data annotation model 110-1.

[0077] The electronic device 100 uses the label generation model 130, and the electronic device 100 can periodically obtain material data (i.e., near-line data) and keywords corresponding to the material data within a preset time T1 from the database 12, where T1 is a positive number. The label generation module 130 will update the labels in the label library based on the keywords of the near-line data. For example, the electronic device 100 will count the word frequency of each keyword in the near-line data, and normalize the word frequency as the weight of the keyword. Then, the weight of the keyword obtained from the near-line data and the weight of the label in the label library after the last update are weighted and fused, and then the keywords with higher weights and no corresponding labels in the label library are saved as new labels, and labels with lower weights are regularly deleted from the label library.

[0078] The electronic device 200 is used to implement various functions according to the data annotation model 110-2 trained by the electronic device 100. For example, through the data annotation model 110-2, the electronic device 200 can identify the label of the material data, and store the material data and its corresponding label in the database 12, so as to recommend the material data to the user according to the label later. The electronic device 200 can also identify the label corresponding to the user preference data according to the material data (user preference data) that the user likes to browse usually input by the client device 13, and obtain the material data corresponding to the label from the database 12, and output it to the client device 13.

[0079] It is worth noting that Figure 3 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 3 In the embodiment, the database 12 is an external memory relative to the electronic device 100. In other cases, the database 12 may also be placed in the electronic device 100 and / or the electronic device 200. This application does not make any specific limitation to this.

[0080] Furthermore, the electronic device 100 and the electronic device 200 may be the same device or different devices. The electronic device 200 and the client device 13 may be the same device or different devices. This application does not make any specific limitation on this.

[0081] The electronic device 100, the electronic device 200 and the client device 13 can be terminals, such as mobile phone terminals, tablet computers, laptop computers, augmented reality / virtual reality, vehicle-mounted terminals, etc., and can also be servers or cloud devices, etc. For example, physical servers, such as X86 servers, ARM servers, etc. It can also be a virtual machine (VM) based on a general physical server combined with network function virtualization (NFV) technology. A virtual machine refers to a complete computer system with complete hardware system functions simulated by software and running in a completely isolated environment, such as a virtual machine in a cloud platform, which is not specifically limited in this application.

[0082] For example, the electronic device 200 may be a cloud server. Figure 4 As shown, the data annotation system 1000 may also include a cloud server 300 and a model service gateway 301 .

[0083] The model service gateway 301 is used to allocate tasks input to the cloud server 300. Specifically, it includes functions such as model diversion, intelligent routing, and multi-model execution. Specifically, the model service gateway 301 uniformly receives the task data to be input into the cloud server 300, determines the model to be applied to each task, and then sends the task data to the corresponding model. Alternatively, when a task requires multiple models to complete together, the model service gateway 301 can also be used for coordination to send the task data to different models in the cloud server 300 for processing.

[0084] The cloud server 300 includes various model services, such as the data annotation model 110 and the data recommendation model. The specific application of the model can also refer to the aforementioned Figure 3 And its related description. The cloud server 300 has virtual machine resources, which can be virtualized central processing unit (CPU), graphics processing unit (GPU)\neural-network process unit (NPU) and other resources, or cloud container engine (CCE) resources, such as CPU, GPU and NPU. The cloud server 300 also has services such as elastic scaling and unified resource adaptation. The elastic scaling service and the unified resource adaptation service can allocate appropriate computing resources to the current task data according to the workload, so that computing resources will not be wasted.

[0085] The following is an introduction to the architecture of the data annotation system 1000 provided in this application. Figure 5As shown in FIG. 1 , the software structure includes a data annotation model 110, a storage module 120, a label generation module 130, a training module 140, and a training data reconstruction module 150. The data annotation model 110 can be composed of Figure 3 The electronic device 200 is implemented, the storage module 120 can be implemented by the database 12 , and the training data reconstruction module 150 and the training module 140 can be implemented by the electronic device 100 .

[0086] The data annotation model 110 specifically includes a keyword extraction module 111, a tag library 112, and a multi-mode tag generation module 113. The keyword extraction module 111 can be as follows Figure 5 The keyword extraction module 111 is included in the data annotation model 110. In some embodiments, the keyword extraction module 111 may also be independent of the data annotation model 110, or included in the tag generation module 130. Figure 1A and Figure 1B , I will not go into details here.

[0087] After acquiring the input material data, the keyword extraction module 111 will identify the text information contained in each modality, then extract the keywords in the text information of each modality, and send the identified keywords to the storage module 120. Among them, the keyword extraction module can be directly extracted from the text (such as the abstract, title and text), or it can be first recognized by optical character recognition (OCR) technology and / or automatic speech recognition (ASR) to the text in the image and video. Then, the keyword extraction module performs word segmentation on the acquired text, and then predicts the weight of the word, and uses the word with a higher weight as the keyword of the material data. Among them, the preset word weight can be implemented by models such as the prediction model SparTerm and the deep contextualized term weighting framework (deepCT).

[0088] The multiple tags in the tag library 112 may be obtained by first extracting keywords from the preset material data, and then the tag generation module 130 may periodically identify near-line data to update the tag library 112 .

[0089] The multi-modal label generation module 113 is used to extract the feature vectors of each modality using each modality encoder according to the acquired material data, and then aggregate the feature vectors of each modality to output the target feature vector that can represent the characteristics of each modality. Among them, the modality encoder includes a text modality encoder (text encoder), a visual modality encoder (video / image encoder), an audio modality encoder (audio encoder) and an OCR / ASR modality encoder (OCR / ASRencoder). The multi-modal label generation module 113 can input the feature vectors of each modality into the transformer model and perform average pooling processing to obtain the target feature vector.

[0090] like Figure 6 As shown on the left, each modal converter in the multi-modal label generation module 113 identifies the feature vectors of each modality of the material data, including video feature vectors, image feature vectors, text feature vectors, and OCR / ASR feature vectors, and then unifies the understanding of each feature vector to generate a target feature vector. Among them, the feature vectors of each modality can be used to represent the feature information of each modality. For example, for the text modality, the feature vector obtained by the text modality encoder can be used to represent the keyword information contained in the text and the frequency of keyword occurrence and other feature information. For the image modality, the feature vector obtained by the visual modality encoder can be used to represent the feature information contained in the image, such as the type of target object contained in the image (such as text, characters, etc.) and the content of the text contained in the image. The feature vector obtained by the audio modality encoder can be used to represent the text information contained in the audio and the corresponding emotion classification results of the sound and other feature information.

[0091] The data annotation model 110 can determine the label that matches the target feature vector based on the target feature vector generated by the multi-mode label generation module 113 and each label in the label library 112, and the matching label is the label of the material data. After the data annotation model determines the material data label, the material data label can be sent to the storage module 120, or the material data label can be sent to the user, or the material data of the relevant label corresponding to the material data label can be obtained from the storage module 120, and the material data of the relevant label can be sent to the user.

[0092] like Figure 6As shown on the right, the label library includes various labels (such as label 1, label 2, ..., label N, where N is a positive integer) and their corresponding feature vectors. The target feature vector is matched with the label feature vector of each label, and then the label whose matching degree meets the preset value is used as the label matched by the target feature vector, that is, the label of the material data. For example, the data annotation model 110 can be to calculate the similarity between the target feature vector and each label feature vector, sort the similarity, and use the label with a similarity greater than a threshold as the label matched by the target feature vector. Among them, the calculation of the similarity can be specifically any one of the methods of calculating the cosine similarity, the minimum hash value (minhash), the Hamming distance, etc. of the target feature vector and each label feature vector.

[0093] For material data with modality default, the data annotation model 110 will also perform mask processing on the default modality, that is, use a preset mask value to represent the feature vector of the default modality. Then, based on the feature vectors of each modality, including the feature vector of the preset mask value, the target feature vector is output. For example, Figure 7 As shown, for material data with video and text (visual text), the image and OCR / ASR need to be masked. For material data with video, image and text (visual text), but there is no text information in the audio of the video or there is no text in the video and image, OCR / ASR needs to be masked. For material data with images and text (graphic text), the video and OCR / ASR need to be masked. For material data with only text, the image, video and OCR / ASR all need to be masked.

[0094] The storage module 120 stores a tag library and a material database, and the material database also includes keywords extracted from material data by the keyword extraction module 111. The storage module 420 can periodically send keywords of near-line data to the tag generation module 130. Optionally, the storage module 420 can be a distributed storage system Hadoop.

[0095] The tag generation module 130 counts the frequency of keywords in the near-line data, and performs weighted fusion of the frequency of keywords obtained from the near-line data and the weight of the tags in the tag library after the last update. Keywords with higher weights are then saved as new tags, and tags with lower weights are deleted from the tag library. Optionally, the tag generation module 130 can also normalize the frequency of keywords, and then use the normalized frequency as the weight of the keyword.

[0096] Among them, the weighted fusion of keywords and labels specifically includes multiplying the weight of the keyword and the weight of the corresponding label by the corresponding coefficient and adding them, and storing the obtained result as the new weight of the label in the label library. The label generation module 130 can determine whether the two match according to whether the keyword and the label are the same or whether they are synonyms. If there is a match between the keyword and the label, it is determined that there is a corresponding relationship between the keyword and the corresponding label. If there is no match between the keyword and any label, it is determined that there is no corresponding relationship between the keyword and any label in the label library. For example, the label generation module 130 can determine whether the keyword and the label are synonyms based on a preset synonym thesaurus, and determine the label that is a synonym of the keyword in the label library as the label matched by the keyword. The label generation module 130 can also first determine the word vector corresponding to each keyword and each label, and by calculating the similarity between the word vectors of each keyword and each label, the label with a similarity greater than a threshold is used as the label matched by the corresponding keyword. The present application does not specifically limit the method for matching labels and keywords.

[0097] The coefficients corresponding to keywords and tags respectively indicate the importance of keywords and tags. For application scenarios with fast content updates, such as news recommendation application scenarios, the coefficients corresponding to keywords can be higher than the coefficients corresponding to tags, so that the tag library can adapt to the current new hot topic content in a timely manner. For application scenarios with slow content updates, such as product recommendation application scenarios, the coefficients corresponding to keywords can be lower than the coefficients corresponding to tags, so that the tag library can save as much classic product content as possible. In other words, the size of the coefficients corresponding to keywords and tags can be set according to the application scenario, and this application does not limit their specific size.

[0098] For keywords that do not have corresponding tags in the tag library, the keyword weight or the keyword weight multiplied by the corresponding coefficient is used as the weight of the keyword in the tag library, and then the keyword and its weight are stored in the tag library as entry tags. Alternatively, after calculating the weight of the keyword that does not have a corresponding tag, it is also determined whether its weight is greater than the threshold, and the keyword greater than the threshold is stored in the tag library as an entry tag. After calculating the weighted fusion of all keywords and tags, the tag with a lower weight is deleted from the tag library as an exit tag according to the tag weight.

[0099] For example, Figure 8As shown, the keyword extraction module 111 will extract near-line data from the database 12 to obtain recent material data, such as material data 1 to material data M in the past week, where M is a positive integer. The keyword extraction module 111 can extract keywords 1 to keyword L based on material data 1 to material data M, where L is a positive integer. Then the label generation module 130 will count the frequency of each keyword appearing in the near-line data to obtain the word frequency F1 to FL corresponding to keyword 1 to keyword L. Then the label generation module 130 will normalize the word frequency as the weight of the keyword, and weight the keyword weight and the label weight to perform weighted fusion. For example, a keyword and a label with exactly the same or similar meaning can be weighted fused. For example, the keyword "smart phone" and the label "smart phone" are synonyms, and the two match each other. The weight of the keyword "smart phone" and the weight of the label "smart phone" can be multiplied by the corresponding coefficient and added. Then the keywords that have no corresponding labels and whose weights are greater than α are stored in the label library as storage labels. After weighted fusion, the keywords with weights less than β are deleted from the label library.

[0100] For example, Fig. 9 As shown, α can be 0.001 and β can be 0.00001. The tag generation module 130 stores keywords with no corresponding tags in the tag library and with weights greater than 0.001, such as folding screen and fifth-generation mobile communication technology (5G), as entry tags in the tag library. Then, keywords with weights less than 0.00001 in the tag library, such as Fig. 9 The "universal charger" and "VCD" (video compact disc) shown in the examples are deleted from the label library as outbound labels.

[0101] Moreover, after updating the label, the label generation module 130 will also send the updated label to the training data reconstruction module 150, and the training data reconstruction module 150 will also obtain the material data corresponding to the updated label from the storage module 120, such as the material data of the newly added label and the material data of the label with a larger weight increase.

[0102] After the training data reconstruction module 150 obtains the material data corresponding to the updated label, it will generate a new training data set, and the new data set will be sent to the training module 140. The training module 140 is used to train the data annotation model 110 according to the new training data set to improve the accuracy of the data annotation model. The trained data annotation model 110 can adapt to the new label changes, and can also identify the specific labels of the material data corresponding to the newly added labels and the material data of the labels with larger weight increases.

[0103] The training module 140 can train the data labeling model 110 in any one or more of the following ways: full supervision, semi-supervision, weak supervision, etc. Among them, full supervision means that the material data used as training data are all pre-labeled with accurate labels. Semi-supervision means that part of the material data used as training data has pre-labeled labels, and part has no labels. Weak supervision means that the material data used as training data has some incomplete and inaccurate labels. Taking full-supervision training as an example, the electronic device 100 can compare the output results of the data labeling model 110 with the labels of the material data when training the model, and continuously adjust the structural parameters (referred to as model parameters) of the data labeling model 110 according to the comparison results, until the value of the model loss function (loss value) meets the preset loss value, thereby completing the training of the data labeling model. This application does not limit the specific method of training the model.

[0104] Moreover, when training the data labeling model 110, the training module 140 can also randomly mask the modal feature vectors of the material data, so that after the data labeling model 110 is trained, it can accurately determine the target feature vectors of material data with modal defaults, and further accurately identify the corresponding labels.

[0105] In summary, the data recommendation system provided by the present application can regularly update the tag library, supplement the tags of new hot topics, and eliminate outdated tags. There is no need to manually update the tag library, which saves labor costs. In addition, when identifying the feature vectors of multimodal material data, the feature information of each mode can also be integrated, avoiding the extraction of each modal data separately, which will lead to the inability to comprehensively understand the characteristics of the multimodality, and the correlation between the contents of each mode is not fully utilized. Furthermore, after matching the target feature vector of the material data with each label in the updated tag library, the label corresponding to the material data can be determined more accurately, thereby improving the accuracy of data recommendation.

[0106] The following describes the method flow of the data annotation method provided by the present application. The method can be applied to the aforementioned electronic device 100, and takes the target model as the data annotation model 110 as an example. Fig.10 As shown, the method includes:

[0107] S1001: Acquire near-line data and keywords of the near-line data.

[0108] The material data within the preset time T1 is the near-line data, such as the material data of the past week or the past month. The material data can be acquired by the aforementioned data acquisition device 11 and stored in the database 12. The electronic device 100 can automatically acquire near-line data regularly at a preset time T2, where T2 is a positive number, or acquire near-line data after the trigger condition of the acquired user is obtained, and this application does not impose specific restrictions on this.

[0109] The electronic device 100 will also obtain the keywords of each material data in the near-line data. Figure 3 The near-line data obtained from the database 12 shown in the figure may be a keyword identified by the keyword extraction module 111 of another electronic device 200. Figure 3 , Figure 5 and its related description.

[0110] In some embodiments, the keywords of the near-line data can also be recognized by the electronic device 100. Specifically, after the electronic device 100 obtains the near-line data, it will identify the text information contained in each material data in each modality, and then extract the keywords in the text information of each modality. The text can be directly extracted from the text (such as the abstract, title, and text), or the text in the image and video can be first recognized through OCR technology and / or ASR. Then, the electronic device 100 performs word segmentation on the obtained text, and then predicts the weight of the words, and uses the words with higher weights as the keywords of the material data. Among them, the preset word weights can be implemented by prediction models such as SparTerm or deepCT. For details, please refer to Figure 5 and its related description.

[0111] S1002: Determine the weight of each keyword in the near-line data.

[0112] After the electronic device 100 obtains the keyword of the near-line data, it will determine the weight of the keyword in the near-line data. The weight can be determined according to the frequency of the keyword appearing in the near-line data (referred to as word frequency). After the electronic device 100 obtains the word frequency of each keyword, it can also normalize the word frequency, and the word frequency after normalization is the weight of the keyword.

[0113] S1003: Update the tag library according to the weight of the keyword and the weight of the tag in the tag library.

[0114] The initial tags in the tag library can be obtained by extracting keywords from the preset material data in advance. After the electronic device 100 obtains the near-line data, it will also determine whether there is a tag matching the keyword in the tag library, and then update the tag library according to the matching relationship between the keyword and the tag in the tag library.

[0115] Specifically, the electronic device 100 can determine whether the keywords and tags match based on whether the keywords and tags are the same. It is also possible to determine whether the keywords and tags are synonyms based on a preset synonym thesaurus, and determine the tags that are synonyms of the keywords in the tag library as tags that match the keywords. The tag generation module 130 can also first determine the word vectors corresponding to each keyword and each tag, and by calculating the similarity between the word vectors of each keyword and each tag, use the tags with a similarity greater than a threshold as tags that match the corresponding keywords. It should be understood that the present application does not impose specific restrictions on the method of matching tags and keywords.

[0116] If the keyword has no matching tag in the tag library, the keyword can be added to the tag library as a new tag. Optionally, the electronic device 100 also determines whether the weight of the keyword that has no corresponding tag in the tag library is greater than a threshold α, and only adds the keyword to the tag library when its weight is greater than the threshold α.

[0117] Optionally, if there is a matching tag for the keyword in the tag library, the electronic device 100 will add the weight of the keyword and the weight of the tag that matches it to the corresponding coefficients to update the weight of the original tag in the tag library. The coefficients corresponding to the keyword and the tag respectively represent the importance of the keyword and the tag, which can be set according to the application scenario. For related descriptions, please refer to the introduction of the tag generation module 130 mentioned above.

[0118] Optionally, after updating the weights of the tags in the tag library, the electronic device 100 may also delete tags with lower weights in the tag library at a preset time T3. T3 is a positive number. In some embodiments, T3 may be greater than or equal to T2. In other words, the electronic device 100 may add tags corresponding to newly appearing keywords to the tag library at the same frequency, and delete tags with lower weights from the tag library. Alternatively, the electronic device 100 may add tags corresponding to newly appearing keywords to the tag library at a higher frequency, and delete tags with lower weights from the tag library at a lower frequency.

[0119] Furthermore, the number of tags that the electronic device 100 deletes from the tag library at the preset time T3 can be determined by a preset number or by the number of tags added to the tag library within T3. For example, within T3, the electronic device 100 adds P tags to the tag library, where P is a positive integer, and the electronic device 100 can delete P tags with lower weights. This ensures that the number of tags in the tag library is consistent before and after T3, so that the operating efficiency of the data annotation model 110 is basically consistent at different times.

[0120] Optionally, the electronic device 100 determines whether the newly added tag or the deleted tag meets the preset standard, and only performs the corresponding adding or deleting operation for the tag that meets the standard. The preset standard can be determined based on a file preset by the user, can be pre-loaded before running, or can be input through other ports. For example, for a newly added tag, the electronic device 100 will also determine whether it is a sensitive word. If it is a sensitive word, it will not be added to the tag library.

[0121] S1004: According to the updated label library, the data annotation model 110 is trained using material data that meets the preset conditions.

[0122] After updating the label library, the electronic device 100 will obtain the material data corresponding to the labels that meet the preset conditions, use these material data as training samples, generate a new training data set, and then train the data annotation model 110. The preset conditions include: newly added labels, labels whose weight growth value is greater than the threshold θ, and θ is a positive number.

[0123] Optionally, when determining the training data set, the electronic device 100 may also determine an equal amount of material data for each tag that meets the preset conditions. Furthermore, when the storage space reserved for the training data annotation model 110 of the electronic device 100 is insufficient to completely store the training data set, the material data corresponding to the tags with higher weights among the tags that meet the preset conditions will be stored preferentially.

[0124] Optionally, when using the training data set to train the data annotation model 110, the electronic device 100 can adopt any one or more of the methods of full supervision, semi-supervision, weak supervision, etc. That is to say, the training data set may include labeled data or unlabeled data. Among them, for unlabeled data, the labels and keywords predicted by the previous data annotation model 110 can be used as the initial labeled data for its training. For details on the methods of full supervision, semi-supervision, weak supervision, etc., please refer to the aforementioned description of the training module 140, which will not be repeated here.

[0125] Optionally, the electronic device 100 may also determine the method of training the data labeling model 110 according to the magnitude of the label library. For example, for news scenarios, the number of labels contained in its label library is relatively large, usually tens of thousands, and the electronic device 100 may adopt a fully supervised training method, that is, the training data set contains labeled data. For advertising scenarios, the number of labels contained in its label library is relatively small, usually hundreds or thousands, and the electronic device 100 may adopt a semi-supervised or weakly supervised training method, that is, the training data set contains unlabeled data.

[0126] Optionally, when the electronic device 100 uses the training data set to train the data annotation model 110, it can specifically include a training phase and an update label phase. The training phase includes using the training data set to train the data annotation model 110, and adjusting the model parameters based on the value of the loss function. The update label phase includes the electronic device 100 keeping the model parameters unchanged, using the training data set for reasoning, predicting one or more labels for each training sample, and then adding or deleting the predicted one or more labels. The electronic device 100 will also loop through the training phase and the update label phase until the value of the loss function is less than a preset loss value. Among them, the loss function can use a cross-entropy function (cross-entropy), a mean square error (MSE), etc.

[0127] Among them, the electronic device 100 updates the label stage, which specifically includes: keeping the model parameters unchanged, using the training data set for inference, and obtaining the predicted label of each training sample. For the predicted label of each training data, the predicted label is compared with the annotated label of the training sample itself, and the confidence of the predicted label is obtained by weighting respectively. Then, the electronic device 100 retains the predicted labels with higher confidence for the training samples, deletes the predicted labels with low confidence, and can also select labels with high confidence from the predicted labels to add to the training sample annotations. The electronic device 100 updates the label stage can help improve the annotation quality of the annotated data, or add annotation labels to the unannotated data.

[0128] Optionally, when training the data annotation model 110, the electronic device 100 may also randomly mask the modal feature vector of the material data, and ensure that the number of modes retained during random masking is greater than 1. Thus, after the data annotation model 110 is trained, it can accurately determine the target feature vector of material data with modal defaults, and further accurately identify its corresponding label.

[0129] It should be understood that in some embodiments, the electronic device 100 will execute steps S1001 to S1004 again after the preset time T2, so that the electronic device 100 can continuously update the tag library and update the data annotation model.

[0130] In summary, the data annotation system provided by the present application can regularly update the tag library, add tags for new hot topics, and eliminate outdated tags. There is no need to manually update the tag library, saving labor costs. Furthermore, the data annotation model 110 can be updated according to the updated tag library, so that the data annotation model 110 can more accurately determine the tags corresponding to the material data, thereby improving the accuracy of data recommendation.

[0131] Combine the following Fig.11The inference process of the data annotation model involved in the training method of the training target model provided in the present application is introduced, and the method process can be applied to the electronic device 200. Among them, the electronic device 200 and the electronic device 100 can be the same device or different devices.

[0132] like Fig.11 As shown, the method specifically includes:

[0133] S1101: Determine multiple unimodal feature vectors of the acquired material data.

[0134] When the electronic device 200 obtains the newly-inflow material data, each modal encoder is used to extract the unimodal feature vector of each modality. Among them, the modal encoder includes a text modal encoder, a visual modal encoder, an audio modal encoder and an OCR / ASR modal encoder. The corresponding unimodal feature vector includes a text feature vector, an image feature vector, an audio feature vector, and an OCR / ASR feature vector.

[0135] For example, a text modality encoder, a visual modality encoder, an audio modality encoder, and an OCR / ASR modality encoder may also be based on a transformer model or a transformer-based bidirectional encoder (bidirectional encoder representations from transformer, BERT), etc., and this application does not make any specific limitations on this.

[0136] For example, the text modality encoder can be based on the transformer. After the text in the material data (such as the title, text and other text information) is serialized, the start of header (SOH) and end of transmission (EOT) flags are added to the beginning and end of the serialized text. Then, the serialized text is added to the position parameter and input into the transformer for processing. The output sequence is a text feature vector.

[0137] The visual modality encoder can also be based on the transformer. The video modality and image modality share the visual modality encoder for feature extraction. The input visual modality (video modality or image modality) is first serialized and expanded to obtain a feature sequence, and after adding a learnable token (class token, cls_token) and position parameters, it is output to the transformer for processing. Among them, before being input into the timing transformer module for processing, the feature sequence also needs to be added with the timing parameters, and finally the image feature vector is generated according to the transformer result.

[0138] The audio modality encoder can generate an audio feature vector through VGGish, a deep neural network for audio signal processing. Alternatively, the audio can be converted into text and processed by the text modality encoder to generate an audio feature vector.

[0139] The OCR / ASR modality encoder is based on the transformer. The text in the video and image is extracted by the OCR method, and the ASR text is obtained after the speech recognition in the audio. Then the two are input into the transformer in the form of text, which can be referred to as the text encoder, and then the OCR / ASR feature vector is obtained.

[0140] S1102: Determine a target feature vector based on multiple unimodal feature vectors.

[0141] The electronic device 200 will uniformly understand each unimodal feature vector and generate a target feature vector. For example, the feature vectors of each modality can be input into the transformer model and average pooled to obtain the target feature vector. Alternatively, the target feature vector can be obtained by performing an outer product on the unimodal feature vector. For details, please refer to the aforementioned Figure 6 And related description about the multi-mode label generation module 113.

[0142] In some embodiments, step S1101 and step S1102 may also be performed simultaneously, or the electronic device 100 may first execute step S1101 and then execute step S1102.

[0143] S1103: Determine the tag feature vector of each tag in the tag library.

[0144] The electronic device 200 inputs each tag of the tag library into the tag encoder to obtain a tag feature vector corresponding to each tag. The tag encoder structure can be based on transformer or BERT, etc., and the text encoder in the aforementioned step S1101 can be specifically referred to.

[0145] S1104: Match the target feature vector with the tag feature vectors of each tag, and determine the matching tags in the tag library.

[0146] The electronic device 200 matches the target feature vector with the tag feature vectors of each tag, and then uses the tag whose matching degree meets the preset value as the tag matched by the target feature vector, that is, the tag of the material data. For example, the target feature vector and each tag feature vector may be calculated for similarity, and after the similarities are sorted, the tags with similarities greater than a threshold value may be used as the tags matched by the target feature vector.

[0147] For material data with default modalities, the data annotation model of the electronic device 200 will also mask the default modalities, that is, use a preset mask value to represent the feature vector of the default modalities. Then, based on the feature vectors of each modality, including the feature vectors of the preset mask value, the target feature vector is output. For details, please refer to the aforementioned Figure 7 The related descriptions will not be repeated here.

[0148] In some embodiments, after determining the tag corresponding to the material data, the electronic device 200 may also store the material data and its tag in the database 12. The electronic device 200 may also recommend the material data corresponding to the tags preferred by the user in the database 12 to the user in combination with the tags preferred by the user, or the electronic device 200 may also sort the tags according to the frequency of occurrence based on the near-line data and push the sorting results to the user.

[0149] In summary, the data annotation model provided by the present application can also integrate the feature information of each modality when identifying the feature vector of multimodal material data, avoiding the extraction of each modality data separately, which will lead to the inability to comprehensively understand the multimodal features and the correlation between the contents of each modality is not fully utilized. Furthermore, after matching the target feature vector of the material data with each label in the updated label library, the label corresponding to the material data can be determined more accurately, improving the accuracy of data recommendation.

[0150] The following describes the experimental results of the method provided by this application in practical applications. Fig.12 As shown, Fig.12 The method provided in this application provides a bar graph of the overlap rate of the news dynamic keyword tag library and the true value tag set in the news recommendation scenario as the number of updates changes, that is, a schematic diagram of the change in the coverage rate of the dynamic keyword tag library. Among them, the initial coverage rate of the tag library is about 58.1%, about 59.1% after the first update, about 59.5% after the second update, and about 61.5% after the third update. That is, as the tag library is continuously updated, the coverage rate of the tag library will become higher and higher, indicating that this method can automatically mine unknown tags in the supplementary data stream.

[0151] Fig.13 A schematic diagram of the training effect of the method when training the model is shown, which shows that when news is used as the test set and other information flows are used as the test set, the reasoning accuracy of the model will gradually increase as the model is trained.

[0152] Table 1 below shows a comparison of the data annotation model trained by this method and natural language processing (NLP) on the news test set. Among them, the data annotation model trained by this method has an accuracy rate of 29.1% and a recall rate of 21.5% compared with NLP. Therefore, the labels obtained by the data annotation model trained by this method are more accurate and complete.

[0153] Model Accuracy (%) Recall rate (%) NLP 44.6 36.4 Data Annotation Model 73.7 57.9

[0154] Table 2 below shows the results of the relevant indicator tests of the data annotation model trained by this method on the advertising test set. As shown in Table 2, the accuracy of the data annotation model trained by this method is improved by 6.66% and the recall rate is improved by 6.52% compared with the baseline model.

[0155] Therefore, the labels obtained by the data annotation model trained by this method are more accurate and complete.

[0156] Model Accuracy (%) Recall rate (%) baseline 59.56 73.77 Data Annotation Model 66.22 80.29

[0157] In order to solve the problem that the above data annotation model cannot determine accurate labels for material data of new topics, the present application provides a data annotation device 1400, such as Fig.14 As shown, the data labeling device 1400 includes an acquisition unit 1410 , a determination unit 1420 , and an update unit 1430 .

[0158] The acquisition unit 1410 is used to acquire at least one keyword in the material data within a preset time from the material data set. The determination unit 1420 is used to determine the weight of at least one keyword. The update unit 1430 is used to update the label contained in the label library according to the at least one keyword and the weight of at least one keyword, and the label library is used for the target model according to the target feature vector of the first material data. The determination unit 1420 determines the target label in the label library, the first material data includes at least one mode, and the target feature vector is used to represent the feature information contained in the at least one mode.

[0159] In some embodiments, the acquisition unit 1410 is further used to acquire the weight of at least one tag in the tag library. The determination unit 1420 is further used to determine the correspondence between at least one keyword and at least one tag according to the matching degree between at least one keyword and at least one tag; the update unit 1430 is further used to update the weight of at least one tag according to the weight of at least one keyword and the weight of the tag corresponding to the first keyword.

[0160] In some other embodiments, the updating unit 1430 is further configured to add the weight of the first keyword in the at least one keyword and the weight of the first tag corresponding to the first keyword in the tag library to obtain the updated weight of the first tag.

[0161] In some other embodiments, the updating unit 1430 is further configured to add the second keyword as a second tag to the tag library based on the absence of a tag corresponding to the second keyword in the tag library for the second keyword among the at least one keyword.

[0162] In some other embodiments, the weight of the second keyword is greater than the first threshold

[0163] In some other embodiments, the updating unit 1430 is further configured to delete the third tag from the tag library based on the weight of the third tag in the at least one updated tag being less than the second threshold.

[0164] In other embodiments, the target model is also used to identify multiple modal feature vectors based on the first material data, and the modal feature vectors are used to represent feature information of the corresponding modality; the target model is also used to determine a target feature vector based on the multiple modal feature vectors, and the target feature vector is used to represent feature information contained in the multiple modal feature vectors; the target model is also used to determine a feature vector of at least one tag in a tag library; the target model is also used to determine a target tag in at least one tag based on the target feature vector and the feature vector of at least one tag.

[0165] In other embodiments, the acquisition unit 1410 is also used to acquire material data belonging to a label that meets preset conditions from the material data set, the preset conditions including that the increase in the weight of the label after the update is greater than a third threshold, or that the label does not belong to the label library before the update; the determination unit 1420 is also used to train the target model based on the material data of the material data belonging to the label that meets the preset conditions.

[0166] In other embodiments, at least one modality includes one or more of video, image, text, text recognized through image, and text recognized through voice.

[0167] In summary, the data recommendation system provided by the present application can regularly update the tag library, supplement the tags of new hot topics, and eliminate outdated tags. There is no need to manually update the tag library, which saves labor costs. In addition, when identifying the feature vectors of multimodal material data, the feature information of each mode can also be integrated, avoiding the extraction of each modal data separately, which will lead to the inability to comprehensively understand the characteristics of the multimodality, and the correlation between the contents of each mode is not fully utilized. Furthermore, after matching the target feature vector of the material data with each label in the updated tag library, the label corresponding to the material data can be determined more accurately, thereby improving the accuracy of data recommendation.

[0168] The method of the embodiment of the present application is described in detail above. In order to facilitate better implementation of the above scheme of the embodiment of the present application, correspondingly, related equipment for cooperating in implementing the above scheme is also provided below.

[0169] Fig.15 1 is a schematic diagram of the structure of an electronic device 1500 provided in the present application. The electronic device 1500 may be the electronic device 100, the electronic device 200 and the cloud server 300 in the aforementioned content. Fig.15 As shown, the electronic device 1500 includes: a processor 1510, a communication interface 1520 and a memory 1530. Among them, the processor 1510, the communication interface 1520 and the memory 1530 can be connected to each other through an internal bus 1540, and can also communicate through other means such as wireless transmission. In the embodiment of the present application, the connection through the bus 1540 is taken as an example. The bus 1540 can be a peripheral component interconnect Express (PCIe) bus, or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a computer express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc. The bus 1540 can be divided into an address bus, a data bus, a control bus, etc. In addition to the data bus, the bus 1540 can also include a power bus, a control bus, and a status signal bus. However, for the sake of clarity, various buses are marked as bus 1540 in the figure.

[0170] The processor 1510 may be composed of at least one general-purpose processor, such as a CPU, or a combination of a CPU and a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The processor 1510 executes various types of digital storage instructions, such as software or firmware programs stored in the memory 1530, which enables the electronic device 1500 to provide a variety of services.

[0171] The memory 1530 is used to store program codes, and is controlled by the processor 1510 to execute the processing steps of the data labeling method in the above embodiment. The program code may include one or more software modules, which may be Fig.14 The software modules provided in the embodiments.

[0172] It should be noted that this embodiment can be implemented by a general physical server, for example, an ARM server or an X86 server, or it can be implemented by a virtual machine based on a general physical server combined with NFV technology. A virtual machine refers to a complete computer system with complete hardware system functions simulated by software and running in a completely isolated environment. This application does not make any specific limitations.

[0173] The memory 1530 may include a volatile memory, such as a random access memory (RAM); the memory 1530 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); the memory 1530 may also include a combination of the above types. The memory 1530 may store program codes, specifically executing Fig.10 Steps S1001 to S1004 in the embodiment, and Fig.11 Steps S1101 to S1104 and their optional steps in the embodiment will not be described in detail here.

[0174] The communication interface 1520 can be a wired interface (such as an Ethernet interface), an internal interface (such as a high-speed serial computer expansion bus (Peripheral Component Interconnect Express, PCIe) bus interface), a wired interface (such as an Ethernet interface) or a wireless interface (such as a cellular network interface or a wireless local area network interface) for communicating with other devices or modules.

[0175] Need to explain, Fig.15 This is only a possible implementation of the embodiment of the present application. In practical applications, the electronic device 1500 may also include more or fewer components, which is not limited here. Fig.10 as well as Fig.11 The relevant explanations in the embodiments will not be repeated here.

[0176] It should be understood that Fig.15 The computing device shown may also be a computer cluster consisting of at least one server, which is not specifically limited in this application.

[0177] The present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a processor, Fig.10 as well as Fig.11 The method flow shown is realized.

[0178] The present application also provides a computer program product. When the computer program product is run on a processor, Fig.10 as well as Fig.11 The method flow shown is realized.

[0179] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media sets. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a high-density digital video disc (DVD), or a semiconductor medium. The semiconductor medium may be an SSD.

[0180] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A data labeling method, characterized in that: Applied to a first device and a second device, the method includes: The first device obtains at least one keyword in the material data within a preset time from the material data set; The first device determines a weight of the at least one keyword; The first device updates the tags included in the tag library according to at least one keyword and a weight of the at least one keyword; The second device determines a target feature vector of first material data, the first material data includes at least one modality, and the target feature vector is used to represent feature information included in the at least one modality; The second device determines, in the label library, a target label corresponding to the first material data according to the target feature vector.

2. The method according to claim 1, characterized in that The first device updates the tags included in the tag library according to at least one keyword and the weight of the at least one keyword, including: The first device obtains a weight of at least one tag in the tag library; The first device is further configured to determine a correspondence between the at least one keyword and the at least one tag according to a matching degree between the at least one keyword and the at least one tag; The first device updates the weight of the at least one tag according to the weight of the at least one keyword and the weight of the tag corresponding to the at least one keyword.

3. The method according to claim 2, characterized in that The first device updates the weight of the at least one tag according to the weight of the at least one keyword and the weight of the tag corresponding to the at least one keyword, including: The first device weights and adds the weight of a first keyword in the at least one keyword and the weight of a first tag in the tag library corresponding to the first keyword to obtain an updated weight of the first tag.

4. The method according to any one of claims 1 to 3, characterized in that: The first device updates the tags included in the tag library according to at least one keyword and the weight of the at least one keyword, including: The first device adds a second keyword among the at least one keyword as a second tag to the tag library based on the fact that there is no tag corresponding to the second keyword in the tag library.

5. The method according to claim 4, characterized in that The weight of the second keyword is greater than a first threshold.

6. The method according to claim 2 or 3, characterized in that: The first device updates the tags included in the tag library according to at least one keyword and the weight of the at least one keyword, including: The first device deletes the third tag from the tag library based on the weight of the third tag in the at least one updated tag being less than a second threshold.

7. The method according to claim 1, characterized in that The second device includes a target model, The second device determines a target feature vector of the first material data, including: The second device obtains a plurality of modal feature vectors of the first material data through the target model recognition, wherein the modal feature vectors are used to represent feature information of corresponding modalities; The second device determines the target feature vector according to the multiple modal feature vectors through the target model; The second device determines, according to the target feature vector, a target label corresponding to the first material data in the label library, including: The second device determines a feature vector of at least one tag in the tag library by using the target model; The second device determines the target tag in the tag library according to the target feature vector and the feature vector of the at least one tag through the target model.

8. The method according to claim 7, characterized in that: Also includes: The first device obtains material data belonging to a tag that meets a preset condition from the material data set, wherein the preset condition includes that an increase in a weight of the tag after update is greater than a third threshold, or that the tag does not belong to the tag library before update; The first device trains the target model according to the material data of the material data belonging to the label that meets the preset condition.

9. The method according to claim 1, characterized in that: The at least one modality includes one or more of video, image, text, text recognized through image, and text recognized through voice.

10. A data annotation method, characterized in that: The method comprises: From the material data set, obtaining at least one keyword in the material data within a preset time; determining a weight of the at least one keyword; According to at least one keyword and the weight of the at least one keyword, the labels contained in the label library are updated, and the label library is used for the target model to determine the target label in the label library according to the target feature vector of the first material data, the first material data includes at least one modality, and the target feature vector is used to represent the feature information contained in the at least one modality.

11. The method according to claim 10, characterized in that The updating of the tags included in the tag library according to the at least one keyword and the weight of the at least one keyword includes: Obtaining a weight of at least one tag in the tag library; Determining a correspondence between the at least one keyword and the at least one tag according to a matching degree between the at least one keyword and the at least one tag; The weight of the at least one tag is updated according to the weight of the at least one keyword and the weight of the tag corresponding to the at least one keyword.

12. The method according to claim 11, characterized in that The updating of the tags included in the tag library according to the at least one keyword and the weight of the at least one keyword includes: The weight of the first keyword in the at least one keyword and the weight of the first tag in the tag library corresponding to the first keyword are weighted and added together to obtain an updated weight of the first tag.

13. The method according to any one of claims 10 to 12, characterized in that: The updating of the tags included in the tag library according to the at least one keyword and the weight of the at least one keyword includes: Based on the fact that a tag corresponding to a second keyword of the at least one keyword does not exist in the tag library, the second keyword is added to the tag library as a second tag.

14. The method according to claim 13, characterized in that The weight of the second keyword is greater than a first threshold.

15. The method according to claim 11 or 12, characterized in that: The method further comprises: Based on the fact that the weight of a third tag in the at least one updated tag is less than a second threshold, the third tag is deleted from the tag library.

16. The method according to claim 10, characterized in that: The target model is also used to identify and obtain a plurality of modal feature vectors according to the first material data, wherein the modal feature vectors are used to represent feature information of corresponding modes; The target model is further used to determine a target feature vector according to the multiple modal feature vectors, wherein the target feature vector is used to represent feature information included in the multiple modal feature vectors; The target model is also used to determine a feature vector of at least one tag in the tag library; The target model is further used to determine a target tag in the at least one tag according to the target feature vector and the feature vector of the at least one tag.

17. The method according to claim 16, characterized in that: The method further comprises: Acquire material data belonging to a tag that meets a preset condition from the material data set, wherein the preset condition includes that an increase in the weight of the tag after the update is greater than a third threshold, or that the tag does not belong to the tag library before the update; The target model is trained based on the material data of the material data belonging to the label that meets the preset conditions.

18. The method according to claim 10, characterized in that: The at least one modality includes one or more of video, image, text, text recognized through image, and text recognized through voice.

19. A readable medium, characterized in that The readable medium contains instructions, which, when executed by a processor of an electronic device, enable the electronic device to implement the method of any one of claims 10 to 18.

20. An electronic device, characterized in that: include: a memory for storing instructions to be executed by one or more processors of the electronic device; and a processor, which is one of the processors of the electronic device, configured to execute the instructions so that the electronic device implements the method according to any one of claims 10 to 18.

Images