Entity labeling method and device and readable storage medium
By combining a pre-set entity annotation model with a verification interface, high-confidence entities are automatically verified and low-confidence entities are manually reviewed, solving the problems of low efficiency and high cost in traditional entity annotation and achieving efficient and low-cost entity annotation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-22
- Publication Date
- 2026-03-13
AI Technical Summary
Traditional entity annotation methods are inefficient, time-consuming, costly, and require a large amount of manpower.
The initial entity data is labeled using a pre-defined entity labeling model, and the confidence score is performed through the target verification interface. Entities with high confidence are automatically added to the pre-defined entity library, while entities with low confidence are entered into the manual review pool.
It improves entity annotation efficiency, reduces costs, and decreases the amount of manual annotation.
Smart Images
Figure CN121659937A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer processing, and in particular to an entity annotation method, apparatus, and readable storage medium. Background Technology
[0002] Entity annotation is a crucial task in natural language processing and machine learning. Its purpose is to identify entities with specific meanings in text and assign them predefined labels. The accuracy of entity annotation significantly impacts subsequent tasks such as information extraction, question answering systems, and machine translation. It should be understood that entity annotation typically involves large datasets. Traditional entity annotation relies on manual annotation of the entire dataset, which is inefficient, time-consuming, and requires substantial manpower, resulting in high costs. Summary of the Invention
[0003] This invention provides an entity annotation method, apparatus, and readable storage medium, which effectively improves the efficiency of entity annotation and reduces the cost of entity annotation.
[0004] In a first aspect, embodiments of the present invention provide an entity annotation method, including: Obtain the initial entity data to be labeled; The initial entity data is input into a preset entity annotation model, and each entity in the initial entity data is annotated by the preset entity annotation model to obtain entity annotation results. The entity annotation results include the entity type of each entity. Based on the entity type of each entity, the target verification interface corresponding to the entity type is called to query the corresponding entity, so as to score the confidence of the entity labeling results; If the entity annotation results contain a target entity with a confidence score greater than or equal to a preset score, the target entity is determined to have been successfully verified, and the target entity is added to the preset entity library.
[0005] In some embodiments, after invoking the target verification interface corresponding to the entity type based on the entity type of each entity to perform confidence scoring on the entity annotation results, the method further includes: If there are entities in the entity annotation results that have a confidence score lower than the preset score and need to be reviewed, the entities to be reviewed will be added to the manual annotation task pool for manual review. The entities that have been manually reviewed are added to the preset entity library.
[0006] In some embodiments, the step of calling the target verification interface corresponding to the entity type to query the corresponding entity based on the entity type of each entity, in order to perform confidence scoring on the entity annotation results, includes: For each entity, if the entity type is geographic information, the entity name and the target location information in the context of the entity are concatenated to generate a target query statement. Based on the target query statement, the first verification interface provided by the preset geographic information database is invoked to query the target query statement through the preset geographic information database, and the confidence score of the entity is determined based on the query results.
[0007] In some embodiments, the step of calling the target verification interface corresponding to the entity type to query the corresponding entity based on the entity type of each entity, in order to perform confidence scoring on the entity annotation results, includes: For each entity, if the entity type is music, the second verification interface provided by the preset music platform is called to query the entity name of the entity through the preset music platform, and the confidence score of the entity is determined based on the query result.
[0008] In some embodiments, the step of calling the target verification interface corresponding to the entity type to query the corresponding entity based on the entity type of each entity, in order to perform confidence scoring on the entity annotation results, includes: For each entity, if the entity type is video, the third verification interface provided by the preset video platform is called to query the entity name of the entity through the preset video platform, and the confidence score of the entity is determined based on the query result.
[0009] In some embodiments, the method further includes: during the process of calling the target verification interface to query the corresponding entity, obtaining a screenshot of the query result of the corresponding entity and a link corresponding to the query result; If the entity annotation results contain entities with confidence scores lower than the preset score that require review, these entities will be added to the manual annotation task pool for manual review, including: The screenshots and links of the query results corresponding to the entity to be reviewed are displayed on the annotation interface so that the annotators can manually annotate the entity through the screenshots and links of the query results.
[0010] In some embodiments, the annotation interface provides recommended correction data and manual correction functionality for the entity to be verified, and the method further includes: If an acceptance operation for recommended corrected data for the entity to be verified is detected, or a manual correction operation for the entity to be verified is detected, training data is generated based on the corrected entity. The preset entity annotation model is updated based on the training data.
[0011] In some embodiments, the method further includes: Based on a preset detection period, the timeliness of entities in the preset entity library is detected; If an expired entity is detected, the expired entity will be marked.
[0012] Secondly, embodiments of the present invention provide an entity annotation apparatus, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the entity annotation method described above.
[0013] Thirdly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described entity annotation method.
[0014] Fourthly, embodiments of the present invention provide a computer program product, the computer program product including a computer program, which, when executed by a processor, is used to load and execute the steps in the above-described entity annotation method.
[0015] The above-described one or more technical solutions in the embodiments of this application have at least the following technical effects: The entity annotation method provided in this specification involves obtaining initial entity data to be annotated, inputting the initial entity data into a preset entity annotation model, and annotating each entity in the initial entity data using the preset entity annotation model to obtain entity annotation results. The entity annotation results include the entity type of each entity. Based on the entity type of each entity, a target verification interface corresponding to the entity type is called to query the corresponding entity to perform a confidence score on the entity annotation results. If a target entity with a confidence score greater than or equal to the preset score exists in the entity annotation results, the target entity is determined to have been successfully verified, and the target entity is added to a preset entity library. In this solution, when annotating the initial entity data, the preset entity annotation model is used to annotate the initial entity data first, and then the verification interface is called to verify the correctness of the entity annotation results. Target entities with a confidence score greater than or equal to the preset score are directly added to the preset entity library, effectively reducing manual annotation, thereby improving the efficiency and reducing the cost of entity annotation. Attached Figure Description
[0016] Figure 1 A flowchart illustrating an entity annotation method provided in the embodiments of this specification; Figure 2 This is a schematic diagram of an entity annotation device provided in an embodiment of this specification; Figure 3This is a schematic diagram of another entity annotation device provided in the embodiments of this specification. Detailed Implementation
[0017] The overall technical solution of this application embodiment is as follows: Initial entity data to be labeled is obtained; the initial entity data is input into a preset entity labeling model, and each entity in the initial entity data is labeled using the preset entity labeling model to obtain entity labeling results, the entity labeling results including the entity type of each entity; based on the entity type of each entity, a target verification interface corresponding to the entity type is called to query the corresponding entity to perform a confidence score on the entity labeling results; if there is a target entity in the entity labeling results with a confidence score greater than or equal to a preset score, the target entity is determined to have been successfully verified, and the target entity is added to a preset entity library.
[0018] In the embodiments of this specification, when annotating the initial entity data, the initial entity data is first annotated by a preset entity annotation model, and then the correctness of the entity annotation results is verified by calling the verification interface. Target entities with confidence scores greater than or equal to the preset scores are directly added to the preset entity library, which effectively reduces manual annotation, thereby improving the efficiency of entity annotation and reducing the cost of entity annotation.
[0019] To better understand the above technical solutions, the technical solutions of the embodiments of this specification will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of this specification and the specific features in the embodiments are detailed descriptions of the technical solutions of the embodiments of this specification, rather than limitations on the technical solutions of this specification. Unless otherwise specified, the embodiments of this specification and the technical features in the embodiments can be combined with each other.
[0020] First, it should be clarified that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.
[0021] like Figure 1 The diagram shown is a flowchart of an entity annotation method provided in an embodiment of this specification. The method includes the following steps: Step S101: Obtain the initial entity data to be labeled; Step S102: Input the initial entity data into a preset entity annotation model, and use the preset entity annotation model to annotate each entity in the initial entity data to obtain entity annotation results. The entity annotation results include the entity type of each entity. Step S103: Based on the entity type of each entity, call the target verification interface corresponding to the entity type to query the corresponding entity, so as to score the confidence of the entity labeling results; Step S104: If there is a target entity in the entity annotation result with a confidence score greater than or equal to the preset score, determine that the target entity has been successfully verified, and add the target entity to the preset entity library.
[0022] The methods provided in the embodiments of this specification can be applied to terminal devices, such as smartphones, voice interaction terminal devices, and in-vehicle systems, as well as to servers that communicate with terminal devices, and to systems consisting of terminal devices and servers, without limitation.
[0023] In step S101, it is necessary to first obtain the initial entity data to be labeled. The initial entity data can be obtained in various ways. For example, the initial entity data can be the collected user voice data, or it can be obtained through a general corpus. In some embodiments, taking the vehicle-machine interaction scenario as an example, the initial entity data can be extracted from the logs of user interaction with the vehicle-machine system. For example, the user's interaction data can be extracted from the logs, such as the interaction data input by the user through voice interaction, the text data input by the user manually, etc.
[0024] In step S102, the preset entity annotation model can be a pre-trained model. The type of the preset entity annotation model can be selected according to actual needs. In some embodiments, the preset entity annotation model can be a Large Language Model (LLM). The preset entity annotation model can be used to perform initial entity annotation on the initial entity data, and to label each entity contained in the initial entity data, labeling the entity type of each entity.
[0025] In step S103, a verification interface is introduced to automatically verify the accuracy of the entity annotation results. This verification interface can be provided by an authoritative external data source. It should be noted that the type of entity verified by the verification interface varies depending on the type of external data source. For example, external data sources may include geographic information databases, music copyright databases, and video metadata indexes. In this case, the geographic information database could correspond to an API (Application Programming Interface) for verifying geographic information type entities, the music copyright database could correspond to an API for verifying music type entities, and the video metadata index could correspond to an API for verifying video type entities.
[0026] Therefore, after annotating each entity using a preset entity annotation model, the accuracy of the entity annotation results can be verified by calling a target verification interface matching the entity type. In some embodiments, when verifying the entity annotation results, verification can be performed based on the entity name and entity type of each entity. During the verification process, while verifying whether the entity type is accurately annotated, it can also verify whether the entity name is incorrectly identified.
[0027] For ease of explanation, the following section uses geographic information, music, and video as examples to illustrate the process of scoring confidence for entities.
[0028] I. Entity type is geographic information type When assigning confidence scores to entities, the following steps can be taken: For each entity, if the entity type is geographic information, the entity name and the target location information in the context of the entity are concatenated to generate a target query statement; based on the target query statement, the first verification interface provided by the preset geographic information database is called to query the target query statement through the preset geographic information database, and the confidence score of the entity is determined based on the query results.
[0029] Specifically, if an entity's entity type is geographic information, for example, an entity named "Lantu Co., Ltd." with a geographic information tag, a preset geographic information database can be invoked. The first verification interface corresponding to this database can be used as the target verification interface to verify the entity's annotation results. It should be noted that the preset geographic information database can be an external authoritative data source as mentioned above. For example, it can be the database corresponding to an existing electronic map (such as Gaode Maps or Baidu Maps), in which case the electronic map API provided by the electronic map can be used as the first verification interface. Alternatively, the preset geographic information database can be a self-built geographic database storing geographic information data for a specific region or a larger area. There are no limitations here.
[0030] When verifying the entity labeling results by calling the first verification interface, in some embodiments, the entity name can be directly searched in the preset geographic information database through the first verification interface. Taking the entity name "Lantu Co., Ltd." as an example, and the electronic map as an example, "Lantu Co., Ltd." can be queried in the electronic map through the first verification interface to obtain the query results, and the confidence score can be determined based on the query results.
[0031] In some embodiments, considering that locations with the same pronunciation may exist in different regions, such as Lantu Co., Ltd. in Wuhan and Lantu Co., Ltd. in Beijing, if a user inputs the command "Go to Lantu Co., Ltd." via voice input during navigation, it may be recognized as "Lantu Co., Ltd." or "Lantu Co., Ltd." due to the existence of two locations with the same pronunciation. In this case, to ensure the accuracy of recognition, the target location information in the context of the entity can be obtained. For example, through the historical interaction data between the vehicle system and the user, the user's target location information can be determined to be Wuhan. Then, Wuhan and the entity name are concatenated. It should be noted that if the entity name is recognized as "Lantu Co., Ltd." during the voice recognition process, the concatenated target query statement is "Wuhan Lantu Co., Ltd.", and if the entity name is recognized as "Lantu Co., Ltd." during the voice recognition process, the concatenated target query statement is "Wuhan Lantu Co., Ltd." The concatenated target query statement is queried through the first verification interface to obtain the query results. For example, for "Wuhan Lantu Co., Ltd.", specific coordinates can be returned, but for "Wuhan Lantu Co., Ltd.", there may be no query results, that is, specific coordinates cannot be returned.
[0032] Furthermore, the confidence score of the entity is determined based on the query results. The confidence score can be implemented in various ways. In some embodiments, after returning specific coordinates, the name of the specific POI (Point of Interest) corresponding to those coordinates in a preset geographic information database can be obtained. This name is then matched with the entity name, and the degree of overlap between the two is used as the confidence score. Of course, the confidence score can also be determined in other ways, which are not limited here.
[0033] II. Entity type is music. When assigning confidence scores to entities, the following steps can be taken: For each entity, if the entity type is music, call the second verification interface provided by the preset music platform to query the entity name of the entity through the preset music platform, and determine the confidence score of the entity based on the query results.
[0034] Specifically, music category tags can include, but are not limited to, artist, song title, album name, and music genre. If an entity's type is any of the music categories mentioned above, it can call the second verification interface provided by the preset music platform as the target verification interface. It should be noted that the preset music platform can be the platform corresponding to the external authoritative data source mentioned above. For example, the preset music platform can be a platform that provides music playback services (such as NetEase Cloud Music, QQ Music, etc.), and the second verification interface can be the track.search interface of the preset music platform. The preset music platform can also be a self-built music data platform that stores massive amounts of artist, song, and album information; this is not limited here.
[0035] The entity labeling results are verified by calling the second verification interface. The corresponding entity name can be queried on a preset music platform through the second verification interface. For example, in a car infotainment system scenario, a user inputs via voice: "Play the song AA by singer Zhou XX". Using the preset entity labeling model, "Zhou XX" can be labeled as the singer, and "AA" as the song. The second verification interface can then be called to query the entity name. In some embodiments, each entity name can be queried directly, and then compared based on the query results and context. For example, querying "Zhou XX" and comparing the results to see if the song "AA" exists, or querying "AA" and comparing the results to see if the song is the singer. In other embodiments, semantic recognition can be performed. Based on the semantic recognition results, the entity names are concatenated to generate a query statement, and the second verification interface is called to query the concatenated query statement. Continuing with the example above, if semantic recognition indicates that the user wants to play "AA" sung by Zhou XX, the entity names "Zhou XX" and "AA" are concatenated, and the second verification interface is called to query "Zhou XX AA" on a preset music platform to obtain the query result. Further, a confidence score is calculated based on the query results.
[0036] In some embodiments, the default music platform can return the hash value of the queried song. Based on the song's hash value, a confidence score is determined by comparing the consistency of information such as the artist and album. For example, when the song's hash value is returned and the artist's information is consistent, the confidence score is high; if the artist's information is inconsistent, the confidence score is low.
[0037] III. Entity type is video type When assigning confidence scores to entities, the following steps can be taken: For each entity, if the entity type is video, call the third verification interface provided by the preset video platform to query the entity name through the preset video platform, and determine the confidence score of the entity based on the query results.
[0038] Specifically, if an entity's entity type is video, for example, an entity named "Kung Fu Panda" with a video tag, then the third-party verification interface of a preset video platform can be called as the target verification interface. It should be noted that the preset video platform can be the platform corresponding to the external authoritative data source mentioned above. For example, the preset video platform can be a platform providing video playback services (such as Tencent Video, Youku Video, etc.), and the third-party verification interface can be the interface of these video playback platforms. The preset video platform can also be a self-built video data platform storing massive amounts of data such as movie titles, lead actors, and directors; this is not limited here.
[0039] When verifying entity annotation results using a third verification interface, the corresponding entity name can be queried on a pre-defined video platform. For example, in a vehicle-to-everything (V2X) interaction scenario, a user inputs via voice: "Play the movie Kung Fu Panda." Using a pre-defined entity annotation model, "Kung Fu Panda" can be tagged with a video tag. The third verification interface can then be called to query the entity name "Kung Fu Panda," obtaining the query results, and a confidence score can be determined based on the query structure. The confidence score can be determined in various ways. In some embodiments, the query results can be matched with the queried entity name, and the degree of overlap between the two can be used as the confidence score. Of course, the confidence score can also be determined in other ways, which are not limited here.
[0040] In step S104, for an entity, if the confidence score corresponding to the entity's annotation result is greater than or equal to the preset score, then the entity is considered a successfully verified target entity and added to the preset entity library. The preset score can be set according to actual needs; for example, the preset score can be 95%, 90%, etc.
[0041] In the embodiments of this specification, if the confidence score corresponding to the annotation result of an entity is greater than or equal to the preset score, the annotation result of the entity can be determined to be correct, without the need for manual review, thereby improving the efficiency of entity annotation.
[0042] On the other hand, if there are entities in the entity annotation results that have a confidence score lower than the preset score and need to be reviewed, the entities to be reviewed are added to the manual annotation task pool for manual review; and the manually reviewed entities are added to the preset entity library.
[0043] Specifically, if the confidence score of the entity annotation result verified by calling the verification interface is low, it indicates that there may be some problems with the entity annotation. In this case, manual review is required, and these entities can be added to the manual annotation task pool as entities to be reviewed. After manual review, the reviewed entities are added to the preset entity library.
[0044] The method in the embodiments of this specification can first perform automatic verification of entity annotation results, and only switch to manual review if the verification fails, thereby greatly reducing the amount of manual annotation and effectively improving entity annotation efficiency.
[0045] In this embodiment of the specification, in order to facilitate manual review and further improve the efficiency of manual review, during the process of calling the target verification interface to query the corresponding entity, a screenshot of the query result of the corresponding entity and the link corresponding to the query result are obtained. Then, during manual review, the screenshot of the query result of the entity to be reviewed and the link can be displayed on the annotation interface so that the annotator can manually annotate the entity to be reviewed through the screenshot of the query result of the entity to be reviewed and the link.
[0046] Specifically, during the verification process of entity annotation results, screenshots and links corresponding to the entity query results can be retained and recorded as evidence. In the embodiments of this specification, an annotation interface is provided to facilitate manual review. This interface displays relevant data of the entity to be reviewed. In some embodiments, the annotation interface can display the entity name, type, query result screenshots and links, and confidence score of the entity to be reviewed. During manual review, annotators can intuitively obtain entity-related data through the annotation interface and can also click on query result screenshots and links to view them, improving retrieval efficiency.
[0047] In some embodiments, when the entity to be reviewed is in a field that the annotator is unfamiliar with, the annotator can also click the skip option on the annotation interface, and the system can assign it to an annotator who is proficient in that field based on the preset historical annotation records of the annotators.
[0048] In some embodiments, the annotation interface provides recommended correction data for the entity to be verified and a manual modification function. In the entity annotation process, the following steps may also be included: if an acceptance operation of recommended correction data for the entity to be verified is detected, or a manual modification operation of the entity to be verified is detected, training data is generated based on the modified entity; and the preset entity annotation model is updated based on the training data.
[0049] Specifically, for each entity to be rechecked, recommended correction data can be displayed. The recommended correction data can include the recommended entity type of the entity to be rechecked and the recommended entity name of the entity to be rechecked. It should be noted that there may be the following problems with the entity names of some entities: there are typos in the entity names, the entity names are in dialect expression forms, the entity names are aliases or abbreviations, etc. Below, in response to the above problems, examples of the recommended correction data are given: For the case where there is a typo in the entity name, such as the entity name being "Tencent", the recommended correction data can be "Tencent"; for the dialect expression form, such as the entity name being "yam", the recommended correction data can be "potato"; for the case where the entity name is an alias, such as the entity name being "soda", the recommended correction data can be "cola". And for the recommended entity type, it can be to recommend the corresponding entity type.
[0050] When the annotator determines that the recommended correction data is correct, they can directly select to accept the recommendation and modify the corresponding entity name or entity type to the recommended correction data. When the annotator determines that the recommended correction data is incorrect or there is no recommended correction data, they can directly make manual modifications, or use the screenshot or link of the query result as a reference for manual modification to modify the entity name and / or entity type. In the embodiments of this specification, training data can be generated based on the data before correction and the data after correction to update the preset entity annotation model, so as to enhance the learning ability of the model in the above situations such as dialects and aliases.
[0051] In the embodiments of this specification, in order to ensure the timeliness of entities, the following steps can also be executed: Based on a preset detection period, detect the timeliness of the entities in the preset entity library; if a failed entity whose timeliness has expired is detected, mark the failed entity.
[0052] Specifically, the preset detection period can be set according to actual needs. For example, the preset detection period can be one week, one month, three months, etc., and there is no limitation here. It should be noted that for some entities, there may be a situation of failure. For example, for entities of the geographic information type, if the entity is specifically a restaurant, when the restaurant closes, the status of the entity is actually in a failed state, and the failed entity needs to be marked in a timely manner.
[0053] In some embodiments, the timeliness of entities in a preset entity database can be detected. This can be achieved by calling the target verification interface corresponding to each entity. That is, every preset detection period, each entity in the preset question bank is queried again through the corresponding target verification interface to determine the entity status. For example, for a geographic information entity, if calling the verification interface reveals that the location no longer exists, the entity can be identified as an invalid entity and marked. In some embodiments, when an entity is detected to be invalid, a change summary can be generated. Taking a restaurant entity as an example, a summary such as "The restaurant closed in XX year" can be generated from the large model.
[0054] In some embodiments, if it is impossible to determine whether an entity has become invalid, the entity can be transferred to the pool of human annotators to trigger a re-annotation task, which is then submitted to a human for review and verification.
[0055] In the embodiments of this specification, when performing timeliness detection on entities, entity filtering can be performed first. For example, entities in a preset entity library can be classified into a set of entities with timeliness attributes and a set of entities without timeliness attributes. For instance, entities such as business premises and song copyright status have timeliness attributes, while entities such as people and natural scenery do not have the concept of expiration and can be classified as entities without timeliness attributes. Furthermore, timeliness detection can be performed periodically on the set of entities with timeliness attributes, thereby saving computing resources.
[0056] In the embodiments of this specification, to better serve manual annotation and provide richer recommendation data, a dynamic knowledge graph can also be constructed. Taking the interaction scenario between the vehicle system and the user as an example, the relationships between entities mentioned by the user can be extracted from the logs. For example, information such as song names, performers, and versions mentioned by the user can be extracted, and the relationships between these entities can be preserved to build a knowledge graph of media entities. Entities of geographic information types mentioned in the logs can be associated with the user's location to build a graph relationship between cities and POIs. In addition, the relationship between common aliases and variant data can be constructed. For example, synthetic data containing common errors and alias abbreviations (such as "Starbucks Coffee" - "Starbucks", "Hema Fresh" - "Hema Fresh") can be constructed using a large model to train a discriminative model to identify variants and associate them with standard entities, assisting in parsing the entities mentioned by the user, providing the correct response, and improving the user experience. In this way, when the entities to be reviewed are identified, recommended data for annotation personnel to choose from can be determined based on the relationships between entities in the dynamic knowledge graph and the relationships between aliases and variant data.
[0057] In summary, the method provided in this specification, by introducing a preset entity annotation model and verification interface, directly adds high-confidence entities to a preset question bank, while low-confidence entities undergo manual review, reducing the workload of manual review. This solution integrates various authoritative external data sources as the retrieval backend and displays the query results for entities to be verified in the annotation interface, effectively reducing the time spent by annotators switching retrieval tools and improving manual annotation efficiency. Furthermore, considering that entities such as business premises and music may become invalid or change over time (e.g., store closures, song removals), traditional static databases cannot be synchronized in real time, requiring repeated manual verification of historical annotation results. This solution, however, can periodically call the verification interface to check the status of existing entities, significantly reducing the cost of manual review and effectively improving the timeliness of the database.
[0058] Based on the same inventive concept, embodiments of this specification also provide an entity annotation device, such as... Figure 2 The device includes: The data acquisition module 201 is used to acquire the initial entity data to be labeled; The annotation module 202 is used to input the initial entity data into a preset entity annotation model, and to annotate each entity in the initial entity data through the preset entity annotation model to obtain entity annotation results. The entity annotation results include the entity type of each entity. The verification module 203 is used to call the target verification interface corresponding to the entity type to query the corresponding entity based on the entity type of each entity, so as to perform confidence scoring on the entity annotation results; The processing module 204 is used to determine that the target entity has been successfully verified if there is a target entity in the entity annotation result with a confidence score greater than or equal to a preset score, and to add the target entity to the preset entity library.
[0059] Regarding the above-described apparatus, the specific implementation methods of each step have been described in detail in the embodiments of the entity annotation method provided in the specification, and will not be elaborated here.
[0060] Based on the same inventive concept, embodiments of the present invention also provide an entity annotation device, such as... Figure 3 The above includes a memory 304, a processor 302, and a computer program stored in the memory 304 and executable on the processor 302. When the processor 302 executes the program, it implements any of the above-described entity annotation methods.
[0061] Among them, Figure 3In this document, a bus architecture (represented by bus 300) is used. Bus 300 may include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 302 and memory represented by memory 304. Bus 300 may also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 305 provides an interface between bus 300 and receiver 301 and transmitter 303. Receiver 301 and transmitter 303 may be the same element, i.e., a transceiver, providing a unit for communicating with various other devices over a transmission medium. Processor 302 is responsible for managing bus 300 and general processing, while memory 304 can be used to store data used by processor 302 during operation.
[0062] Based on the same inventive concept, embodiments of this specification provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described entity annotation method.
[0063] Based on the same inventive concept, embodiments of this specification provide a computer program product, which includes a computer program. When the computer program is executed by a processor, it is used to load and execute the above-described entity annotation method steps.
[0064] The functions described herein can be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions can be stored as one or more instructions or codes on or transmitted via a computer-readable medium. Other examples and embodiments are within the scope and spirit of this invention and the appended claims. For example, due to the nature of software, the functions described above can be implemented using software executed by a processor, hardware, firmware, hardwired, or any combination thereof. Furthermore, the functional units can be integrated into a single processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit.
[0065] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.
[0066] The units described as separate components may or may not be physically separate. Similarly, the components of the control device may or may not be physical units; they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.
[0067] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.
[0068] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of the claims of the present invention.
Claims
1. An entity annotation method, characterized in that, include: Obtain the initial entity data to be labeled; The initial entity data is input into a preset entity annotation model, and each entity in the initial entity data is annotated by the preset entity annotation model to obtain entity annotation results. The entity annotation results include the entity type of each entity. Based on the entity type of each entity, the target verification interface corresponding to the entity type is called to query the corresponding entity, so as to score the confidence of the entity labeling results; If the entity annotation results contain a target entity with a confidence score greater than or equal to a preset score, the target entity is determined to have been successfully verified, and the target entity is added to the preset entity library.
2. The method as described in claim 1, characterized in that, After invoking the target verification interface corresponding to the entity type based on the entity type of each entity to perform confidence scoring on the entity annotation results, the method further includes: If there are entities in the entity annotation results that have a confidence score lower than the preset score and need to be reviewed, the entities to be reviewed will be added to the manual annotation task pool for manual review. The entities that have been manually reviewed are added to the preset entity library.
3. The method as described in claim 1, characterized in that, The step of calling the target verification interface corresponding to the entity type to query the corresponding entity based on the entity type of each entity, and then performing a confidence score on the entity annotation results, includes: For each entity, if the entity type is geographic information, the entity name and the target location information in the context of the entity are concatenated to generate a target query statement. Based on the target query statement, the first verification interface provided by the preset geographic information database is invoked to query the target query statement through the preset geographic information database, and the confidence score of the entity is determined based on the query results.
4. The method as described in claim 1, characterized in that, The step of calling the target verification interface corresponding to the entity type to query the corresponding entity based on the entity type of each entity, and then performing a confidence score on the entity annotation results, includes: For each entity, if the entity type is music, the second verification interface provided by the preset music platform is called to query the entity name of the entity through the preset music platform, and the confidence score of the entity is determined based on the query result.
5. The method as described in claim 1, characterized in that, The step of calling the target verification interface corresponding to the entity type to query the corresponding entity based on the entity type of each entity, and then performing a confidence score on the entity annotation results, includes: For each entity, if the entity type is video, the third verification interface provided by the preset video platform is called to query the entity name of the entity through the preset video platform, and the confidence score of the entity is determined based on the query result.
6. The method as described in claim 2, characterized in that, The method further includes: during the process of calling the target verification interface to query the corresponding entity, obtaining a screenshot of the query result of the corresponding entity and the link corresponding to the query result; If the entity annotation results contain entities with confidence scores lower than the preset score that require review, these entities will be added to the manual annotation task pool for manual review, including: The screenshots and links of the query results corresponding to the entity to be reviewed are displayed on the annotation interface so that the annotators can manually annotate the entity through the screenshots and links of the query results.
7. The method as described in claim 6, characterized in that, The annotation interface provides recommended correction data and manual correction functions for the entity to be verified, and the method further includes: If an acceptance operation for recommended corrected data for the entity to be verified is detected, or a manual correction operation for the entity to be verified is detected, training data is generated based on the corrected entity. The preset entity annotation model is updated based on the training data.
8. The method as described in claim 1, characterized in that, The method further includes: Based on a preset detection period, the timeliness of entities in the preset entity library is detected; If an expired entity is detected, the expired entity will be marked.
9. A solid annotation device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method as described in any one of claims 1-8.
10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-8.