Text data processing method and device

Abstract

The embodiment of the invention discloses a text data processing method and device. According to the embodiment of the invention, after to-be-processed text data is obtained, candidate entity sets corresponding to entity names in the text data are determined, and target entities linked with the entity names are determined in the candidate entity sets; and marking the position and the represented type of each target entity in the text data to determine an additional vector, and inputting the additional vector and the embedded representation of the text data into a target model to determine intention information and slot information in the text data through the target model. Therefore, the target entities linked with the entity names in the text data are determined, and the position information of the target entities in the text data and the represented type information are input into the model together with the embedded representation of the text data in the form of additional vectors; according to the embodiment of the invention, the influence of ambiguous entities in the text data on a model recognition result can be reduced, so that the accuracy and recall rate of the model are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and particularly to a method and apparatus for processing text data. Background Art

[0002] Natural Language Understanding (NLU) is an important branch in the field of artificial intelligence, aiming to enable a computer to understand and apply the natural language of human society. Natural language understanding generally includes two important tasks: intent classification and slot filling. Among them, intent classification refers to determining the user intent in text data, and slot filling refers to determining the specific information required to execute the user intent in the text data.

[0003] Currently, the prior art usually uses corresponding models to determine the intent information and slot information in text data. However, since the model cannot disambiguate the ambiguous entities in the text data, when the text data contains ambiguous entities, the existence of the ambiguous entities will have a great impact on the model recognition result, resulting in a decrease in the accuracy and recall rate of the model. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method and apparatus for processing text data to reduce the impact of ambiguous entities in the text data on the model recognition result, thereby improving the accuracy and recall rate of the model.

[0005] In a first aspect, an embodiment of the present invention provides a method for processing text data, the method comprising:

[0006] Obtaining the text data to be processed;

[0007] Determining a candidate entity set corresponding to each entity name in the text data;

[0008] For each entity name, determining a target entity linked by the entity name in the corresponding candidate entity set;

[0009] Annotating the position and the represented type of each target entity in the text data to determine an additional vector;

[0010] Inputting the additional vector and the embedding representation of the text data into a target model to determine the intent information and slot information in the text data through the target model.

[0011] In a second aspect, an embodiment of the present invention provides a device for processing text data, the device comprising:

[0012] An obtaining unit, configured to obtain the text data to be processed;

[0013] A candidate entity determination unit, configured to determine a candidate entity set corresponding to each entity name in the text data;

[0014] A target entity determination unit, configured to, for each of the entity names, determine a target entity linked by the entity name in the corresponding candidate entity set;

[0015] A vector determination unit, configured to label the position and represented type of each target entity in the text data to determine an additional vector;

[0016] An information determination unit, configured to input the additional vector and the embedding representation of the text data into a target model to determine intent information and slot information in the text data through the target model.

[0017] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, on which computer program instructions are stored, and the computer program instructions, when executed by a processor, implement the method described in any item of the first aspect.

[0018] In a fourth aspect, an embodiment of the present invention provides an electronic device, and the device includes:

[0019] A memory, configured to store one or more computer program instructions;

[0020] A processor, and the one or more computer program instructions are executed by the processor to implement the method described in any item of the first aspect.

[0021] In a fifth aspect, an embodiment of the present invention provides a computer program product, and when the computer program product runs on a computer, the computer is caused to execute the method described in any item of the first aspect.

[0022] After obtaining the text data to be processed, the embodiment of the present invention determines a candidate entity set corresponding to each entity name in the text data, determines a target entity linked by each entity name in each candidate entity set, then labels the position and represented type of each target entity in the text data to determine an additional vector, and further inputs the additional vector and the embedding representation of the text data into a target model to determine intent information and slot information in the text data through the target model. Thus, by determining the target entity linked by each entity name in the text data and inputting the position information and represented type information of each target entity in the text data into the model in the form of an additional vector together with the embedding representation of the text data, the embodiment of the present invention can reduce the influence of ambiguous entities in the text data on the model recognition result, thereby improving the accuracy and recall rate of the model. Description of the Drawings

[0023] Through the following description of the embodiments of the present invention with reference to the accompanying drawings, the above and other objects, features, and advantages of the present invention will become clearer. In the drawings:

[0024] Figure 1 is a flowchart of the text data processing method according to an embodiment of the present invention;

[0025] Figure 2 is a flowchart of the target entity determination method according to an embodiment of the present invention;

[0026] Figure 3 is a schematic diagram of the text data processing process according to an embodiment of the present invention;

[0027] Figure 4 is a schematic diagram of the text data processing process according to an embodiment of the present invention;

[0028] Figure 5 is a schematic diagram of the text data processing device according to an embodiment of the present invention;

[0029] Figure 6 is a schematic diagram of the electronic device according to an embodiment of the present invention. Detailed Embodiments

[0030] The following is a description of the present application based on embodiments, but the present application is not limited to these embodiments. In the following detailed description of the present application, some specific details are described in detail. Those skilled in the art can fully understand the present application without the description of these details. To avoid obscuring the essence of the present application, well-known methods, processes, procedures, elements, and circuits are not described in detail.

[0031] In addition, those of ordinary skill in the art should understand that the accompanying drawings provided here are for illustrative purposes only, and the drawings are not necessarily drawn to scale.

[0032] Unless the context clearly requires otherwise, words such as "including" and "comprising" in the entire application document should be interpreted as having an inclusive meaning rather than an exclusive or exhaustive meaning; that is, the meaning of "including but not limited to".

[0033] In the description of the present application, it should be understood that terms such as "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present application, unless otherwise specified, the meaning of "a plurality" is two or more.

[0034] In the solutions described in this specification and the embodiments, if personal information processing is involved, it will be processed on the premise of having a legal basis (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract, etc.), and will only be processed within the specified or agreed scope. If a user refuses to process personal information other than the necessary information required for the basic functions, it will not affect the user's use of the basic functions.

[0035] It should be noted that the execution subject of the text data processing method in the embodiments of the present invention can be a terminal for providing services to users according to user instructions. By executing the text data processing method, the terminal can determine the intent information and slot information in the instruction text, and provide services matching the issued instructions to the user according to the intent information and slot information. Optionally, the terminal can be any type of terminal device such as a smart phone, a tablet computer, a smart wearable device, a laptop computer, a desktop computer, or a smart speaker. In some embodiments, the terminal can also be a machine device with data processing functions, such as a car, a robot, or a drone. It should be understood that the present application does not specifically limit the type of services provided by the terminal to the user, and it can correspond to the functions possessed by the terminal itself.

[0036] Alternatively, the execution subject of the text data processing method in the embodiments of the present invention can also be a server for providing information determination services externally. By executing the text data processing method, the server can determine the intent information and slot information in the text data provided by the service requester, and feedback the intent information and slot information to the service requester. Optionally, the server can be a single computer, a cluster composed of multiple computers, or a cloud server that elastically adjusts computing resources through cloud technology.

[0037] Figure 1 is a flowchart of the text data processing method in the embodiments of the present invention. It should be understood that Figure 1 the execution subject of the text data processing method shown can be the corresponding data processing device. The data processing device can be either a terminal or a server. As Figure 1 shown, the text data processing method can specifically include the following steps:

[0038] Step S100: Obtain the text data to be processed.

[0039] Specifically, the data processing device can obtain the text data to be processed.

[0040] Further, when the data processing device is a terminal, the text data to be processed can be an instruction text issued by the user.

[0041] Optionally, this embodiment may support the user to issue the text data in a text input manner or a voice input manner. Among them, when the user issues the text data in a text input manner, the data processing device may obtain the text data to be processed by receiving the input text. When the user issues the text data in a voice input manner, the data processing device may obtain the text data to be processed by receiving the input voice data and performing speech recognition on the voice data.

[0042] Further, when the data processing device is a server, the text data to be processed may be the text data provided by the service requester.

[0043] Optionally, the text data to be processed may be carried in the information determination request sent by the service requester. The data processing device may parse the information determination request when receiving it to obtain the text data to be processed.

[0044] Step S200: Determine the candidate entity set corresponding to each entity name in the text data.

[0045] Specifically, after obtaining the text data to be processed, the data processing device may determine the entity names included in the text data and determine the candidate entity set corresponding to each entity name.

[0046] In an optional implementation manner, this embodiment may adopt a dictionary matching method to determine the entity names included in the text data.

[0047] Specifically, in this embodiment, relevant personnel may pre-create an entity dictionary containing all currently known entities. Among them, each known entity in the entity dictionary may have one or more associated extraction rules, and the extraction rules may be used to indicate that relevant entities are extracted when the text or text structure in the text data meets certain conditions. In step S200, the data processing device may match the text data with the entity dictionary based on the rule matching method to determine the entity names included in the text data.

[0048] In another optional implementation manner, this embodiment may also adopt a coarse-grained word segmentation method to determine the entity names included in the text data.

[0049] Specifically, in step S200, the data processing device can perform coarse-grained word segmentation on the text data, and match the word segmentation results with each entity in the entity dictionary to determine the entity names included in the text data. Among them, performing coarse-grained word segmentation on the text data specifically refers to splitting the text data into the most basic words. Optionally, the data processing device can use an open-source word segmentation tool or a corresponding word segmentation model to perform coarse-grained word segmentation on the text data.

[0050] In another optional implementation manner, this embodiment can also use an entity extraction model to determine the entity names included in the text data.

[0051] Specifically, in step S200, the data processing device can input the text data into the entity extraction model to determine the entity names included in the text data through the entity extraction model. Optionally, the entity extraction model can be a BERT-BiLSTM-CRF model or a Span-based pointer network model, etc., and this application does not limit this. Among them, the BERT-BiLSTM-CRF model is a deep learning model that combines BERT (Bidirectional Encoder Representations from Transformers, bidirectional encoder representations from based on Transformer), bidirectional long short-term memory network (BiLSTM) and conditional random field (CRF). This model is mainly used for sequence labeling tasks. The Span-based pointer network model is a deep learning model used to process sequence labeling and extraction tasks. This model is mainly applied to tasks such as named entity recognition (NER) and relation extraction.

[0052] It should be understood that during the process of the data processing device determining the entity names, it can only determine the entity names without predicting the category information of each entity name. At the same time, the above-mentioned determination methods of entity names are only for illustration. In the actual application process, the determination method of the entity names included in the text data can be specifically set and adjusted by relevant personnel according to actual needs, and this application does not limit this.

[0053] Furthermore, the candidate entity set can include multiple candidate entities that the entity name can point to.

[0054] In an optional implementation manner, to ensure the accuracy of recalling candidate entities, this embodiment can perform entity recall according to each entity name to determine the candidate entity set corresponding to each entity name.

[0055] Specifically, in step S200, for each entity name, the data processing device can recall multiple candidate entities with the entity name in the entity dictionary, so as to determine the candidate entity set corresponding to the entity name. Optionally, in order to avoid recalling too many candidate entities and thus affecting the subsequent processing efficiency, when recalling candidate entities, the data processing device can only recall a preset number of candidate entities with the highest popularity.

[0056] It should be understood that in this embodiment, the popularity of the candidate entity can specifically refer to the historical recall popularity of the candidate entity, or the click search popularity of the candidate entity on a third-party website. The present application does not limit this.

[0057] In another optional implementation manner, in order to ensure the generalization and recall rate of the recall, this embodiment can also determine the vector representation of the text data and perform entity recall according to the vector representation of the text data to determine the candidate entity set corresponding to each entity name.

[0058] Specifically, each known entity in the entity dictionary can also have corresponding description information. Among them, the description information can be used to describe and explain the entity. Optionally, the description information can include relevant information such as the name information, type information, impression label information, profile information, and relationship information with other entities of the entity. It should be understood that the information content included in the description information of different entities can be different, and it can be specifically set and adjusted by relevant personnel according to actual needs. The present application does not limit the information content included in the description information. Further, the description information can be pre-stored by the data processing device in the form of a vector representation. In step S200, the data processing device can determine the vector representation of the text data, and recall multiple candidate entities in the entity dictionary whose vector representation of the description information is similar to the vector representation of the text data, and then determine the candidate entity set corresponding to each entity name. It should be understood that the vector representation of the description information and the vector representation of the text data should have the same vector dimension, such as 256 dimensions. Optionally, in order to avoid recalling too many candidate entities and thus affecting the subsequent processing efficiency, when recalling candidate entities, the data processing device can only recall a preset number of candidate entities with the highest similarity between the vector representation of the description information and the vector representation of the text data. Optionally, the data processing device can use a vector retrieval tool based on something like Faiss (Facebook AI Similarity Search) to recall candidate entities. Among them, Faiss is a library for efficient vector retrieval, which is mainly used to process large-scale high-dimensional vector datasets to perform fast similarity search and clustering operations in large-scale datasets.

[0059] It should be understood that the above - given method for determining the set of candidate entities is only for illustration. In the actual application process, the method for determining the set of candidate entities corresponding to the entity name can be specifically set and adjusted by relevant personnel according to actual needs, and this application does not limit it.

[0060] Step S300: For each of the entity names, determine the target entity linked by the entity name in the corresponding set of candidate entities.

[0061] Specifically, after determining the set of candidate entities corresponding to each entity name in the text data, the data - processing device can determine the target entity linked by each entity name in the set of candidate entities corresponding to each entity name. Among them, the target entity of each entity name is the correct entity that each entity name should link to.

[0062] Optionally, when determining the target entity linked by the entity name, in this embodiment, the target entity linked by the entity name can be determined among the candidate entities at least according to the matching degree between each candidate entity in the set of candidate entities and the text data. Figure 2 This is a flowchart of the method for determining the target entity according to an embodiment of the present invention. As Figure 2 shown, the method for determining the target entity can specifically include the following steps:

[0063] Step S310: For each candidate entity in the set of candidate entities, determine the matching degree between the candidate entity and the text data.

[0064] Specifically, the data - processing device can determine the matching degree between each candidate entity in the set of candidate entities and the text data.

[0065] Optionally, in this embodiment, the matching degree between the candidate entity and the text data can be specifically reflected by the matching degree between the description information of the candidate entity and the text data. In step S310, for each candidate entity in the set of candidate entities, the data - processing device can determine the matching degree between the description information of the candidate entity and the text data, and determine the matching degree between the description information of the candidate entity and the text data as the matching degree between the candidate entity and the text data.

[0066] Step S320: Determine the target entity linked by the entity name among the candidate entities at least according to the matching degree between each candidate entity and the text data.

[0067] Specifically, after determining the matching degree between each candidate entity and the text data, the data - processing device can determine the target entity linked by the entity name among the candidate entities at least according to the matching degree between each candidate entity and the text data.

[0068] Optionally, in step S320, when determining the target entity linked to the entity name among the candidate entities at least according to the matching degree between each candidate entity and the text data, the data processing device may directly determine the candidate entity with the highest matching degree with the text data as the target entity linked to the entity name.

[0069] Optionally, in some scenarios, due to reasons such as less information in the text data or high similarity between candidate entities, if the data processing device determines the target entity linked to the entity name among the candidate entities only according to the matching degree between each candidate entity and the text data, it is very likely that the target entity linked to the entity name cannot be accurately determined.

[0070] For example, assume that the text data is "Play JKL of abc". The entity names included in this text data are "abc" and "JKL". Among them, "JKL" is the entity name for which the data processing device currently needs to determine the target entity. The candidate entities corresponding to this entity name include the album "JKL" and the song "JKL". Since the information contained in the text data is less and the similarity between candidate entities is high, if the data processing device determines the target entity linked to this entity name among the candidate entities only according to the matching degree between each candidate entity and the text data, then it is very likely that the target entity linked to this entity name cannot be accurately determined.

[0071] In this regard, in order to improve the accuracy of determining the target entity, in this embodiment, the target entity linked to the entity name may also be determined among the candidate entities according to the matching degree between each candidate entity and the text data and the disambiguation features. Among them, the disambiguation features may be relevant features that affect the result of determining the target entity. Optionally, in this embodiment, the disambiguation features may at least include one or more combinations of the following: the popularity of the candidate entity, the type of the candidate entity, and the current scenario information. Further, when determining the target entity linked to the entity name among the candidate entities according to the matching degree between each candidate entity and the text data and the disambiguation features, the data processing device may comprehensively evaluate each candidate entity according to the matching degree between each candidate entity and the text data and the disambiguation features (such as how popular each candidate entity is, whether the type of each candidate entity is the user-preferred type, and whether each candidate entity matches the current scenario, etc.) to obtain the evaluation scores of each candidate entity. After obtaining the evaluation scores of each candidate entity, the data processing device may determine the candidate entity with the highest evaluation score as the target entity linked to the entity name. It should be understood that when comprehensively evaluating each candidate entity to obtain the evaluation scores of each candidate entity, the matching degree between the candidate entity and the text data and different disambiguation features may each have corresponding influence weights, and the specific influence weights may be set by relevant personnel, and this application does not limit this.

[0072] It should be understood that the above-mentioned disambiguation features are only for illustration. In the actual application process, the disambiguation features specifically referred to by the data processing device when determining the target entity linked to the entity name may be set by relevant personnel according to actual needs, and this application does not limit this. For example, the disambiguation feature may also be set to include the similarity between the vector representation of the description information of the candidate entity and the vector representation of the text data.

[0073] Optionally, the above steps S310 - S320 may be implemented by the data processing device through a corresponding entity ranking model.

[0074] Specifically, the data processing device may input relevant information into the entity ranking model to evaluate each candidate entity through the entity ranking model and determine the target entity linked to the entity name according to the evaluation result. Optionally, the entity ranking model may specifically be a pre-trained Cross-Encoder model. Among them, the Cross-Encoder model is a deep learning model that can be used to handle ranking problems in natural language processing tasks.

[0075] Step S400: Label the positions and represented types of each target entity in the text data to determine an additional vector.

[0076] Specifically, after determining the target entities linked by each entity name, the data processing device can annotate the positions and represented types of the target entities in the text data to determine the additional vectors.

[0077] Furthermore, since the additional vectors need to be input into the target model together with the embedded representation of the text data and participate in the calculation in subsequent steps, when determining the additional vectors, the data processing device needs to ensure that the additional vectors and the embedded representation of the text data are in the same dimension. Optionally, in step S400, the data processing device can annotate the positions and represented types of the target entities in the text data to generate an additional matrix, and perform dimension conversion on the additional matrix to determine the additional vectors.

[0078] Step S500: Input the additional vectors and the embedded representation of the text data into the target model to determine the intent information and slot information in the text data through the target model.

[0079] Specifically, after determining the additional vectors, the data processing device can input the additional vectors and the embedded representation of the text data into the target model to determine the intent information and slot information in the text data through the target model.

[0080] Optionally, the embedded representation of the text data may include token embeddings, segment embeddings, and position embeddings. Among them, the token embeddings are used to represent each character in the text data in the form of a vector. The segment embeddings are used to distinguish different sentences in the text data. The position embeddings are used to identify the position of each character in the text data.

[0081] Optionally, the target model may be a pre-trained BERT model. Among them, during the training process, the target model can train the two tasks of intent classification and slot filling simultaneously, and add the loss values of the two tasks as the loss value that the model finally needs to optimize, so as to optimize the loss values of the two tasks simultaneously. In step S500, after the data processing device inputs the additional vectors and the embedded representation of the text data into the target model, the target model can identify and output the intent information and slot information in the text data. Optionally, the loss functions used by the target model when training the two tasks of intent classification and slot filling may both be cross-entropy loss functions.

[0082] Further, when the data processing device is a terminal, after determining the intent information and slot information in the text data, the data processing device may provide a service matching the text data to the user according to the intent information and slot information. When the data processing device is a server, after determining the intent information and slot information in the text data, the data processing device may feedback the determined intent information and slot information to the service requester.

[0083] Figure 3 and Figure 4 is a schematic diagram of the text data processing process according to an embodiment of the present invention. As Figure 3 and Figure 4 shown, the data processing device may obtain the text data 31 to be processed, that is, obtain the text data "play ABCDEF of abc".

[0084] Further, after obtaining the text data 31 to be processed, the data processing device may determine the entity names 32 included in the text data 31, that is, determine the entity names "ab", "abc", "ABCDEF", "ABC", and "EF" included in the text data "play ABCDEF of abc". Optionally, the manner in which the data processing device determines the entity names may specifically refer to the foregoing content and will not be elaborated herein.

[0085] Further, after determining the entity names 32 included in the text data 31, the data processing device may determine at least one candidate entity corresponding to each entity name 32, so as to obtain a candidate entity set 33 corresponding to each entity name 32. It should be understood that Figure 3 the candidate entity set 33 corresponding to each entity name 32 shown in [] is only for illustration. Optionally, the manner in which the data processing device determines each candidate entity set may specifically refer to the foregoing content and will not be elaborated herein.

[0086] Further, after determining the candidate entity set 33 corresponding to each entity name 32, the data processing device may determine the target entity 34 linked by each entity name 32 in each candidate entity set 33. Optionally, the manner in which the data processing device determines the target entity may specifically refer to the foregoing content and will not be elaborated herein. It should be understood that if, in the above steps, the candidate entity set obtained by the data processing device only includes one candidate entity, the candidate entity may be directly determined as the target entity linked by the corresponding entity name.

[0087] Further, after determining the target entity 34, the data processing device may label the position and the represented type of each target entity 34 in the text data 31 to generate an additional matrix 35.

[0088] Optionally, in this embodiment, multiple preset types may be preset by relevant personnel (such as the three preset types of singer, song, and movie shown in Figure 3 ). Among them, each preset type may have three corresponding sub-vectors B-xxx, I-xxx, and E-xxx. The B-xxx may indicate whether the corresponding character in the text data is at the start position of the corresponding preset type, the I-xxx may indicate whether the corresponding character in the text data is at the middle position of the corresponding preset type, and the E-xxx may indicate whether the corresponding character in the text data is at the end position of the corresponding preset type. When performing annotation, the data processing device may annotate the positions and represented types of each target entity in the text data in the sub-vectors corresponding to the multiple preset types to generate an additional matrix.

[0089] It should be understood that for a preset type without a target entity, when the data processing device performs annotation, each element in the three sub-vectors corresponding to the preset type may be marked as 0 (as shown in the movie preset type in Figure 3 ).

[0090] It should be understood that before performing annotation, the data processing device may also add two characters CLS and SEP before and after the text data respectively to indicate the start and end of the input.

[0091] It should be understood that Figure 3 the additional matrix shown in

[0092] is only for illustration. In actual application, the scale of the additional matrix is not limited to this. Further, after determining the additional matrix 35, the data processing device may perform dimension conversion on the additional matrix 35 to determine an additional vector 36. After determining the additional vector 36, the data processing device may input the additional vector 36 and the embedding representation 37 of the text data 31 into the target model 38 together to determine the intent information and slot information 39 in the text data 31 through the target model 38. Among them, the embedding representation 37 of the text data 31 may include word embedding 371, segment embedding 372, and position embedding 373.

[0093] After obtaining the text data to be processed in an embodiment of the present invention, a candidate entity set corresponding to each entity name in the text data will be determined, and a target entity linked by each entity name will be determined in each candidate entity set. Then, the position and the represented type of each target entity in the text data will be marked to determine an additional vector. Furthermore, the additional vector and the embedded representation of the text data will be input into a target model to determine the intent information and slot information in the text data through the target model. Thus, by determining the target entity linked by each entity name in the text data and inputting the position information and the represented type information of each target entity in the text data into the model in the form of an additional vector together with the embedded representation of the text data, the embodiment of the present invention can reduce the influence of ambiguous entities in the text data on the model recognition result, thereby improving the accuracy and recall rate of the model.

[0094] Figure 5 It is a schematic diagram of the text data processing device according to an embodiment of the present invention. As Figure 5 shown, the text data processing device according to an embodiment of the present invention includes an acquisition unit 51, a candidate entity determination unit 52, a target entity determination unit 53, a vector determination unit 54, and an information determination unit 55.

[0095] Specifically, the acquisition unit 51 is configured to acquire the text data to be processed;

[0096] The candidate entity determination unit 52 is configured to determine a candidate entity set corresponding to each entity name in the text data;

[0097] The target entity determination unit 53 is configured to determine, for each entity name, the target entity linked by the entity name in the corresponding candidate entity set;

[0098] The vector determination unit 54 is configured to mark the position and the represented type of each target entity in the text data to determine an additional vector;

[0099] The information determination unit 55 is configured to input the additional vector and the embedded representation of the text data into a target model to determine the intent information and slot information in the text data through the target model.

[0100] After obtaining the text data to be processed in an embodiment of the present invention, candidate entity sets corresponding to each entity name in the text data are determined, and the target entity linked by each entity name is determined in each candidate entity set. Then, the position and the represented type of each target entity in the text data are marked to determine an additional vector. Furthermore, the additional vector and the embedding representation of the text data are input into a target model to determine the intent information and slot information in the text data through the target model. Thus, by determining the target entity linked by each entity name in the text data and inputting the position information and the represented type information of each target entity in the text data in the form of an additional vector together with the embedding representation of the text data into the model, the embodiment of the present invention can reduce the influence of ambiguous entities in the text data on the model recognition result, thereby improving the accuracy and recall rate of the model.

[0101] Figure 6 is a schematic diagram of the electronic device according to an embodiment of the present invention. As Figure 6 shown, Figure 6 the electronic device shown is a terminal or a server, which includes a general computer hardware structure, and at least includes a processor 61 and a memory 62. The processor 61 and the memory 62 are connected through a bus 63. The memory 62 is adapted to store instructions or programs executable by the processor 61. The processor 61 can be an independent microprocessor or a set of one or more microprocessors. Thus, the processor 61 processes data and controls other devices by executing the instructions stored in the memory 62 to implement the method flow of the embodiment of the present invention as described above. The bus 63 connects the above-mentioned multiple components together, and at the same time connects the above-mentioned components to a display controller 64, a display device, and an input / output (I / O) device 65. The input / output (I / O) device 65 can be a mouse, a keyboard, a modem, a network interface, a touch input device, a somatosensory input device, a printer, and other devices well known in the art. Typically, the input / output (I / O) device 65 is connected to the system through an input / output (I / O) controller 66.

[0102] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a device (equipment), or a computer program product. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0103] The present application is described with reference to the flowcharts of the method, the device (equipment), and the computer program product according to the embodiments of the present application. It should be understood that each process in the flowchart can be implemented by computer program instructions.

[0104] These computer program instructions can be stored in a computer-readable memory capable of guiding a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction device that implements the process Figure 1 or the functions specified in one process or multiple processes.

[0105] These computer program instructions can also be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one process Figure 1 or multiple processes.

[0106] Another embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program, and the computer-readable program is used for a computer to execute some or all of the above method embodiments.

[0107] That is, those skilled in the art can understand that all or part of the steps for implementing the above method embodiments can be completed by specifying relevant hardware through a program. The program is stored in a storage medium and includes several instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0108] The above are only the preferred embodiments of the present application and are not used to limit the present application. For those skilled in the art, various modifications and changes can be made to the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included within the protection scope of the present application.

Claims

1. A method for processing text data, characterized in that, The method includes: Obtaining text data to be processed; Determining a candidate entity set corresponding to each entity name in the text data; For each entity name, determining a target entity linked by the entity name in the corresponding candidate entity set; Annotating the position and represented type of each target entity in the text data to determine an additional vector; Inputting the additional vector and the embedding representation of the text data into a target model to determine intent information and slot information in the text data through the target model.

2. The method according to claim 1, characterized in that, Determining a candidate entity set corresponding to each entity name in the text data includes: Performing entity recall according to each entity name to determine a candidate entity set corresponding to each entity name; and / or Determining a vector representation of the text data and performing entity recall according to the vector representation of the text data to determine a candidate entity set corresponding to each entity name.

3. The method according to claim 1 or 2, characterized in that, Before determining a candidate entity set corresponding to each entity name in the text data, the method further includes: Determining entity names in the text data.

4. The method according to claim 3, characterized in that Determining entity names in the text data includes: Matching the text data with an entity dictionary based on a rule matching method to determine entity names in the text data; and / or Performing coarse-grained word segmentation on the text data and matching the word segmentation results with each entity in the entity dictionary to determine entity names in the text data; and / or Inputting the text data into an entity extraction model to determine entity names in the text data through the entity extraction model.

5. The method according to claim 1, wherein Determining a target entity linked by the entity name in the corresponding candidate entity set includes: For each candidate entity in the candidate entity set, determining the matching degree between the candidate entity and the text data; Determining the target entity linked by the entity name among the candidate entities at least according to the matching degree between each candidate entity and the text data.

6. The method according to claim 5, wherein Determining the matching degree between the candidate entity and the text data includes: Determining the matching degree between the description information of the candidate entity and the text data; Determining the matching degree between the candidate entity and the text data as the matching degree between the description information of the candidate entity and the text data.

7. The method according to claim 5, characterized in that Determining the target entity linked by the entity name among the candidate entities at least according to the matching degree between each candidate entity and the text data includes: Determining the target entity linked by the entity name among the candidate entities according to the matching degree between each candidate entity and the text data and disambiguation features; Wherein, the disambiguation features at least include a combination of one or more of the following: the popularity of the candidate entity, the type of the candidate entity, and the current scene information.

8. The method according to claim 1, wherein Annotating the position and represented type of the target entity in the text data to determine an additional vector includes: Annotating the position and represented type of the target entity in the text data to generate an additional matrix; Perform dimensionality transformation on the additional matrix to determine an additional vector, where the dimension of the additional vector is the same as the dimension of the embedded representation of the text data.

9. The method according to claim 1 or 8, characterized in that, The embedded representation of the text data includes word embeddings, segment embeddings, and position embeddings.

10. The method according to claim 1, wherein Obtaining the text data to be processed includes: Obtaining the text data to be processed by receiving the input text; or Obtaining the text data to be processed by receiving the input voice data and performing speech recognition on the voice data.

11. A text data processing device, characterized in that, The device includes: An obtaining unit, configured to obtain text data to be processed; A candidate entity determination unit, configured to determine a set of candidate entities corresponding to each entity name in the text data; A target entity determination unit, configured to determine, for each entity name, a target entity linked by the entity name in the corresponding set of candidate entities; A vector determination unit, configured to label the positions and represented types of the target entities in the text data to determine an additional vector; An information determination unit, configured to input the additional vector and the embedded representation of the text data into a target model to determine the intent information and slot information in the text data through the target model.

12. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, The computer program instructions, when executed by a processor, implement the method according to any one of claims 1-10.

13. An electronic device, characterized in that, The device includes: A memory, configured to store one or more computer program instructions; A processor, where the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1-10.

14. A computer program product, characterized in that, When the computer program product runs on a computer, the computer is caused to execute the method according to any one of claims 1-10.

Images