Text and table semantic interaction data processing method and device, and storage medium
By uniformly transforming text and table features through the embedding layer to form three types of embeddings, and performing deep semantic interaction in the transformer layer, the problem of insufficient interaction between table features and text features in existing technologies is solved, thereby improving the modeling effect and making it suitable for scenarios such as recommendation, advertising, and risk control.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA PING AN LIFE INSURANCE CO LTD
- Filing Date
- 2022-07-21
- Publication Date
- 2026-06-05
AI Technical Summary
In existing technologies, modeling methods for table features and text features have failed to effectively achieve deep interaction, resulting in poor modeling performance, especially in recommendation, advertising, and risk control scenarios where information is not fully utilized.
The text and table features are uniformly transformed through the embedding layer to form token embedding, position embedding and token type embedding. These are then input into the transformer layer of the neural network for deep semantic interaction, and finally the probability distribution information is obtained through the classification layer.
It achieves deep semantic interaction between table features and text features, improves modeling performance, and can better utilize information from both modalities, making it suitable for scenarios such as recommendation, advertising, and risk control.
Smart Images
Figure CN115204300B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a data processing method, apparatus and storage medium for semantic interaction of text and tables. Background Technology
[0002] In scenarios such as recommendation, advertising, and risk control, it is often necessary to model both tabular and textual features simultaneously, but the application results are not ideal. Related technologies for modeling tabular and textual features often fail to consider the deep interactions between them. For example, modeling only tabular or only textual features discards information from one modality. Another example is a textual feature model that outputs a probability value, 0 / 1 feature, or embedding vector based on textual information and uses it as input to a tabular feature model. Yet another example is a dual-tower model where the tabular and textual feature models are built separately, and their latent vectors interact superficially at the final output layer. These methods either only model a single modality or only achieve superficial interactions between the two modalities of features or information. Summary of the Invention
[0003] The main objective of this application is to propose a data processing method, apparatus, and storage medium for semantic interaction between text and tables, which can realize deep interaction between text features and table features, thereby achieving deep and complex semantic interaction.
[0004] To achieve the above objectives, a first aspect of this application proposes a data processing method for semantic interaction between text and tables, comprising:
[0005] Obtain input data, which includes text information and table information;
[0006] The text information of the input data is subjected to feature extraction to obtain text features, and the table information of the input data is subjected to feature extraction to obtain table features;
[0007] The text features and the table features are input into the embedding layer of a preset neural network model, so that the embedding layer transforms the text features and the table features to obtain token embedding, position embedding and token type embedding; wherein the token embedding represents the feature value of the characters in the text information and the value of the table information, the position embedding represents the positional relationship between the characters in the text information and the value of the table information, and the token type embedding is used to distinguish the text information and the table information;
[0008] Each embedding item obtained by the embedding layer is input into the transformer layer of the preset neural network to obtain a hybrid semantic vector;
[0009] The hybrid semantic vector is input into the classification layer of the preset neural network to obtain the predicted probability information of the text and table semantics corresponding to the input data.
[0010] In some embodiments, during the transformation of the text features at the embedding layer, the data processing method further includes:
[0011] Based on the text features, determine the position of each character in the text information, the starting position of the words formed by the characters, and the separation position between words;
[0012] In the embedding layer, each character is used as a regular text token according to its position, [cls] is set as a special text token at the beginning position, and [sep] is set as a special text token at the separator position;
[0013] For each text token, take the corresponding embedding to obtain the tokenembedding corresponding to the text information.
[0014] In some embodiments, during the transformation of the table features at the embedding layer, the data processing method further includes:
[0015] Based on the positional order of [cls], each character, and [sep], assign a numeric identifier starting from 0 to each token;
[0016] Based on the table features, the value of each cell in the table information is determined, and in the embedding layer, each cell value is used as a table token;
[0017] The table header identifier is obtained by identifying the table token corresponding to the table header.
[0018] In the embedding layer, the numeric identifier and the header identifier are used as the position;
[0019] For each position, take the corresponding embedding to obtain the position embedding.
[0020] In some embodiments, the token type embedding is obtained in the following way:
[0021] Set the first type of token type for each text token;
[0022] Set a second type of token type for each form token;
[0023] For each token type of the first type and each token type of the second type, take the corresponding embedding to obtain the token type embedding.
[0024] In some embodiments, the step of inputting the various embedding terms obtained by the embedding layer into the transformer layer of the preset neural network to obtain a hybrid semantic vector includes:
[0025] Each embedding item is treated as a whole sample, and the size of the whole sample is defined as (L, H), where L represents the length of the token list, H represents the length of the embedding latent vector, and the length of the token list represents the number of token embedding items.
[0026] The overall sample is transformed using a transformer to output a hybrid semantic vector.
[0027] In some embodiments, the hybrid semantic vector is represented as A = transformer(E), where E is the total representation of all embedding terms, and the predicted probability information output by the classification layer is represented as P; the step of inputting the hybrid semantic vector into the classification layer of the preset neural network to obtain the predicted probability information of the text and table semantics corresponding to the input data includes:
[0028] Starting with the first term of the mixed semantic vector A, assign the value O = A0;
[0029] The predicted probability information is obtained by calculating the following formula:
[0030]
[0031] Where H represents the length of the embedding latent vector, W and b are learnable model parameters, and W is a vector of size H.
[0032] In some embodiments, after obtaining the predicted probability information, the data processing method further includes:
[0033] Calculate the loss based on the predicted probability information;
[0034] The gradient is calculated backwards based on the loss, and the model parameters are updated.
[0035] To achieve the above objectives, a second aspect of this application provides a data processing apparatus, comprising:
[0036] A data acquisition unit is used to acquire input data, which includes text information and table information;
[0037] The feature extraction unit is used to extract features from the text information of the input data to obtain text features, and to extract features from the table information of the input data to obtain table features;
[0038] An embedding unit is used to input the text features and the table features into the embedding layer of a preset neural network model, so that the embedding layer transforms the text features and the table features to obtain token embedding, position embedding, and token type embedding; wherein the token embedding represents the feature values of the characters in the text information and the values in the table information, the position embedding represents the positional relationship between the characters in the text information and the values in the table information, and the token type embedding is used to distinguish the text information and the table information;
[0039] The transformer unit is used to input the various embedding items obtained by the embedding layer into the transformer layer of the preset neural network to obtain a hybrid semantic vector;
[0040] The classification output unit is used to input the hybrid semantic vector into the classification layer of the preset neural network to obtain the predicted probability information of the text and table semantics corresponding to the input data.
[0041] To achieve the above objectives, a third aspect of this application provides a computer device, the computer device including a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for implementing communication between the processor and the memory, wherein the program, when executed by the processor, implements the data processing method described in the first aspect above.
[0042] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs that can be executed by one or more processors to implement the data processing method described in the first aspect.
[0043] The data processing method, apparatus, and storage medium for text and table semantic interaction proposed in this application perform a unified transformation on both text and table features through an embedding layer, forming three different types of embeddings: token embedding representing the content of text and table information, position embedding representing the location of the token, and token type embedding used to distinguish between text and table features. These three types of embeddings are then input into the transformer layer of a neural network to achieve deep semantic interaction between text and tables. Finally, probability distribution information is obtained through a classification layer. Through this process, the format of table and text features is consistent at the embedding level, and both table and text features are modeled as token units to facilitate deep interaction within the transformer. Attached Figure Description
[0044] Figure 1 This is a flowchart of a data processing method for semantic interaction between text and tables provided in an embodiment of this application;
[0045] Figure 2 This is a flowchart of obtaining the token embedding corresponding to text information based on text features, provided in an embodiment of this application.
[0046] Figure 3 This is a flowchart provided in an embodiment of the present application for obtaining the token embedding corresponding to the table information based on the table features, and obtaining the position embedding;
[0047] Figure 4 This is a flowchart of obtaining the token type embedding provided in the embodiments of this application;
[0048] Figure 5 This is a flowchart illustrating the process of obtaining a hybrid semantic vector, as provided in an embodiment of this application.
[0049] Figure 6 This is a flowchart of obtaining predicted probability information provided in an embodiment of this application;
[0050] Figure 7 This is a flowchart of the updated model parameters provided in the embodiments of this application;
[0051] Figure 8 This is a block diagram of the module structure of the data processing method apparatus for semantic interaction of text and tables provided in the embodiments of this application;
[0052] Figure 9 This is a schematic diagram of the hardware structure of the computer device provided in the embodiments of this application. Detailed Implementation
[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0054] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.
[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.
[0056] First, let's analyze some of the terms used in this application:
[0057] Artificial Intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.
[0058] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information and image processing, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.
[0059] Information Extraction (NER) is a text processing technique that extracts factual information such as entities, relationships, and events from natural language text and outputs it as structured data. Information extraction is a technique for extracting specific information from text data. Text data is composed of specific units, such as sentences, paragraphs, and chapters. Text information is composed of smaller, specific units, such as characters, words, phrases, sentences, paragraphs, or combinations of these units. Extracting noun phrases, names of people, and place names from text data is an example of text information extraction. Of course, text information extraction techniques can extract information of various types.
[0060] Artificial Neural Networks (ANNs), also simply called neural networks (NNs) or connection models, are mathematical models that mimic the behavioral characteristics of animal neural networks to perform distributed parallel information processing. These networks rely on system complexity to adjust the connections between a large number of internal nodes to process information. Their main task is to build practical artificial neural network models based on the principles of biological neural networks and the needs of real-world applications, design corresponding learning algorithms, simulate certain intelligent activities of the human brain, and then technically implement them to solve practical problems. Therefore, biological neural networks mainly study the mechanisms of intelligence; artificial neural networks mainly study the implementation of these mechanisms, and the two complement each other. Regardless of the type, artificial neural networks share common characteristics such as large-scale parallel processing, distributed storage, flexible topology, high redundancy, and nonlinear computation. Consequently, they possess high computing speed, strong associative ability, strong adaptability, strong fault tolerance, and self-organizing ability.
[0061] Currently, table analysis often requires analyzing the table's structure and the relationships between its contents to provide analytical results that help analysts apply the data to data analysis scenarios such as advertising and risk control. Using neural network algorithms for table analysis necessitates feature extraction, yielding both table features and text features. Table features primarily refer to fields recorded as discrete values or numerical values within the system, while text features mainly refer to the fixed text within the system. Due to the dual nature of tables, modeling and analysis in related technologies often cannot achieve bimodal modeling or can only achieve shallow interactions between the two modalities in bimodal modeling.
[0062] Based on this, the main objective of this application is to provide a data processing method, apparatus, and storage medium for semantic interaction between text and tables. The method involves uniformly transforming both text features and table features through an embedding layer to form three different types of embeddings: token embedding representing the content of text and table information, position embedding representing the location of the token, and tokentype embedding used to distinguish between text features and table features. Then, based on the three types of embeddings obtained from the transformation, the data is input into the transformer layer of a neural network to achieve deep semantic interaction between text and tables. Finally, probability distribution information is obtained through a classification layer.
[0063] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.
[0064] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.
[0065] The data processing method for text and table semantic interaction provided in this application relates to the field of artificial intelligence technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the data processing method for text and table semantic interaction, but is not limited to the above forms.
[0066] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.
[0067] This application provides a data processing method, apparatus, and storage medium for semantic interaction between text and tables. The specific implementation details are provided in the following embodiments. First, the data processing method for semantic interaction between text and tables in this application is described.
[0068] Figure 1 This is an optional flowchart of a data processing method for semantic interaction between text and tables provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S107.
[0069] Step S101: Obtain input data, which includes text information and table information;
[0070] Step S102: Extract features from the text information of the input data to obtain text features; extract features from the table information of the input data to obtain table features.
[0071] Step S103: Input the text features and table features into the embedding layer of the preset neural network model, so that the embedding layer transforms the text features and table features to obtain token embedding, position embedding and token type embedding; where token embedding represents the feature value of the characters in the text information and the value of the table information, position embedding represents the positional relationship between the characters in the text information and the value of the table information, and token type embedding is used to distinguish between text information and table information;
[0072] Step S104: Input the various embedding terms obtained by the embedding layer into the transformer layer of the preset neural network to obtain the hybrid semantic vector;
[0073] Step S105: Input the hybrid semantic vector into the classification layer of the preset neural network to obtain the predicted probability information of the text and table semantics corresponding to the input data.
[0074] The input data is generally a table containing data. In the table, the fixed text part is used as text information, and the discrete text and discrete values part is used as table features. Based on this, a deep interactive semantic analysis is performed on the table.
[0075] Specifically, the text information is first subjected to feature extraction to obtain text features, and the table information is subjected to feature extraction to obtain table features. The extracted text features and table features are then input into a pre-defined neural network model. This pre-defined neural network model includes an embedding layer, which is used to transform the text features and table features to obtain three types of embedding items. The embedding layer is designed to embed three types of objects, including token, position, and token type. The token is transformed into a token embedding in the embedding layer, the position is transformed into a position embedding, and the token type is transformed into a token type embedding. The token represents the characters in the text information and the values in the table information. Therefore, the token itself can be divided into character tokens and table tokens, which will be explained in detail later. The position represents the position of the token in the table. The position is used to locate each token, and the token type is used to distinguish between text tokens and table tokens. It can be distinguished by 0 and 1 values, or directly by "text" and "table", or by other types of binarization distinction methods. The above transformation is performed at the embedding layer, converting both text and table features into a unified form, that is, using token units for representation. Then, for different token units, their position and type are distinguished by position and token type, thereby enabling deep semantic interaction between text tokens and table tokens in subsequent transformer layers. It can be understood that the embedding layer needs to transform token, position, and token type, resulting in token embedding, position embedding, and token type embedding, respectively.
[0076] Therefore, after obtaining each embedding item (token embedding, position embedding, and token type embedding), these embedding items are input into the transformer layer for interaction. After interaction, a hybrid semantic vector is obtained, that is, text features and table features are mixed and transformed in the transformer layer. Then, the hybrid semantic vector is input into the classification layer of the preset neural network to predict the probability distribution, thereby obtaining the predicted probability information of the text and table semantics corresponding to the input data.
[0077] The above process describes the overall data processing procedure for the table. To clearly illustrate the data processing method of this application, the above steps will be described in detail below.
[0078] Reference Figure 2 As shown, in some embodiments, during the transformation of text features at the embedding layer, the data processing method further includes the following steps:
[0079] Step S201: Determine the starting position of each character in the text information, the starting position of the words formed by the characters, and the separating positions between words based on the text features;
[0080] In step S202, in the embedding layer, each character is used as a regular text token according to its position order, and [cls] is set as a special text token at the beginning and [sep] is set as a special text token at the separator position.
[0081] Step S203: Take the corresponding embedding for each text token to obtain the token embedding corresponding to the text information.
[0082] In the text features, each character of the text information is identified, along with the starting position of words composed of these characters and the separating positions between words. Each character is treated as a text token. The starting position is set to [cls] and [cls] is used as a special text token. The separating position is set to [sep] and [sep] is also used as a special text token. Therefore, for text information, there are three formats of text tokens in the embedding layer: the special text token corresponding to [cls], the regular text token, and the special text token corresponding to [sep].
[0083] The above describes how the embedding layer transforms text features to obtain a text token. The embedding layer also receives table features and transforms them to obtain a table token. Specifically, refer to... Figure 3As shown, the data processing methods during the transformation of table features in the embedding layer also include:
[0084] Step S301: Assign a numbered identifier starting from 0 to each token according to the positional order of [cls], each character and [sep];
[0085] Step S302: Determine the value of each cell in the table information based on the table features. In the embedding layer, each cell value serves as the table token.
[0086] Step S303: Identify the header according to the table token to obtain the header identifier;
[0087] Step S304: In the embedding layer, the numeric identifier and the header identifier are used as positions;
[0088] Step S305: Take the corresponding embedding for each position to obtain the position embedding.
[0089] The table tokens obtained from the table feature transformation each represent a feature value. Therefore, unlike text tokens, table tokens are not divided by characters but are related to the table feature values. The position is obtained by sorting the table tokens and text tokens. Since text tokens are inherently character-based and have a set order, the position can be directly obtained from the order of the text tokens during embedding. Table tokens, however, have their position determined by the order of discrete text and discrete values within the table. For example, they can be sorted from left to right according to the table header order, or the order can be predefined using annotations, etc., without further limitation.
[0090] The above sorting process uses text tokens, which are obtained based on character order. Therefore, the order of text tokens can be determined by numbers, such as 0, 1, 2, 3, etc., which can be represented as numeric identifiers. Table tokens are sorted according to the relationship of the table headers, so table identifiers are used to record the sorting. These table identifiers can directly use the values from the table headers to determine the sorting method. Therefore, positions are generated based on the numeric identifiers and table header representations. Then, the corresponding embedding is taken for each position to obtain the position embedding.
[0091] On the other hand, token types are directly distinguished based on textual and table features. Specifically, refer to... Figure 4 As shown, the token type embedding is obtained in the following way:
[0092] Step S401: Set the first type of token type for each text token;
[0093] Step S402: Set the second type of token type for each form token;
[0094] Step S403: Take the corresponding embedding for each first type of token type and each second type of token type to obtain the token type embedding.
[0095] For text tokens, the first type of token type is used, and for table tokens, the second type of token type is used. The values of the first type of token type and the second type of token type can be set according to actual needs. For example, the first type of token type can be directly represented by "text", and the second type of token type can be directly represented by "table". There is no restriction here.
[0096] Then, the corresponding embeddings for the first type of token type and the second type of token type are obtained to get the token type embedding.
[0097] Here's a simple example illustrating the conversion of embedding:
[0098] The input to the embedding layer can be as shown in Table 1 below:
[0099] Table 1. Input of the embedding layer
[0100] text Department Code Warning line Reason for application hypertension 17 Y disease
[0101] After embedding processing, the three types of embedding items are represented in Table 2 below:
[0102]
[0103]
[0104] After the embedding layer, both text features and table features are converted into a unified format, namely token units. The position of the token units is distinguished by their position, and the type of the token units is distinguished by their type, which facilitates processing in the subsequent transformer layer.
[0105] The Transformer model is widely used in natural language processing, including machine translation, question answering systems, text summarization, and speech recognition. Like the Attention model, the Transformer employs an encoder-decoder architecture. The encoder generates an attention-based representation, enabling it to locate a specific piece of information within a larger context. The decoder function extracts information from the encoder's representation. The structure is very similar to the encoder, except that the decoder contains two multi-head attention sub-modules instead of one in each identical repeating module. The first multi-head attention sub-module is masked to prevent positional traversal.
[0106] This application can be applied to conventional transformer layers. It is understood that the transformer layer considers positional information to ensure temporal sequence in the output. The positional information is provided by the aforementioned `position`, and the temporal characteristics of the transformer layer's output are ensured through the vector of sequence features corresponding to `positionembedding`. Since this application does not impose any restrictions on the transformer layer, its specific structure is not limited.
[0107] The transformer layer of this application takes as input the aforementioned embedding terms, and therefore can output a hybrid semantic vector. Specifically, refer to... Figure 5 As shown, the various embedding terms obtained from the embedding layer are input into the transformer layer of a preset neural network to obtain a hybrid semantic vector, including:
[0108] Step S501: Treat each embedding item as a whole sample and determine the size of the whole sample as (L, H), where L represents the length of the token list and H represents the length of the embedding latent vector. The length of the token list represents the number of token embedding items.
[0109] Step S502: Transform the overall sample according to the transformer and output the hybrid semantic vector.
[0110] Each embedding item is represented as a total sample and input into the transformer layer. After encoding and decoding within the transformer layer, a hybrid semantic vector is obtained. The size of the total sample is represented as (L, H), where L represents the length of the token list. The token list is obtained from the text tokens and table tokens mentioned above. The token list is formed by listing all token units. Since the embedding layer needs to embed each token unit, a token list is formed based on each token unit and the length of the token list is recorded before embedding. H represents the length of the embedding latent vector, which will not be explained in detail here.
[0111] In summary, by transforming the overall samples at the transformer layer, a hybrid semantic vector can be obtained, which is then sent to the classification layer to obtain the final output.
[0112] Reference Figure 6 As shown, assuming the hybrid semantic vector is represented as A = transformer(E), where E is the total representation of all embedding terms, and the predicted probability information output by the classification layer is represented as P; the hybrid semantic vector is input into the classification layer of a predefined neural network to obtain the predicted probability information of the text and table semantics corresponding to the input data, including:
[0113] Step S601: Starting from the first vector of the mixed semantic vector A, assign the value O = A0;
[0114] Step S602, calculate the following formula to obtain the predicted probability information:
[0115]
[0116] Where H represents the length of the embedding latent vector, W and b are learnable model parameters, and W is a vector of size H.
[0117] Token embedding can be represented as:
[0118]
[0119] Position embedding can be represented as:
[0120]
[0121] The token type can be represented as:
[0122]
[0123] The total population E, which is the population sample mentioned above, can be represented as:
[0124] E = [e0, e1, e2, ...]
[0125] in, i is a natural number.
[0126] Since the hybrid semantic vector A is obtained by encoding and decoding E through a transformer layer, A can also be represented as:
[0127] A = [A0, A1, A2, ... A L-1 ]
[0128] Where L represents the length of the token list.
[0129] The mixed semantic vector is input into the classification layer, and the final probability distribution is obtained based on the following formula.
[0130]
[0131] W and b are learnable model parameters. The gradient can be calculated backward based on the loss to update these two model parameters, thereby optimizing the data processing process.
[0132] Reference Figure 7 As shown, in terms of optimization, after obtaining the predicted probability information, the data processing methods also include:
[0133] Step S701: Calculate the loss based on the predicted probability information;
[0134] Step S702: Calculate the gradient in reverse based on the loss and update the model parameters.
[0135] The above steps S701 and S702 can be applied to the training process of the preset neural network model, or after the preset neural network model has been trained and put into use. In fact, it is to update the model parameters based on the loss. The calculation of the backward gradient of the loss will not be explained in detail here.
[0136] During training, the input data can be batched. Each input data entry contains two parts: a text portion and a tabular portion, as mentioned above. Therefore, batch processing is possible during model training, with each batch containing multiple data entries. The obtained text and tabular data portions are then input into a pre-defined neural network model that is either untrained or incompletely trained. The model's predicted cover probability is calculated, and the loss is calculated based on the predicted probability and the labels. The gradient is then calculated in reverse, and the model parameters are updated based on the calculation results.
[0137] After training is complete, the actual text and tabular data can be input into the pre-trained neural network model to obtain the actual model prediction probability.
[0138] In summary, the data processing method described above utilizes an embedding layer to uniformly transform both text and table features, resulting in three different types of embeddings: token embedding representing the content of text and table information, position embedding representing the location of the token, and token type embedding used to distinguish between text and table features. These three types of embeddings are then input into the transformer layer of the neural network to achieve deep semantic interaction between text and tables. Finally, the probability distribution information is obtained through a classification layer. This process ensures that the format of table and text features is consistent at the embedding level, modeling both features as token units to facilitate deep interaction within the transformer.
[0139] Please see Figure 8 This application also provides a data processing apparatus that can implement the above-described data processing method for semantic interaction between text and tables. Figure 8The present application provides a block diagram of the module structure of a data processing device, which includes: a data acquisition unit 801, a feature extraction unit 802, an embedding unit 803, a transformer unit 804, and a classification output 805. The system includes a data acquisition unit 801 for acquiring input data, which includes text information and table information; a feature extraction unit 802 for extracting features from the text information of the input data to obtain text features, and extracting features from the table information of the input data to obtain table features; an embedding unit 803 for inputting the text features and table features into the embedding layer of a preset neural network model, such that the embedding layer transforms the text features and table features to obtain token embedding, position embedding, and token type embedding; wherein the token embedding represents the feature value of the characters in the text information and the value of the table information, the position embedding represents the positional relationship between the characters in the text information and the value of the table information, and the token type embedding is used to distinguish between the text information and the table information; a transformer unit 804 for inputting each embedding item obtained by the embedding layer into the transformer layer of the preset neural network to obtain a hybrid semantic vector; and a classification output unit 805 for inputting the hybrid semantic vector into the classification layer of the preset neural network to obtain the predicted probability information of the text and table semantics corresponding to the input data.
[0140] The data processing apparatus of this application embodiment can uniformly transform both text features and table features through an embedding layer, forming three different types of embeddings: token embedding representing the content of text and table information, position embedding representing the location of the token, and token type embedding used to distinguish between text features and table features. Then, based on the three types of embeddings obtained through transformation, the data is input into the transformer layer of a neural network to achieve deep semantic interaction between text and tables. Finally, probability distribution information is obtained through a classification layer. Through the above process, the format of table features and text features is made consistent at the embedding level, and both table features and text features are modeled as token units to facilitate deep interaction in the transformer.
[0141] It should be noted that the specific implementation of this data processing device is basically the same as the specific implementation of the data processing method for text and table semantic interaction described above, and will not be repeated here.
[0142] This application also provides a computer device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned data processing method for text and table semantic interaction. This computer device can be any smart terminal, including tablet computers, in-vehicle computers, etc.
[0143] Please see Figure 9 , Figure 9 The hardware structure of a computer device according to another embodiment is illustrated. The computer device includes:
[0144] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.
[0145] The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is invoked by the processor 901 to execute the data processing method for text and table semantic interaction in the embodiments of this application.
[0146] The input / output interface 903 is used to implement information input and output;
[0147] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).
[0148] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);
[0149] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.
[0150] This application also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, which can be executed by one or more processors to implement the above-described data processing method for semantic interaction of text and tables.
[0151] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0152] The text and table semantic interaction data processing method, data processing device, computer equipment, and storage medium provided in this application embodiment extract text from the original corpus to obtain a preliminary summary text that best expresses the meaning of the original corpus. The preliminary summary text is then divided according to a preset segment length to obtain target segment text. The target segment text is filtered to determine target candidate word groups. Semantic parsing of the target candidate word groups is then performed. The importance of each word segment is determined by its word type, and the word segments are linked to a preset hyper- and hypo-level cognitive graph based on their word types to obtain the target hyper- and hypo-level cognitive graph. Since the hyper- and hypo-level information of each word segment and the relationships between each word segment can be easily obtained through the target hyper- and hypo-level cognitive graph, the text and table semantic interaction data processing method of this application embodiment can improve the efficiency of information acquisition.
[0153] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.
[0154] It will be understood by those skilled in the art that Figure 1-7 The technical solutions shown do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.
[0155] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0156] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.
[0157] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0158] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.
[0159] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0160] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0161] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0162] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0163] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.
Claims
1. A data processing method for semantic interaction between text and tables, characterized in that, include: Obtain input data, which includes text information and table information; The text information of the input data is subjected to feature extraction to obtain text features, and the table information of the input data is subjected to feature extraction to obtain table features; The text features and the table features are input into the embedding layer of a preset neural network model, so that the embedding layer transforms the text features and the table features to obtain token embedding, position embedding and token type embedding; wherein the token embedding represents the feature value of the characters in the text information and the value of the table information, the position embedding represents the positional relationship between the characters in the text information and the value of the table information, and the token type embedding is used to distinguish the text information and the table information; Each embedding item obtained by the embedding layer is input into the transformer layer of the preset neural network to obtain a hybrid semantic vector; The hybrid semantic vector is input into the classification layer of the preset neural network to obtain the predicted probability information of the text and table semantics corresponding to the input data; The step of inputting each embedding item obtained from the embedding layer into the transformer layer of the preset neural network to obtain a hybrid semantic vector includes: Each embedding item is treated as a whole sample, and the size of the whole sample is defined as (L, H), where L represents the length of the token list and H represents the length of the embedding latent vector. The length of the token list represents the number of token embedding items. The overall sample is transformed using a transformer to output a hybrid semantic vector.
2. The data processing method according to claim 1, characterized in that, During the transformation of the text features in the embedding layer, the data processing method further includes: Based on the text features, determine the position of each character in the text information, the starting position of the words formed by the characters, and the separation position between words; In the embedding layer, each character is used as a regular text token according to its position, [cls] is set as a special text token at the beginning position, and [sep] is set as a special text token at the separator position; For each text token, take the corresponding embedding to obtain the token embedding corresponding to the text information.
3. The data processing method according to claim 2, characterized in that, During the transformation of the table features in the embedding layer, the data processing method further includes: Based on the positional order of [cls], each character, and [sep], assign a numeric identifier starting from 0 to each token; Based on the table features, the value of each cell in the table information is determined, and in the embedding layer, each cell value is used as a table token; The table header identifier is obtained by identifying the table token corresponding to the table header. In the embedding layer, the numeric identifier and the header identifier are used as the position; For each position, take the corresponding embedding to obtain the position embedding.
4. The data processing method according to claim 3, characterized in that, The token type embedding is obtained in the following way: Set the first type of token type for each text token; Set a second type of token type for each form token; For each token type of the first type and each token type of the second type, take the corresponding embedding to obtain the token type embedding.
5. The data processing method according to claim 1, characterized in that, The hybrid semantic vector is represented as: E is the total representation of all embedding terms, and the predicted probability information output by the classification layer is represented as P; the step of inputting the hybrid semantic vector into the classification layer of the preset neural network to obtain the predicted probability information of the text and table semantics corresponding to the input data includes: Starting with the first term of the mixed semantic vector A, assign values... ; The predicted probability information is obtained by calculating the following formula: Where H represents the length of the embedding latent vector, W and b are learnable model parameters, and W is a vector of size H.
6. The data processing method according to claim 5, characterized in that, After obtaining the predicted probability information, the data processing method further includes: Calculate the loss based on the predicted probability information; The gradient is calculated backwards based on the loss, and the model parameters are updated.
7. A data processing apparatus, characterized in that, The device includes: A data acquisition unit is used to acquire input data, which includes text information and table information; The feature extraction unit is used to extract features from the text information of the input data to obtain text features, and to extract features from the table information of the input data to obtain table features; An embedding unit is used to input the text features and the table features into the embedding layer of a preset neural network model, so that the embedding layer transforms the text features and the table features to obtain token embedding, position embedding, and token type embedding; wherein the token embedding represents the feature values of the characters in the text information and the values in the table information, the position embedding represents the positional relationship between the characters in the text information and the values in the table information, and the token type embedding is used to distinguish the text information and the table information; The transformer unit is used to input the various embedding items obtained by the embedding layer into the transformer layer of the preset neural network to obtain a hybrid semantic vector; The classification output unit is used to input the hybrid semantic vector into the classification layer of the preset neural network to obtain the predicted probability information of the text and table semantics corresponding to the input data; Specifically, the transformer unit is used for: Each embedding item is treated as a whole sample, and the size of the whole sample is defined as (L, H), where L represents the length of the token list and H represents the length of the embedding latent vector. The length of the token list represents the number of token embedding items. The overall sample is transformed using a transformer to output a hybrid semantic vector.
8. A computer device, characterized in that, The computer device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for establishing communication between the processor and the memory. When the program is executed by the processor, it implements the steps of the data processing method as described in any one of claims 1 to 6.
9. A storage medium, said storage medium being a computer-readable storage medium for computer-readable storage, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the data processing method according to any one of claims 1 to 6.