Text data processing method and text data processing device

By fusing the pre-trained language model with compiled neural networks to generate a text data processing model, the problem of language model poor performance in rule-intensive tasks is solved, efficient symbolic and arithmetic reasoning is achieved, and the generalization of the model is improved. Adaptability and flexibility.

CN118982003APending Publication Date: 2024-11-19INST OF AUTOMATION CHINESE ACAD OF SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410994182.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

Existing language models perform poorly in rule-intensive tasks, especially inaccurate efficiency, poor scalability and lack of complete variability, limiting their universality and flexibility.

Method used

By fusing the pre-trained language model with a compiled neural network, a text data processing model is generated. The compiled neural network dynamically allocates neural network weights according to the task and performs rule-intensive tasks, avoiding dependence on external tools.

Benefits of technology

It significantly improves the performance of language models in rule-intensive tasks, especially in symbolic reasoning and arithmetic reasoning tasks, improves inference efficiency and scalability, and enhances the model's rule understanding ability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982003A_ABST
    Figure CN118982003A_ABST
Patent Text Reader

Abstract

The invention provides a text data processing method and a text data processing device. The text data processing method includes: acquiring text data, and inputting the text data into a text data processing model generated by fusing a language model and a compilation type neural network, the language model being trained in advance, and the compilation type neural network being a compilation type neural network; corresponding neural network weights are allocated to the compilation type neural network according to the rule-intensive tasks to be executed under the condition that the compilation type neural network is not trained in advance; determining a text data processing task corresponding to the text data from the text data based on a language model in the text data processing model; and in response to the text data processing task indicating the rule-intensive task, executing the rule-intensive task based on a compilation neural network in the text data processing model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and more specifically, to a text data processing method and a text data processing device. Background Art

[0002] Language models have shown excellent performance in processing text data, especially in language understanding tasks. However, when faced with tasks that are rule-intensive and / or require precise reasoning (e.g., rule-intensive tasks, such as mathematical reasoning or symbolic manipulation), statistical learning based solely on models cannot meet the requirements, which limits the performance of models in these tasks.

[0003] Existing technologies enhance the computing power of language models by writing fine-grained tasks or using tools to support these rule-intensive tasks. However, these methods not only require external assistance, but also have problems such as low reasoning efficiency, poor scalability, and the lack of complete model differentiation, which restricts the universality and flexibility of the model. Summary of the invention

[0004] Therefore, it is particularly important and urgent to invent a method that can directly improve the performance of language models in rule-intensive tasks without the need for external tools.

[0005] The purpose of the present disclosure is to provide a text data processing method and a text data processing device.

[0006] In one general aspect, a text data processing method is provided, the text data processing method comprising: acquiring text data, and inputting the text data into a text data processing model generated by fusing a language model with a compiled neural network, wherein the language model is pre-trained, and the compiled neural network is assigned corresponding neural network weights according to a rule-intensive task to be performed if it is not pre-trained; determining a text data processing task corresponding to the text data from the text data based on the language model in the text data processing model; and executing the rule-intensive task based on the compiled neural network in the text data processing model in response to the text data processing task indicating a rule-intensive task.

[0007] Optionally, the compiled neural network includes a first compiled neural network and a second compiled neural network, wherein the rule-intensive task includes a first rule-intensive task and a second rule-intensive task, and wherein the step of executing the rule-intensive task based on the compiled neural network in the text data processing model includes: in response to the text data processing task indicating the first rule-intensive task, executing the first rule-intensive task based on the first compiled neural network, and in response to the text data processing task indicating the second rule-intensive task, executing the second rule-intensive task based on the second compiled neural network.

[0008] Optionally, the step of executing the rule-intensive task based on the compiled neural network in the text data processing model includes: determining task data related to the text data processing task to be executed from the text data based on the language model; and executing the rule-intensive task on the task data based on the compiled neural network.

[0009] Optionally, the rule-intensive tasks include one or more of symbolic operation tasks, symbolic reasoning tasks, and arithmetic reasoning tasks.

[0010] Optionally, the text data processing method further comprises: in response to the text data processing task indicating another task different from the rule-intensive task, executing the another task based on the language model.

[0011] Optionally, the text data processing method further includes: outputting a task result corresponding to a rule-intensive task outputted based on the compiled neural network executing the rule-intensive task to a language model; and outputting a final task result by processing the task result corresponding to the rule-intensive task based on the language model.

[0012] In one general aspect, a text data processing device is provided, comprising: a text data acquisition module configured to acquire text data and input the text data into a text data processing model generated by fusing a language model with a compiled neural network, wherein the language model is pre-trained, and the compiled neural network is assigned corresponding neural network weights according to a rule-intensive task to be performed if it is not pre-trained; a text data processing module configured to: determine a text data processing task corresponding to the text data from the text data based on the language model in the text data processing model; and execute the rule-intensive task based on the compiled neural network in the text data processing model in response to the text data processing task indicating a rule-intensive task.

[0013] Optionally, the compiled neural network includes a first compiled neural network and a second compiled neural network, wherein the rule-intensive task includes a first rule-intensive task and a second rule-intensive task, and wherein the text data processing module is configured to: in response to the text data processing task indicating the first rule-intensive task, execute the first rule-intensive task based on the first compiled neural network, and in response to the text data processing task indicating the second rule-intensive task, execute the second rule-intensive task based on the second compiled neural network.

[0014] Optionally, the text data processing module is configured to: determine task data related to the text data processing task to be performed from the text data based on the language model; and perform rule-intensive tasks on the task data based on the compiled neural network.

[0015] Optionally, the rule-intensive tasks include one or more of symbolic operation tasks, symbolic reasoning tasks, and arithmetic reasoning tasks.

[0016] Optionally, the text data processing method further comprises: in response to the text data processing task indicating another task different from the rule-intensive task, executing the another task based on the language model.

[0017] Optionally, the text data processing module is further configured to: output the task results corresponding to the rule-intensive tasks output based on the compiled neural network to the language model; and output the final task results by processing the task results corresponding to the rule-intensive tasks based on the language model.

[0018] In a general aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by one or more computing devices, the one or more computing devices implement the text data processing method as described in any one of the above items. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other objects and features of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings which exemplarily show an example.

[0020] Figure 1 A flowchart showing a text data processing method according to an exemplary embodiment of the present disclosure is shown.

[0021] Figure 2 A flowchart showing a text data processing method according to an exemplary embodiment of the present disclosure is shown.

[0022] Figure 3 A flowchart showing a text data processing method according to an exemplary embodiment of the present disclosure is shown.

[0023] Figure 4 A flowchart showing a text data processing method according to an exemplary embodiment of the present disclosure is shown.

[0024] Figure 5 A flowchart showing a text data processing method according to an exemplary embodiment of the present disclosure is shown.

[0025] Figure 6 A schematic diagram showing a text data processing model according to an exemplary embodiment of the present disclosure.

[0026] Figure 7 A schematic diagram showing an example code for performing a parity check task according to an exemplary embodiment of the present disclosure.

[0027] Figure 8 Experimental results on multiple datasets are shown.

[0028] Fig. 9 The experimental performance comparison in complex symbolic reasoning tasks of the text data processing method according to the example embodiment of the present disclosure is shown.

[0029] Fig.10 An experimental performance comparison of the AddSub+ dataset of the text data processing method according to an example embodiment of the present disclosure is shown.

[0030] Fig.11 An experimental performance comparison of the GSM8K-Hard dataset of the text data processing method according to an example embodiment of the present disclosure is shown.

[0031] Fig.12 Showing the impact of different functional combinations of compiled neural networks.

[0032] Fig.13 A text data processing apparatus according to an exemplary embodiment of the present disclosure is shown.

[0033] Fig.14 A block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] The following specific embodiments are provided to help the reader gain a comprehensive understanding of the methods, devices and / or systems described herein. However, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear after understanding the disclosure of the present application. For example, the order of operations described herein is only an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for greater clarity and simplicity, descriptions of features known in the art may be omitted.

[0035] The features described herein may be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have been provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein, many of which will be clear after understanding the disclosure of this application.

[0036] As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more.

[0037] Although terms such as "first", "second", and "third" may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions should not be limited by these terms. Instead, these terms are only used to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Therefore, without departing from the teachings of the examples described herein, the first member, first component, first region, first layer, or first portion referred to in the examples may also be referred to as the second member, second component, second region, second layer, or second portion.

[0038] In the specification, when an element (such as a layer, a region, or a substrate) is described as being “on”, “connected to”, or “coupled to” another element, the element may be directly “on”, “connected to”, or “coupled to” another element, or one or more other elements may be present therebetween. Conversely, when an element is described as being “directly on”, “directly connected to”, or “directly coupled to” another element, there may be no other elements present therebetween.

[0039] The terms used herein are only used to describe various examples and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "comprise", "include" and "have" indicate the presence of the described features, quantities, operations, components, elements and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof.

[0040] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by a person of ordinary skill in the art to which the present disclosure belongs after understanding the present disclosure. Unless explicitly defined as such herein, terms (such as those defined in a general dictionary) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and should not be interpreted in an idealized or overly formal manner.

[0041] Furthermore, in the description of examples, when it is considered that a detailed description of a well-known related structure or function would cause vague interpretation of the present disclosure, such a detailed description will be omitted.

[0042] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. However, the embodiments may be implemented in various forms and are not limited to the embodiments described herein.

[0043] Figure 1 A flowchart showing a text data processing method according to an exemplary embodiment of the present disclosure is shown.

[0044] Reference Figure 1In step S110, text data may be acquired and input into a text data processing model generated by fusing a language model (LM) with a compiled neural network (CoNN).

[0045] The text data may be acquired in any manner. In one example, the text data may be acquired by an input device (e.g., a keyboard, etc.). In another example, the text data may be acquired by converting the voice data into text. However, the acquisition method of the text data is not limited thereto and may be acquired in any manner.

[0046] The language model is pre-trained. For example, the language model can handle a variety of natural language tasks (such as but not limited to, text classification, question answering, dialogue).

[0047] The compiled neural network is assigned corresponding neural network weights according to the rule-intensive tasks to be performed without being pre-trained. Rule-intensive tasks may refer to tasks that are rule-intensive and / or require precise reasoning (such as mathematical reasoning or symbol manipulation).

[0048] In step S120 , a text data processing task corresponding to the text data may be determined from the text data based on the language model in the text data processing model.

[0049] That is to say, the language model can process text data and understand task requirements. In one example, the language model can determine the text data processing task of the text data "200 plus 100 equals how much" as the text data processing task "calculate the sum of 200 and 100". In another example, the language model can determine the text data processing task of the text data "200 minus 100 equals how much" as the text data processing task "calculate the difference between 200 and 100". In another example, the language model can determine the text data processing task of the text data "the reciprocal of 200 equals how much" as the text data processing task "calculate the reciprocal of 200". However, the above examples are exemplary, and the text data and the text data processing tasks corresponding to the text data are not limited thereto. The language model can be implemented with any neural network to determine the text data processing tasks corresponding to the text data from the text data.

[0050] In step S130 , in response to the text data processing task indicating a rule-intensive task, the rule-intensive task is executed based on the compiled neural network in the text data processing model.

[0051] Rule-intensive tasks may indicate tasks that are rule-intensive and / or require precise reasoning. For example, rule-intensive tasks may include, but are not limited to, symbolic operation tasks (e.g., by way of example only, parity tasks, reverse tasks, addition tasks, subtraction tasks, and / or various other computational tasks), symbolic reasoning tasks (e.g., by way of example only, symbolic reasoning tasks based on the Coin Flip dataset, symbolic reasoning tasks based on the LastLetter Concatenation dataset), and / or arithmetic reasoning tasks (e.g., by way of example only, arithmetic reasoning tasks based on the GSM8K-Hard dataset, arithmetic reasoning tasks based on the AddSub+ dataset).

[0052] The CoNN according to the example embodiment of the present disclosure is a neural network based on the Transformer architecture, and its weights are obtained by special design rather than pre-training. Each attention layer and multi-layer perceptron layer of the CoNN represents a specific sequence transformation, thereby forming a neural network module that embodies explicit and interpretable operations. The CoNN according to the example embodiment of the present disclosure uses artificially compiled attention weights to execute rules instead of gradient learning. The CoNN according to the example embodiment of the present disclosure can accurately perform symbolic operations and rule-based tasks. The CoNN according to the example embodiment of the present disclosure has higher interpretability because its operations are explicitly defined. The CoNN according to the example embodiment of the present disclosure can be combined with a pre-trained language model to enhance the model's rule understanding ability. The CoNN according to the example embodiment of the present disclosure does not require large-scale pre-training, has a small number of parameters, but can perform complex symbolic operations.

[0053] The prior art enhances the computing power of the language model by fine-grained task writing or using tools to support these rule-intensive tasks. However, these methods not only require external assistance, but also have problems such as low reasoning efficiency, poor scalability, and the model does not have complete differentiation, which restricts the universality and flexibility of the model. In contrast, according to the example embodiments of the present disclosure, a text data processing model is generated by fusing a language model with a compiled neural network, so that when a text data processing task indicates a rule-intensive task, the rule-intensive task is performed based on the compiled neural network in the text data processing model, without the need to enhance the computing power of the language model through fine-grained task writing or using tools. Therefore, according to the example embodiments of the present disclosure, the text data processing method can enhance the rule-intensive task performance of the language model, especially in symbolic reasoning and arithmetic reasoning tasks, showing significant performance advantages.

[0054] Figure 2A flowchart showing a text data processing method according to an exemplary embodiment of the present disclosure is shown.

[0055] In one embodiment, the compiled neural network may include a first compiled neural network and a second compiled neural network, and the rule-intensive task may include a first rule-intensive task and a second rule-intensive task. However, the number of compiled neural networks and rule-intensive tasks of the present disclosure is not limited thereto and may be any other number. For example, each of the compiled neural networks of the present disclosure may execute a corresponding rule-intensive task of a plurality of rule-intensive tasks.

[0056] Reference Figure 2 In step S210, in response to the text data processing task indicating a first rule-intensive task, the first rule-intensive task is executed based on the first compiled neural network.

[0057] For example, the first rule-intensive task can be performed by directly encoding the task rules in the first compiled neural network through manual or automatic compilation, so that the language model does not need to implicitly learn rules from text data when processing the first rule-intensive task (for example, symbolic operations and rule-based tasks).

[0058] In step S220, in response to the text data processing task indicating a second rule-intensive task, the second rule-intensive task is performed based on the second compiled neural network.

[0059] For example, the second rule-intensive task can be performed by directly encoding the task rules in the second compiled neural network through manual or automatic compilation, so that the language model does not need to implicitly learn rules from text data when processing the second rule-intensive task (for example, symbolic operations and rule-based tasks).

[0060] According to an example embodiment of the present disclosure, a text data processing model may include a language model and multiple compiled neural networks (e.g., a first compiled neural network and a second compiled neural network) configured to respectively execute multiple rule-intensive tasks (e.g., a first rule-intensive task and a second rule-intensive task), and the compiled neural network corresponding to the rule-intensive task to be executed is called only when the rule-intensive task needs to be executed, so that due to the fine design of the text data processing model, it ensures that the compiled neural network accurately executes the assigned task without interference from irrelevant modules. Each compiled neural network is good at generating appropriate outputs when needed, thereby preventing erroneous results from irrelevant components.

[0061] Figure 3 A flowchart showing a text data processing method according to an exemplary embodiment of the present disclosure is shown.

[0062] Reference Figure 3 In step S310, task data of a text data processing task to be executed is determined from the text data based on the language model.

[0063] In one example, the language model may determine the task data of the text data "200 plus 100 equals what" as the task data "200" and "100" to be executed for the text data processing task "calculate the sum of 200 and 100". In another example, the language model may determine the task data of the text data "200 minus 100 equals what" as the task data "200" and "100" to be executed for the text data processing task "calculate the difference between 200 and 100". In another example, the language model may determine the task data of the text data "the reciprocal of 200 equals what" as the task data "200" to be executed for the text data processing task "calculate the reciprocal of 200". However, the above examples are exemplary, and the task data may be data that needs to be processed in the text data processing task.

[0064] In step S320, a rule-intensive task is performed on the task data based on the compiled neural network.

[0065] As described above, the compiled neural network is assigned corresponding neural network weights according to the rule-intensive task to be performed without being pre-trained. The compiled neural network can receive task data as input and be assigned corresponding neural network weights according to the rule-intensive task to be performed to process the task data, so as to perform the rule-intensive task and output the task result corresponding to the rule-intensive task.

[0066] Figure 4 A flowchart showing a text data processing method according to an exemplary embodiment of the present disclosure is shown.

[0067] Reference Figure 4 In step S410, in response to the text data processing task indicating another task different from the rule-intensive task, another task is performed based on the language model.

[0068] Another task may be any task that can be performed by a language model other than a rule-intensive task. For example, as an example only, another task may include but is not limited to machine translation, speech recognition, text summarization, text classification, sentiment analysis, and / or named entity recognition, etc. In addition, the language model may be a combination of multiple language models.

[0069] Figure 5 A flowchart showing a text data processing method according to an exemplary embodiment of the present disclosure is shown.

[0070] Reference Figure 5In step S510, the task result corresponding to the rule-intensive task outputted based on the compiled neural network executing the rule-intensive task can be outputted to the language model.

[0071] In one example, the compiled neural network executes a rule-intensive task and outputs a task result "300" corresponding to the rule-intensive task "calculate the sum of 200 and 100". In another example, the compiled neural network executes a rule-intensive task and outputs a task result "100" corresponding to the rule-intensive task "calculate the difference between 200 and 100". In another example, the compiled neural network executes a rule-intensive task and outputs a task result "0.005" corresponding to the rule-intensive task "calculate the inverse of 200". However, the above examples are exemplary, and the task result may be the task result of other rule-intensive tasks.

[0072] In step S520 , a final task result is output by processing the task result corresponding to the rule-intensive task based on the language model.

[0073] In one example, the language model may output the final task result “200 plus 100 equals 300” based on the task result “300” corresponding to the rule-intensive task “calculate the sum of 200 and 100”. In another example, the language model may output the final task result “200 minus 100 equals 100” based on the task result “100” corresponding to the rule-intensive task “calculate the difference between 200 and 100”. In another example, the language model may output the final task result “the inverse of 200 equals 0.005” based on the task result “0.005” corresponding to the rule-intensive task “calculate the inverse of 200”. However, the above examples are exemplary, and the final task results may be other final task results based on the task results corresponding to the rule-intensive tasks.

[0074] Figure 6 A schematic diagram showing a text data processing model according to an exemplary embodiment of the present disclosure.

[0075] According to an example embodiment of the present disclosure, in order to encode rules into a language model, the text data processing model includes a series of compiled neural networks, each of which can be customized for a specific rule-intensive task. Here, this framework may also be referred to as neural understanding. Through intuitively defined rules, the compiled neural network uses specially designed attention weights to perform tasks. In the neural understanding framework, predetermined rules are dynamically introduced through an end-to-end neural network system that includes a pre-trained language model and a compiled neural network.

[0076] Reference Figure 6, the text data processing model may include a language model (e.g., a pre-trained neural network) and a compiled neural network. The text data processing model may accept the input text data "Xiao Ming ran a distance of 364,425 meters and walked a distance of 216,582 meters. How much farther is Xiao Ming's running distance than his walking distance?". The language model in the text data processing model may determine the text data processing task "calculate the difference between Xiao Ming's running distance and his walking distance" corresponding to the text data from the text data "Xiao Ming ran a distance of 364,425 meters and walked a distance of 216,582 meters. How much farther is Xiao Ming's running distance than his walking distance?". In response to the text data processing task "calculate the difference between Xiao Ming's running distance and his walking distance" indicating a rule-intensive task, the compiled neural network in the text data processing model may execute the rule-intensive task (i.e., the text data processing task "calculate the difference between Xiao Ming's running distance and his walking distance") and output the task result "147,843" of executing the rule-intensive task. The compiled neural network can send the task result "147843" to the language model. The language model can output the final task result "Xiao Ming's running distance is 147843 meters farther than his walking distance" based on the task result "147843".

[0077] In an example embodiment of the present disclosure, a pre-trained LM and CoNN can be combined into a neural network framework in a hybrid expert system to perform generative calculations. When calculating 364425-216582, the pre-trained LM outputs 148843, which is incorrect. The specially compiled neural network can correctly output the result 147843. This dynamic encoding process improves the interpretability of the intermediate process and the accuracy of the final result. Specifically, the output H of the pre-trained LM including multiple hidden nodes L1 To H LdL The hidden layer representation output is:

[0078]

[0079] The output H of CoNNs with multiple hidden nodes C1 To H CdC The hidden layer representation output is:

[0080]

[0081] At this time, the predicted target Token is calculated as:

[0082]

[0083] Using the identity matrix according to an exemplary embodiment of the present disclosure and To connect the representation outputs of the two sets of models.

[0084] The determination process of the parameter β is dynamic and based on the type of task currently being processed. When a rule-intensive task (e.g., a symbol manipulation task) needs to be executed, β=1. In other cases, β=0.

[0085] Specifically, in the predicted target Token, when β = 1, the output H of the compiled neural network C When β = 0, only the output H of the language model is considered L .

[0086] For example, for the text data "Xiao Ming has 3 apples, Xiao Hong has 8 apples, how many apples does Xiao Ming have less than Xiao Hong?"

[0087] The processing process is as follows:

[0088] Understand the problem (β = 0). The language model in the text data processing model processes the entire problem and understands the operations that need to be performed. The output may be "need to calculate the result of 8 minus 3. 8-3 =".

[0089] The language model in the text data processing model recognizes that a subtraction operation needs to be performed (β=1). The compiled neural network in the text data processing model is called to perform a symbolic operation (subtraction). The compiled neural network receives the input "8-3" and performs the operation. The output of the compiled neural network: 5.

[0090] The text data processing model generates the final answer (β=0). The language model receives the calculation results of CoNN and generates a complete answer output "Xiao Ming has 5 fewer apples than Xiao Hong".

[0091] In this example, β changes as follows: At the beginning, β = 0, and the language model processes the text understanding part. When a subtraction operation is required (here, the rule processing is set, that is, when encountering the = sign, β = 1), β becomes 1, enabling CoNN to perform precise calculations. After the calculation is completed, β becomes 0 again, allowing the language model to generate the final text answer.

[0092] Figure 7 A schematic diagram showing an example code for performing a parity check task according to an exemplary embodiment of the present disclosure.

[0093] The following describes the rule-intensive task by taking a parity check task as an example, however, the rule-intensive task is not limited to the parity check task and may be any other rule-intensive task.

[0094] The goal of the parity check task is to determine whether the number of 1s in the input binary sequence is odd or even. The following description takes the input [1 0] as an example.

[0095] First, a selection operation (Select(Indices, Indices, True) = [[1 1], [1 1]]) is performed. The selection operation can be used to create an all-1 matrix for subsequent aggregation operations. In the selection operation, an n times n all-1 matrix corresponding to the number of input bits n can be generated. The selection operation can prepare the weight matrix for the next aggregation operation to ensure that every input bit is considered.

[0096] After that, we can perform an aggregation operation (Aggregate([[1 1],[1 1]],[1 0]) = [1 1])). The aggregation operation counts the total number of 1s in the input sequence. More specifically, we perform a matrix multiplication of the input [1 0] with a matrix of all 1s. This step actually counts the number of 1s in the input sequence. Each 1 in the result [1 1] represents the total number of 1s in the input (in this case, 1).

[0097] Finally, a compression dictionary operation can be performed (Zipmap([1 1], Lambda x:0if x%2==0else1)=

[11] ). The compression dictionary operation determines whether the number of 1s is odd or even. In the compression dictionary operation, the lambda function is applied to each element in [1 1]. If it is an even number, it returns 0, and if it is an odd number, it returns 1. This step performs the actual parity check. The result [1 1] indicates an odd number of 1s (if it is an even number, the result will be [0 0]).

[0098] The operation of performing the parity check task requires the selection operation, aggregation operation and compression dictionary operation as described above. Each operation represents a special form of sequence transformation.

[0099] The following will be combined Figures 8 to 12 Description In order to test the effectiveness of the text data processing method according to the exemplary embodiment of the present disclosure, experiments were conducted on eight different generative question answering datasets, including the symbol operation tasks Parity, Reverse, Addition and Subtraction datasets; the symbol reasoning tasks Coin Flip and LastLetter Concatenation datasets; the arithmetic reasoning tasks GSM8K-Hard and AddSub+ datasets. The methods used for comparison include: language model fine-tuning, language model few-sample generation, CoT prompts, and methods based on external tool calls.

[0100] Figure 8 Experimental results on multiple datasets are shown.

[0101] According to the text data processing method of the exemplary embodiment of the present disclosure, a corresponding CoNN model is constructed and combined with the pre-trained LM to produce a neural understanding model that can accurately perform these tasks. Figure 8 In the table, common methods such as fine-tuning, few-shot, Scratchpad and Algorithmic are selected for comparison to illustrate their effectiveness. It can be found from the table that the performance of the text data processing method according to the exemplary embodiment of the present disclosure is significantly better than the benchmark method, which also illustrates the effectiveness of the text data processing method according to the exemplary embodiment of the present disclosure.

[0102] Fig. 9 The experimental performance comparison in complex symbolic reasoning tasks of the text data processing method according to the example embodiment of the present disclosure is shown.

[0103] In one example, in order to further test whether the neural understanding framework has a robust improvement effect on language models of different sizes, a series of complex symbolic reasoning experiments were selected, which involved the combination of language understanding and symbolic manipulation tasks. Fig. 9 As can be seen, the performance of the T5-Small language model with a parameter scale of 60M to the T5-Large language model with a parameter scale of 770M has been significantly improved with the increase in training.

[0104] Fig.10 An experimental performance comparison of the AddSub+ dataset of the text data processing method according to an example embodiment of the present disclosure is shown. Fig.11 An experimental performance comparison of the GSM8K-Hard dataset of the text data processing method according to an example embodiment of the present disclosure is shown.

[0105] exist Fig.10 In the paper, we selected the AddSub+ dataset and the GSM8K-Hard dataset respectively. These datasets involve real-world arithmetic reasoning tasks. It is worth noting that they need to calculate values ​​up to 20 digits long.

[0106] exist Fig.10 and Fig.11In this paper, CoT (using only language models for arithmetic reasoning) and PAL (requiring language models to generate code and call additional tools for execution) were selected as baseline experiments. Compared with the two, Neural Understanding has a higher degree of arithmetic reasoning performance. First, through special design, Neural Understanding minimizes the need for additional processing steps or external tools, thereby achieving a more efficient direct calculation method. This is in stark contrast to tool-based methods, which usually require additional programming and execution steps, increasing complexity and computing resources. In addition, since compiled neural networks are end-to-end differentiable, it is essential for models to adapt to new data or tasks. In contrast, tool-based methods are non-differentiable, limiting their adaptability in reinforcement learning settings or tasks with sparse delayed rewards. In addition, the modularity of compiled neural networks enhances performance across a variety of model sizes and can be applied regardless of the ability of the language model to call functions, unlike tools that can only operate in large, additionally trained models with code. Therefore, the efficiency, unified end-to-end neural network architecture, and wide applicability of the Neural Understanding framework constitute its obvious advantages over tool-based methods, providing a powerful and scalable solution for a variety of language and computing challenges.

[0107] Fig.12 Showing the impact of different functional combinations of compiled neural networks.

[0108] Efficiently deploying multiple compiled neural networks is critical to achieving superior neural understanding performance. Fig.12As shown, different compiled neural networks tailored for symbolic and arithmetic reasoning tasks within the language model framework and merging them can bring significant benefits. Combining multiple different compiled neural networks into the framework according to the exemplary embodiment of the present disclosure enables the language model to have multiple functions. Experiments were conducted on the Last Letter concatation class and the AddSub+ dataset, and the results showed that the plug-and-play gating mechanism can still well control what these compiled neural network outputs should output. It is observed that integrating related compiled neural networks can enhance the performance of the initial language model, while the inclusion of irrelevant language models rarely causes adverse effects, whether it is a single or multiple compiled neural network combinations. This can be attributed to the fine design of the neural understanding framework, which ensures that the compiled neural network accurately performs the assigned tasks without interference from irrelevant modules. Each compiled neural network module is good at generating appropriate outputs when needed, thereby preventing erroneous results from irrelevant components. Importantly, the parameter count of each compiled neural network module ranges from 1 / 1000 to 1 / 1000000 of GPT-3, and the reasoning delay in the neural understanding framework is only increased by 1%-3% compared to normal generation. This observation highlights the remarkable scalability of the neural understanding framework, which has the ability to adapt not only to existing knowledge concepts, but also to absorb new knowledge concepts as the number of compiled neural networks increases. In theory, integrating thousands of CoNN modules in a language model has the potential to promote comprehensive understanding of concepts.

[0109] Fig.13 A text data processing apparatus according to an exemplary embodiment of the present disclosure is shown.

[0110] Reference Fig.13 The text data processing device 1300 includes a text data acquisition module 1310 and a text data processing module 1320 .

[0111] The text data acquisition module 1310 is configured to acquire text data and input the text data into a text data processing model generated by fusing a language model with a compiled neural network, wherein the language model is pre-trained and the compiled neural network is assigned corresponding neural network weights according to a rule-intensive task to be performed if it is not pre-trained.

[0112] The text data processing module 1320 is configured to: determine a text data processing task corresponding to the text data from the text data based on a language model in a text data processing model; and execute the rule-intensive task based on a compiled neural network in the text data processing model in response to the text data processing task indicating a rule-intensive task.

[0113] Reference Figures 1 to 12One or more of the foregoing describes the acquisition operation performed by the text data acquisition module 1310 and the text data processing operation performed by the text data processing module 1320. Here, in order to avoid redundancy, the acquisition operation performed by the text data acquisition module 1310 and the text data processing operation performed by the text data processing module 1320 are not repeated.

[0114] Fig.14 A block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure.

[0115] Reference Fig.10 According to an embodiment of the present disclosure, the electronic device 1400 may include one or more computing devices (e.g., processors) 1410 and one or more storage devices 1420. Here, the one or more storage devices 1420 store a computer program, wherein the computer program implements the reference when executed by the one or more computing devices 1410. Figures 1 to 12 For the sake of brevity, the reference method performed by one or more electronic devices 1400 is not repeatedly described here. Figures 1 to 12 Any method described.

[0116] The various modules in the device of the quantized neural network model of the present disclosure shown can be configured as software, hardware, firmware or any combination of the above items to perform specific functions. For example, each module may correspond to a dedicated integrated circuit, or to a pure software code, or to a module that combines software and hardware. In addition, one or more functions implemented by each module may also be uniformly executed by components in a physical entity device (e.g., a processor, a client or a server, etc.).

[0117] In addition, the text data processing method of the neural network model of the present disclosure described above can be implemented by a program (or instruction) recorded on a computer-readable storage medium. For example, according to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions can be provided, wherein when the instructions are executed by at least one computing device, the at least one computing device is prompted to execute the text data processing method of the neural network model according to the present disclosure.

[0118] The computer program in the computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. It should be noted that the computer program can also be used to perform additional steps in addition to the above steps or perform more specific processing when performing the above steps. The contents of these additional steps and further processing have been described in reference to Figures 1 to 14 One or more of the related methods are mentioned in the description process, so they will not be repeated here to avoid repetition.

[0119] It should be noted that the various modules in the device of the quantized neural network model according to the exemplary embodiment of the present disclosure can completely rely on the operation of the computer program to realize the corresponding functions, that is, each module corresponds to each step in the functional architecture of the computer program, so that the entire system is called through a special software package (for example, lib library) to realize the corresponding function.

[0120] On the other hand, the various modules according to the various embodiments of the present disclosure may also be implemented by hardware, software, firmware, middleware, microcode or any combination thereof. When implemented by software, firmware, middleware or microcode, the program code or code segment for performing the corresponding operation may be stored in a computer-readable medium such as a storage medium, so that the processor can perform the corresponding operation by reading and running the corresponding program code or code segment.

[0121] For example, the exemplary embodiments of the present disclosure may also be implemented as a computing device, which includes a storage component and a processor, wherein a computer executable instruction set is stored in the storage component, and when the computer executable instruction set is executed by the processor, a text data processing method according to the exemplary embodiments of the present disclosure is executed.

[0122] Specifically, the computing device can be deployed in a server or client, or can be deployed on a node device in a distributed network environment. In addition, the computing device can be a PC, a tablet device, a personal digital assistant, a smart phone, a web application, or other device capable of executing the above instruction set.

[0123] Here, the computing device is not necessarily a single computing device, but may also be any device or circuit collection that can execute the above instructions (or instruction sets) individually or jointly. The computing device may also be part of an integrated control system or system manager, or may be configured as a portable electronic device that is interconnected with a local or remote (e.g., via wireless transmission) interface.

[0124] In a computing device, a processor may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, a processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0125] According to the text data processing method of the neural network model of the exemplary embodiment of the present disclosure, some operations described in the method can be implemented by software, some operations can be implemented by hardware, and in addition, these operations can also be implemented by a combination of software and hardware.

[0126] The processor may execute instructions or codes stored in one of the storage components, which may also store data. Instructions and data may also be sent and received over a network via a network interface device, which may employ any known transmission protocol.

[0127] The storage component may be integrated with the processor, for example, RAM or flash memory is arranged within an integrated circuit microprocessor, etc. In addition, the storage component may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The storage component and the processor may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor can read files stored in the storage component.

[0128] In addition, the computing device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the computing device may be connected to each other via a bus and / or a network.

[0129] The text data processing method of the neural network model according to the exemplary embodiment of the present disclosure can be described as various interconnected or coupled functional blocks or functional diagrams. However, these functional blocks or functional diagrams can be equally integrated into a single logical device or operated according to non-exact boundaries.

[0130] Therefore, refer to Figures 1 to 12 At least one of the described text data processing methods can be implemented by a system including at least one computing device and at least one storage device storing instructions.

[0131] According to an exemplary embodiment of the present disclosure, at least one computing device is a computing device for executing a text data processing method according to an exemplary embodiment of the present disclosure, and a computer executable instruction set is stored in a storage device. When the computer executable instruction set is executed by the at least one computing device, the text data processing method described with reference to the figure is executed.

[0132] The above describes various exemplary embodiments of the present disclosure, and it should be understood that the above description is only exemplary and not exhaustive, and the present disclosure is not limited to the disclosed exemplary embodiments. Without departing from the scope and spirit of the present disclosure, many modifications and changes are obvious to those of ordinary skill in the art. Therefore, the scope of protection of the present disclosure should be based on the scope of the claims.

Claims

1. A text data processing method, characterized in that: The text data processing method comprises: Acquire text data and input the text data into a text data processing model generated by fusing a language model with a compiled neural network, wherein the language model is pre-trained and the compiled neural network is assigned corresponding neural network weights according to a rule-intensive task to be performed without being pre-trained; determining, from the text data, a text data processing task corresponding to the text data based on a language model in the text data processing model; and In response to the text data processing task indicating a rule-intensive task, the rule-intensive task is executed based on the compiled neural network in the text data processing model.

2. The text data processing method according to claim 1, characterized in that: The compiled neural network includes a first compiled neural network and a second compiled neural network. The rule-intensive task includes a first rule-intensive task and a second rule-intensive task, and The steps of executing rule-intensive tasks based on the compiled neural network in the text data processing model include: In response to the text data processing task indicating a first rule-intensive task, performing the first rule-intensive task based on a first compiled neural network, and In response to the text data processing task indicating a second rule-intensive task, the second rule-intensive task is performed based on a second compiled neural network.

3. The text data processing method according to claim 1, characterized in that: The steps of executing rule-intensive tasks based on compiled neural networks in the text data processing model include: determining task data corresponding to a text data processing task to be performed from the text data based on the language model; Perform rule-intensive tasks on task data based on compiled neural networks.

4. The text data processing method according to claim 1, characterized in that: Rule-intensive tasks include one or more of symbolic manipulation tasks, symbolic reasoning tasks, and arithmetic reasoning tasks.

5. The text data processing method according to claim 1, characterized in that: The text data processing method further comprises: In response to the text data processing task indicating another task different from the rule-intensive task, the another task is performed based on the language model.

6. The text data processing method according to claim 1, characterized in that: The text data processing method further comprises: Outputting a task result corresponding to the rule-intensive task outputted by executing the rule-intensive task based on the compiled neural network to the language model; The final task result is output by processing the task results corresponding to the rule-intensive task based on the language model.

7. A text data processing device, characterized in that: The text data processing device comprises: a text data acquisition module configured to acquire text data and input the text data into a text data processing model generated by fusing a language model with a compiled neural network, wherein the language model is pre-trained, and the compiled neural network is assigned corresponding neural network weights according to a rule-intensive task to be performed when it is not pre-trained; The text data processing module is configured as follows: Determining a text data processing task corresponding to the text data from the text data based on a language model in the text data processing model; In response to the text data processing task indicating a rule-intensive task, the rule-intensive task is executed based on the compiled neural network in the text data processing model.

8. The text data processing device according to claim 7, characterized in that: The compiled neural network includes a first compiled neural network and a second compiled neural network. The rule-intensive task includes a first rule-intensive task and a second rule-intensive task, and Wherein, the text data processing module is configured as follows: In response to the text data processing task indicating a first rule-intensive task, performing the first rule-intensive task based on a first compiled neural network, and In response to the text data processing task indicating a second rule-intensive task, the second rule-intensive task is performed based on a second compiled neural network.

9. The text data processing device according to claim 7, characterized in that: The text data processing module is configured as follows: determining task data corresponding to a text data processing task to be performed from the text data based on the language model; Perform rule-intensive tasks on task data based on compiled neural networks.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by one or more computing devices, enables the one or more computing devices to implement the text data processing method according to any one of claims 1 to 6.