Data processing method and device, nonvolatile storage medium and electronic equipment

Abstract

The invention discloses a data processing method and device, a nonvolatile storage medium and electronic equipment. The method comprises the steps that input information is obtained, the input information comprises table data and user requirements, and the user requirements are used for indicating requirements met by texts generated according to the table data; the user demand is rewritten according to the table data, a target user demand is obtained, and the target user demand comprises a newly-added field generated according to the table data; performing multi-stage screening in the table fragments according to a target user demand to obtain a target table fragment; a target text is generated according to the target table fragment and the target user requirement, and the target text comprises data corresponding to the newly-added field. According to the method and the device, the technical problem that a result obtained by performing content recreation based on the table data is inaccurate due to small association degree between the input user demand and the table data when the table data is analyzed and the content is recreated in the related technology is solved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular, to a data processing method and device, a non-volatile storage medium, and an electronic device. Background Art

[0002] With the in-depth development of big data technology, the exponential growth of information volume has put forward higher requirements for data processing and knowledge extraction technology. As a tool for carrying structured information, tables standardize data formats and promote efficient communication and analysis of data. In this context, methods for parsing and recreating content of table data have been widely used. In related technologies, when parsing and recreating content of table data based on query information input by users, there is a problem of inaccurate results obtained by content recreating based on table data.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a data processing method and device, a non-volatile storage medium, and an electronic device to at least solve the technical problem in the related art that when parsing table data and recreating content, the results of content recreating based on table data are inaccurate due to the low correlation between the input user needs and the table data.

[0005] According to one aspect of an embodiment of the present application, a method for data processing is provided, including: obtaining input information, wherein the input information includes: table data and user requirements, the user requirements are used to indicate requirements met by text generated based on the table data; rewriting the user requirements according to the table data to obtain target user requirements, wherein the target user requirements include newly added fields generated based on the table data; performing multi-stage screening in multiple table fragments according to the target user requirements to obtain target table fragments, wherein the multiple table fragments are obtained by segmenting the table data, and in the multi-stage screening, the results obtained by screening in the previous stage are used as objects to be screened in the next stage; generating a target text according to the target table fragments and the target user requirements, wherein the target text includes data corresponding to the newly added fields.

[0006] Optionally, user needs are rewritten according to the table data to obtain target user needs, including: segmenting the table data to obtain multiple table fragments, wherein the number of table fragments is the same as the number of rows of the table data; determining key fields in the multiple table fragments, wherein the key fields are fields containing header data of the table data; and rewriting the user needs into target user needs according to the key fields.

[0007] Optionally, rewriting user needs into target user needs based on key fields includes: filling key fields and user needs into a rewriting template, wherein the rewriting template is used to merge the user needs with the key fields, and the rewriting template includes at least: a first to-be-filled field for filling the user needs, and a second to-be-filled field for filling the key fields; and determining the rewriting template filled with the key fields and user needs as the target user needs.

[0008] Optionally, multi-stage screening is performed on multiple table fragments according to target user needs to obtain target table fragments, including: determining a first semantic similarity between each table fragment and the target user needs; performing a first-stage screening on multiple table fragments according to the first semantic similarity to obtain a first screening result; using a similarity prediction model to determine a second semantic similarity between each table fragment in the first screening result and the user needs; performing a second-stage screening on the first screening result according to the second semantic similarity to obtain a second screening result, wherein the table fragment included in the second screening result is the target table fragment.

[0009] Optionally, determining the first semantic similarity between each table fragment and the target user demand includes: determining multiple third semantic similarities corresponding to each table fragment, wherein the multiple third semantic similarities are semantic similarities between the table fragment and the target user demand calculated using different semantic similarity algorithms; determining a weight value corresponding to each third semantic similarity, wherein the weight value is used to indicate the importance of a semantic similarity algorithm for generating the third semantic similarity among multiple semantic similarity algorithms, and the importance is positively correlated with the weight value; for each third semantic similarity, determining the product of each third semantic similarity and the weight value corresponding to the third semantic similarity to obtain multiple products; and adding the multiple products to obtain the first semantic similarity.

[0010] Optionally, a first-stage screening is performed on multiple table fragments according to a first semantic similarity to obtain a first screening result, including: sorting all table fragments that constitute the table data in order from large to small according to the first semantic similarity to obtain a first sorting result; determining the first M table fragments in the first sorting result as the first screening result, where M is an integer; performing a second-stage screening on the first screening result according to the second semantic similarity to obtain a second screening result, including: sorting multiple table fragments in the first screening result in order from large to small according to the second semantic similarity to obtain a second sorting result; determining the first K table fragments in the second sorting result as the second screening result, where K is an integer and K is less than M.

[0011] Optionally, generating a target text according to a target table fragment and target user needs includes: filling the target table fragment and target user needs into a fusion template, wherein the fusion template is used to merge the target user needs with the target table fragment, and the fusion template includes at least: a third field to be filled in for filling in the target user needs, and a fourth field to be filled in for filling in the target table fragment.

[0012] According to another aspect of an embodiment of the present application, a data processing device is also provided, including: an acquisition module, used to acquire input information, wherein the input information includes: table data and user requirements, and the user requirements are used to indicate the requirements met by the text generated based on the table data; a rewriting module, used to rewrite the user requirements according to the table data to obtain target user requirements, wherein the target user requirements include newly added fields generated based on the table data; a screening module, used to perform multi-stage screening in multiple table fragments according to the target user requirements to obtain target table fragments, wherein the multiple table fragments are obtained by segmenting the table data, and in the multi-stage screening, the results obtained by screening in the previous stage are used as objects to be screened in the next stage; a generation module, used to generate a target text according to the target table fragments and the target user requirements, wherein the target text includes data corresponding to the newly added fields.

[0013] According to another aspect of an embodiment of the present application, a non-volatile storage medium is provided, in which a computer program is stored, wherein the above-mentioned data processing method is executed by running the computer program on a device where the non-volatile storage medium is located.

[0014] According to another aspect of an embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above-mentioned data processing method through the computer program.

[0015] According to another aspect of an embodiment of the present application, a computer program product is also provided, including computer instructions, which implement the steps of the above-mentioned data processing method when executed by a processor.

[0016] In an embodiment of the present application, input information is obtained, wherein the input information includes: table data and user requirements, wherein the user requirements are used to indicate requirements satisfied by text generated based on the table data; the user requirements are rewritten according to the table data to obtain target user requirements, wherein the target user requirements include newly added fields generated based on the table data; multi-stage screening is performed in multiple table fragments according to the target user requirements to obtain target table fragments, wherein the multiple table fragments are obtained by segmenting the table data, and in the multi-stage screening, the results obtained by screening in the previous stage are used as objects to be screened in the next stage; a target text is generated according to the target table fragments and the target user requirements, wherein the target text includes data corresponding to the newly added fields, by The table is segmented, and each table segment obtained by segmentation is analyzed separately, thereby reducing the amount of data processing when analyzing the table data at a single time; the table segments are screened through multiple stages, and the screened table segments are merged with the demand information input by the user, so as to achieve the purpose of rewriting the demand input according to the table data architecture into demand information with higher consistency with the table data, and the table data is analyzed and the content is regenerated based on the rewritten demand information, thereby achieving the technical effect of improving the accuracy of the results obtained by content regeneration based on the table data, and further solving the technical problem of inaccurate results obtained by content regeneration based on the table data due to the low correlation between the input user demand and the table data when parsing the table data and recreating the content in the related technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 is a hardware structure block diagram of a computer terminal for implementing a data processing method according to an embodiment of the present application;

[0019] Figure 2 is a flowchart of a method for data processing according to an embodiment of the present application;

[0020] Figure 3 is a structural diagram of a data processing device according to an embodiment of the present application;

[0021] Figure 4 It is a workflow diagram of a data processing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] In the related art, all the input table data is analyzed. Therefore, when the input table data is a "large wide table" containing multiple rows / columns of data (for example, a table with more than 1000 rows or more than 1000 columns), the results regenerated based on the table data lack key information in the table data due to the input capacity limitation of the model used to analyze the table data and the lack of understanding of the complex structure, resulting in inaccurate results based on the table data. In the related art, the Retrieval-Augmented Generation (RAG) mechanism is introduced.

[0025] An attempt is made to improve the processing capability and response speed of table data by enhancing retrieval. Although the RAG mechanism alleviates input restrictions to a certain extent, it is still difficult to ensure high recall and accuracy of content generation in the face of highly diverse and complex table attributes. At the same time, it is difficult to associate the user input demand information with the table data. Therefore, there is still the problem of missing information in the results of content regeneration based on table data, which reduces practicality and user experience. In order to solve this problem, a relevant solution is provided in the embodiment of the present application, which is described in detail below.

[0026] According to an embodiment of the present application, a method embodiment of a data processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0027] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a data processing method. Figure 1 As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0028] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0029] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data processing method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned data processing method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0030] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0031] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0032] The present application embodiment provides a data processing method that can be run in the above-mentioned operating environment. Figure 2 is a flowchart of the steps of the data processing method provided in the embodiment of the present application, such as Figure 2 As shown, the method comprises the following steps:

[0033] Step S202, obtaining input information, wherein the input information includes: table data and user requirements, and the user requirements are used to indicate requirements satisfied by the text generated based on the table data.

[0034] The method provided in the embodiment of the present application integrates the methods of retrieval enhancement generation (RAG) and re-ranking into the processing method of text generation based on table data, and rewrites the text generation requirements (i.e., user requirements) input by the user using table data, so that the rewritten text generation requirements (i.e., user requirements) include a part of the input table data (at least including the key data in the table data), so that when the system / device regenerates the table data according to the user requirements, the analysis of the user requirements is more accurate, and the result of content regeneration is also more accurate. As mentioned above, since the user requirements are rewritten using the table data to be processed, the user requirements and the table data to be processed are received at the same time in step S202, wherein the user requirements are the description information of the result of the table data content regeneration, which can be used to indicate the requirements met by the content regeneration result (i.e., the text generated based on the table data), and the above-mentioned "requirements" can be the format of the content regeneration result, the type of data included in the content regeneration result, etc. For example, the user requirement can be: "Please generate an analysis report based on the provided data." The input table data includes multiple forms, for example, file form and picture form. When the form of the table data is file form, the input table data is a file containing a table; when the form of the table data is picture form, the input table data is a picture containing a table.

[0035] Step S204 , rewriting the user requirements according to the table data to obtain target user requirements, wherein the target user requirements include newly added fields generated according to the table data.

[0036] In step S204, the user needs received at the same time as the table data are preliminarily rewritten according to the received table data; in this embodiment of the present application, in order to improve the correlation between the rewritten user needs and the table data, when the user needs are rewritten, the key data in the basic data (i.e., the table data) used for content regeneration is added to the user needs. Therefore, the rewritten user needs add new information relative to the user needs obtained in step S202; in step S204, the preliminarily rewritten user needs (i.e., the target user needs) add fields not included in the initial user needs (i.e., the user needs received in step S202) relative to the initial user needs (i.e., the user needs received in step S202), and these newly added fields are fields generated by analyzing the table data.

[0037] Optionally, user needs are rewritten according to the table data to obtain target user needs, including: segmenting the table data to obtain multiple table fragments, wherein the number of table fragments is the same as the number of rows of the table data; determining key fields in the multiple table fragments, wherein the key fields are fields containing header data of the table data; and rewriting the user needs into target user needs according to the key fields.

[0038] Since the user requirements are rewritten by integrating the table data into the user requirements, before rewriting the user requirements, it is necessary to retrieve the table data that can be used to integrate into the user requirements. In this embodiment, in order to reduce the amount of data processed in a single search, before rewriting the user requirements, the complete table data obtained in step S202 is first segmented into multiple table segments, so that the amount of data processed when performing table data analysis and user requirements rewriting is less. Specifically, the number of rows of the table storing the table data is used as the segmentation basis, and each row of the table used to store the table data is segmented into a table segment; for example, if the table data obtained in step S202 is stored in a table containing a total of N rows, then the segmentation will obtain N table segments. Furthermore, part of the table data used for preliminary rewriting of user needs is screened from all the table fragments obtained by segmenting the table data. In this embodiment, the header data contained in the table data is used as the data for preliminary rewriting of user needs, and the field storing the header data is determined as the key field to be used in the preliminary rewriting process, and the key field is stored in at least one table fragment; in this embodiment, the above-mentioned key field can also be stored in a knowledge base. In this way, the close association between user needs and table data can be ensured, and the pertinence and accuracy of text generation can be improved.

[0039] It should be noted that the method provided in the embodiment of the present application does not stipulate that the key field contains data. It only needs to clarify which type of data should be filled in the key field, that is, the key field can be used to indicate the type of data.

[0040] According to an optional embodiment of the present application, user needs are rewritten into target user needs based on key fields, including: filling key fields and user needs into a rewriting template, wherein the rewriting template is used to merge the user needs with the key fields, and the rewriting template at least includes: a first to-be-filled field for filling the user needs, and a second to-be-filled field for filling the key fields; and determining the rewriting template filled with the key fields and user needs as the target user needs.

[0041] After the table data (i.e., key fields) for preliminary rewriting are obtained by screening according to the method provided in the previous embodiment, in this embodiment, the user requirements are preliminarily rewritten in combination with the rewriting instructions, and the rewriting instructions are used to provide a rewriting template, which is used to merge the key fields with the user requirements into a new user requirement (i.e., the target user requirement); since the key fields need to be merged with the user requirements, it is obvious that the rewriting template should at least include a field for writing the user requirements (i.e., the first field to be filled) and a field for writing the key fields (i.e., the second field to be filled); in order to ensure that the user requirements and the key fields have correct semantics after being merged, the rewriting template can also include some preset texts, which are used to concatenate the user requirements and the key fields. When performing the preliminary rewriting, the user requirements and the key fields are filled into the corresponding fields in the rewriting template, that is, the merging of the user requirements and the key fields is completed, and the rewriting template that is finally filled is the user requirements after the preliminary rewriting (i.e., the target user requirements). For example, the user demand obtained in step S202 is: "Please generate an analysis report on broadband users in Province A in year B based on the data provided", then the rewrite instruction can be "Please rewrite the user's document requirements (i.e., user needs) in combination with specific data indicators so that the indicators in the requirements meet the data", and the rewrite template is: "[C field], wherein the data type that must be included is [D field]", wherein the C field in the rewrite template (i.e., the first field to be filled in) is used to fill in the user demand, and the D field (i.e., the second field to be filled in) is used to fill in the extracted key field, and "wherein the data type that must be included is" is the preset text in the rewrite template; if the extracted key fields are [broadband access], [broadband service], and [broadband service ranking], then the user demand after preliminary rewriting (i.e., the target user demand) is: "Please generate an analysis report on broadband users in Province A in year B based on the data provided, wherein the data types that must be included are [broadband access], [broadband service], and [broadband service ranking]". By rewriting the template to process user needs in a templated manner, user needs can be effectively standardized, avoiding the problem of low accuracy of the results of content regeneration of table data according to user needs due to low correlation between user needs and table data.

[0042] It should also be noted that the user needs after the initial rewrite (i.e., the target user needs) only have new key fields added compared to the initial user needs, but these key fields are not necessarily filled with data (they may be filled or not, depending on the pre-set settings); the data to be filled in the key fields needs to be filtered from the table data.

[0043] Step S206, performing multi-stage screening in multiple table fragments according to the target user's needs to obtain a target table fragment, wherein the multiple table fragments are obtained by segmenting the table data. In the multi-stage screening, the results obtained by the previous stage of screening are used as the objects to be screened in the next stage.

[0044] In step S206, in the process of regenerating content based on table data, the table data obtained in step S202 is divided into multiple table segments, and the length of the table segment is shorter than the length of the complete table data received in step S202, that is, the data volume of the table segment is less than the data volume of the complete table data received in step S202. Therefore, dividing the table data into table segments can reduce the amount of data to be processed when parsing the table data at a single time, improve the speed of data processing, and improve the speed of content regeneration based on table data. Further, according to the user needs after preliminary rewriting (i.e., the target user needs), the multiple table segments obtained by dividing the table data are screened in multiple stages, wherein in the process of multi-stage screening, each stage uses the results obtained by the previous stage screening as the screening object of this stage, for example, in the first stage, all the table segments obtained by dividing the table data are used as the objects to be screened, and all the table segments are screened, and the screening results obtained are the partial table segments in all the table segments that meet the screening requirements of the first stage; then in the second stage, the partial table segments that meet the screening requirements of the first stage are used as the objects to be screened. The results obtained by multi-stage screening according to the above method are used as table fragments (ie, target table fragments) for integration into the preliminary rewritten user requirements (ie, target user requirements) to complete the complete rewriting process of user requirements.

[0045] Optionally, multi-stage screening is performed on multiple table fragments according to target user needs to obtain target table fragments, including: determining a first semantic similarity between each table fragment and the target user needs; performing a first-stage screening on multiple table fragments according to the first semantic similarity to obtain a first screening result; using a similarity prediction model to determine a second semantic similarity between each table fragment in the first screening result and the user needs; performing a second-stage screening on the first screening result according to the second semantic similarity to obtain a second screening result, wherein the table fragment included in the second screening result is the target table fragment.

[0046] As mentioned in the above embodiment, in order to improve the relevance between the rewritten final user demand (i.e., the target text) and the table data, the method provided in the embodiment of the present application integrates part of the data in the table data into the user demand. Specifically, part of the table data is filled into the key fields included in the target user demand. The data used to fill in the key fields is obtained by screening from all the table data obtained in step S202 using the method of this embodiment; specifically, the method provided in the embodiment of the present application uses a multi-stage screening method to screen all the table data obtained in step S202. The multi-stage screening is based on semantic similarity. In the first stage, all the table data obtained in step S202 are preliminarily screened (i.e., the first stage screening), and in the second stage, the screening results of the first stage are finely screened (i.e., the second stage screening). The multiple table fragments obtained after the second stage screening are the table fragments (i.e., the target table fragments) used for the final rewriting of the user demand. Through multi-stage screening, the search scope can be gradually narrowed, the accuracy of screening can be improved, and it can be ensured that the text generation results (i.e., the results of content regeneration of table data according to user needs) are highly relevant to user needs; in this embodiment, a semantic similarity algorithm is used to calculate the similarity between the table fragment and the target user needs during the first stage screening, and a trained model (such as encoder BERT) is used to calculate the similarity between the table fragment and the target user needs during the second stage screening. In this embodiment, the calculation result of the semantic similarity of the table fragment in the first stage screening is called the first semantic similarity, and the calculation result of the semantic similarity of the table fragment in the second stage screening is called the second semantic similarity. The second semantic similarity is output by the similarity prediction model after analyzing the target user needs and the first stage screening results. The similarity prediction model is obtained by training a neural network model (such as encoder BERT) using training data.

[0047] According to an optional embodiment of the present application, determining a first semantic similarity between each table fragment and a target user demand includes: determining a plurality of third semantic similarities corresponding to each table fragment, wherein the plurality of third semantic similarities are semantic similarities between the table fragment and the target user demand calculated using different semantic similarity algorithms; determining a weight value corresponding to each third semantic similarity, wherein the weight value is used to indicate the importance of a semantic similarity algorithm for generating the third semantic similarity among a plurality of semantic similarity algorithms, and the importance is positively correlated with the weight value; for each third semantic similarity, determining the product of each third semantic similarity and the weight value corresponding to the third semantic similarity to obtain a plurality of products; and adding the plurality of products to obtain the first semantic similarity.

[0048] In this embodiment, during the first stage of screening, the (first) semantic similarity between each table segment and the user requirement after preliminary rewriting (i.e., the target user requirement) is first determined. In order to avoid the problem of deviation in the similarity calculation result due to different emphases of the similarity calculation method, in this embodiment, multiple semantic similarity algorithms are used to determine the semantic similarity between the table segment and the target user requirement. The (first) semantic similarity of the table segment in the first stage is jointly determined by the results of multiple different semantic similarity algorithms (i.e., the third semantic similarity); for example, a cosine similarity algorithm can be used to convert the table segment and the target user requirement into a vector form, and then the cosine similarity between the vector converted from the table segment and the vector converted from the target user requirement is used as the (third) semantic similarity between the table segment and the target user requirement; an information retrieval algorithm (BM25) is used to calculate the (third) semantic similarity between the table segment and the target user requirement based on the frequency of occurrence of the data recorded in the table segment in the target user requirement, the length of the table segment, etc.; a semantic similarity algorithm based on term frequency (Term Frequency–Inverse Document 2) is used. Frequency (TF-IDF), based on the frequency of the data recorded in the table fragment appearing in the target user's needs and the frequency of appearing in the entire table data, the (third) semantic similarity between the table fragment and the target user's needs is determined. In this embodiment, the formula To determine the (first) semantic similarity of each table fragment for the first stage screening, where represents the (first) semantic similarity of the jth table segment, l is the number of third semantic similarities corresponding to each table segment, and s ji is the third semantic similarity of the i-th table, w i is the weight value of the third semantic similarity of the ith i The (third) semantic similarity calculated by combining different algorithms can more comprehensively evaluate the relevance between the table fragment and the user's needs, thereby improving the accuracy of the screening results.

[0049] According to another optional embodiment of the present application, a first-stage screening is performed on multiple table fragments according to a first semantic similarity to obtain a first screening result, including: sorting all table fragments constituting the table data in order from large to small according to the first semantic similarity to obtain a first sorting result; determining the first M table fragments in the first sorting result as the first screening result, where M is an integer; performing a second-stage screening on the first screening result according to a second semantic similarity to obtain a second screening result, including: sorting multiple table fragments in the first screening result in order from large to small according to the second semantic similarity to obtain a second sorting result; determining the first K table fragments in the second sorting result as the second screening result, where K is an integer and K is less than M.

[0050] In this embodiment, when the first stage screening is performed based on the (first) semantic similarity, multiple table segments with high relevance to the target user's needs are screened out, that is, multiple table segments with high first semantic similarity are screened out. In this embodiment, multiple table segments with high first semantic similarity are determined by sorting the first semantic similarity from large to small according to numerical values. Specifically, the first M table segments in the (first) sorting result obtained by sorting the first semantic similarity from large to small according to numerical values ​​are used as the result of the first stage screening (i.e., the first screening result), where M is an integer less than the total number of tables. Next, based on the results of the first stage screening (i.e., the first screening results), fine screening (i.e., the second stage screening) is to screen out some table data with the highest correlation with the target user's needs. Therefore, when performing fine screening, the screening is still based on semantic similarity. Specifically, the first M table fragments in the (second) sorting result obtained by arranging the second semantic similarity from large to small according to the numerical value are used as the results of the second stage screening (i.e., the second screening results). The table fragments obtained by the second stage screening are also the table fragments used for the final rewriting of user needs (i.e., the target table fragments). Since the second stage screening uses the results of the first stage screening as the objects to be screened, it is obvious that the number of table fragments obtained by the second stage screening (i.e., K) is less than the number of table fragments obtained by the first stage screening (i.e., M). By the method of preliminary screening and then fine screening, high-quality table fragments can be obtained. This high-quality table fragment can improve the accuracy of text generation based on the table.

[0051] Step S208 , generating a target text according to the target table fragment and the target user's needs, wherein the target text includes data corresponding to the newly added field.

[0052] After the table fragment (i.e., target table fragment) used for integrating the preliminary rewritten user requirements (i.e., target user requirements) is screened out in step S206, in step S208, the final screening result (i.e., target table fragment) in step S206 and the user requirements after preliminary rewriting in step S204 (i.e., target user requirements) are integrated again to obtain the final user requirement rewriting result (i.e., target text); as mentioned above, since the initial user requirements are rewritten by integrating the table data into the initial user requirements, the final rewritten result (i.e., target text) has new data relative to the initial user requirements; the newly added data in the final rewritten result (i.e., target text) is extracted from the table data and is the key data in the table data.

[0053] Optionally, generating a target text according to a target table fragment and target user needs includes: filling the target table fragment and target user needs into a fusion template, wherein the fusion template is used to merge the target user needs with the target table fragment, and the fusion template includes at least: a third field to be filled in for filling in the target user needs, and a fourth field to be filled in for filling in the target table fragment.

[0054] In this embodiment, when the user demand is finally rewritten using the table fragment (i.e., the target table fragment) screened in step S206, the target table fragment is merged with the user demand (i.e., the target user demand) obtained by the preliminary rewriting. Specifically, in this embodiment of the application, a fusion template is used to merge the target user demand and the target table fragment. The fusion template defines the specific positions of the target demand and the target table fragment in the target text generated after the fusion. The fusion template should at least include a field for writing the target user demand (i.e., the third field to be filled) and a field for writing the target table fragment (i.e., the fourth field to be filled). The fusion template may also include preset text for concatenating the target user demand and the target table fragment so that the generated target text has accurate semantics. Next, an example is given in combination with the above embodiment. For example, in the above example, the key fields selected are: [broadband access], [broadband service], [broadband service ranking], and the data stored in the selected target table fragments are respectively a type of data related to broadband access, a type of data related to broadband service, and a type of data related to broadband service ranking, and these data are stored in {table fragment 1}, {table fragment 2}, {table fragment 3}, ... {table fragment 10}, and the target table fragments include: {table fragment 1}, {table fragment 2}, {table fragment 3}, ... {table fragment 10}. The fusion template is "[E field], where the data used is [F field]", and the E field (i.e., the third field to be filled) in the above fusion template is used to fill the target user needs (or part of the target user needs), and the F field (i.e., the fourth field to be filled) is used to fill the target table fragment; "wherein, the data used is" is the preset text in the fusion template. For example, the target requirement is: "Please generate an analysis report on broadband users in Province A in year B based on the data provided, where the data types that must be included are [broadband access], [broadband service], [broadband service ranking]", and the target table fragments screened out for storing the data indicated by the key fields include: {table fragment 1}, {table fragment 2}, {table fragment 3}, ... {table fragment 10}. Then the target text after fusion by the fusion template is: "Please generate an analysis report on broadband users in Province A in year B based on the data provided, where the data types that must be included are [broadband access], [broadband service], [broadband service ranking]. I'm afraid to buy one when I go to Europe. The data used are {table fragment 1}, {table fragment 2}, {table fragment 3}, ... {table fragment 10}".

[0055] The final rewriting result of the user's needs (i.e., the target text) obtained by the method provided in the embodiment of the present application can be input into the model for content regeneration simultaneously with the table data. The model collects the final rewriting result of the user's needs (i.e., the target text) and performs content regeneration processing on the table data to obtain a more accurate result that better meets the user's needs. The method provided in the embodiment of the present application can also adapt to different data formats and user needs, has a high degree of flexibility and versatility, and can meet the data processing needs of different industries and fields. For example, in medical data analysis, it helps doctors quickly generate analysis reports for specific cases; in the field of education, it helps teachers generate data analysis reports based on student performance.

[0056] The table data screening and user demand rewriting (preliminary rewriting and final rewriting) in the above embodiment can be performed by a trained neural network model, which can be obtained by training a natural language processing model (LLaMA or TeleChat) and an encoder (Bidirectional Encoder Representation from Transformers, BERT) based on deep learning using text data in the form of question-answer pairs (including pre-rewriting data, post-rewriting data, and rewriting instructions). During the training process, the model to be trained (LLaMA or TeleChat) is used to perform similarity calculation and text rewriting on the pre-rewriting data in the training data. When training the model to be trained, the model to be trained can be loaded into the memory, for example, the original data of the model to be trained can be loaded from the non-volatile memory into the volatile memory, so that the processor runs the model to be trained. The raw data of the model to be trained refers to unprocessed data (such as LLaMA or TeleChat before training), which usually includes parameters and structural data of the neural network model. The structural data can be a calculation relationship based on the parameters, such as the forward propagation calculation relationship between intermediate layers and neurons. Specifically, the structural data can include code related to the structure of the neural network model, such as code for performing related calculations between intermediate layers and neurons.

[0057] In one embodiment, an area for loading a neural network model can be divided in the memory, which may include a structure data storage area and a parameter storage area. The structure data storage area is used to store structure-related codes, and the parameters referenced by it can point to the address of a specific parameter in the parameter storage area through a pointer. During the training process of the neural network model, it may be necessary to frequently update the parameters, and the parameter values ​​in the parameter storage area can be updated.

[0058] Through the above steps, the table data and user needs input by the user can be processed step by step and intelligently screened, and the table segments in the table data that are highly relevant to the user needs can be screened out. Intelligent user requirement rewriting technology is adopted, combined with the table data, to optimize the semantics of the user needs and improve their relevance to the table data. The matching degree between user needs and table data is significantly improved, which provides a solid foundation for subsequent table data retrieval and content regeneration, overcomes the problem of slow data understanding and analysis when content is regenerated based on table data, especially based on "large wide tables" with huge data volumes, and improves the conversion efficiency and accuracy of table data when content is regenerated according to user needs.

[0059] Figure 3 is a structural diagram of a data processing device provided according to an embodiment of the present application, such as Figure 3 As shown, the data processing device includes: an acquisition module 30, which is used to acquire input information, wherein the input information includes: table data and user requirements, the user requirements are used to indicate the requirements met by the text generated based on the table data, and the multiple table fragments obtained by segmenting the table data can be used to construct a table knowledge base, and the table knowledge base is used to rewrite the user requirements; a rewriting module 32, which is used to rewrite the user requirements according to the table data to obtain target user requirements, wherein the target user requirements include newly added fields generated based on the table data; a screening module 34, which is used to perform multi-stage screening in multiple table fragments according to the target user requirements to obtain target table fragments, wherein the multiple table fragments are obtained by segmenting the table data, and in the multi-stage screening, the results obtained by the screening in the previous stage are used as the objects to be screened in the next stage; a generation module 36, which is used to generate a target text according to the target table fragments and the target user requirements, wherein the target text includes data corresponding to the newly added fields.

[0060] Figure 4 It is a workflow diagram of the data processing device, such as Figure 4As shown, the data processing device starts to work, and obtains the table data used for content regeneration processing and the user's requirements for the content regeneration result (i.e., the text generated according to the table data) (i.e., user needs) through the acquisition module 30. The acquisition module 30 inputs the user needs and the table data into the rewriting module 32 for preliminary rewriting of the user needs, and obtains the user needs after preliminary rewriting (i.e., the target user needs). Compared with the user needs before rewriting, the user needs after preliminary rewriting have a new field (i.e., the newly added field) for indicating the data type to be added from the table data; next, the rewriting module 32 inputs the user needs after preliminary rewriting (i.e., the target user needs) into the screening module 32. In module 34, multi-stage screening is performed on all table fragments obtained by segmenting the table data based on the semantic similarity between the target user needs and the table fragments, so as to obtain multiple table fragments (i.e., target table fragments) with a high correlation with the target user needs; finally, the target user needs are merged with the results of the multi-stage screening (i.e., the target table fragments) by the generator 36 to obtain the final result of the user demand rewriting (i.e., the target text), and part of the data in the table data is newly added to the final result of the user demand rewriting (i.e., the target text), and the type of the newly added part of the data is the data type indicated by the field (i.e., the newly added field) newly added in the user demand by the rewriting module 32.

[0061] It should be noted that Figure 3 The preferred implementation of the illustrated embodiment can be found in Figure 2 The relevant description of the illustrated embodiment will not be repeated here.

[0062] An embodiment of the present application further provides a non-volatile storage medium, in which a computer program is stored, wherein the above data processing method is executed by running the computer program on a device where the non-volatile storage medium is located.

[0063] The above-mentioned non-volatile storage medium is used to store a program that performs the following functions: obtaining input information, wherein the input information includes: table data and user requirements, and the user requirements are used to indicate the requirements met by the text generated based on the table data; rewriting the user requirements according to the table data to obtain target user requirements, wherein the target user requirements include newly added fields generated based on the table data; performing multi-stage screening in multiple table fragments according to the target user requirements to obtain target table fragments, wherein the multiple table fragments are obtained by segmenting the table data, and in the multi-stage screening, the results obtained by screening in the previous stage are used as the objects to be screened in the next stage; generating a target text according to the target table fragments and the target user requirements, wherein the target text includes data corresponding to the newly added fields.

[0064] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above data processing method through the computer program.

[0065] The processor in the above-mentioned electronic device is used to run a program that performs the following functions: obtain input information, wherein the input information includes: table data and user requirements, and the user requirements are used to indicate the requirements met by the text generated based on the table data; rewrite the user requirements according to the table data to obtain target user requirements, wherein the target user requirements include newly added fields generated based on the table data; perform multi-stage screening in multiple table fragments according to the target user requirements to obtain target table fragments, wherein the multiple table fragments are obtained by segmenting the table data, and in the multi-stage screening, the results obtained by screening in the previous stage are used as the objects to be screened in the next stage; generate a target text according to the target table fragments and the target user requirements, wherein the target text includes data corresponding to the newly added fields.

[0066] An embodiment of the present application also provides a computer program product, including computer instructions, which implement the steps of the above data processing method when executed by a processor.

[0067] It should be noted that the various modules in the above-mentioned data processing device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0068] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0069] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0070] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0071] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0072] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0073] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0074] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for data processing, characterized in that: include: Acquire input information, wherein the input information includes: table data and user requirements, wherein the user requirements are used to indicate requirements satisfied by the text generated according to the table data; Rewriting the user requirements according to the table data to obtain target user requirements, wherein the target user requirements include a newly added field generated according to the table data; Performing multi-stage screening in a plurality of table segments according to the target user's needs to obtain a target table segment, wherein the plurality of table segments are obtained by segmenting the table data, and in the multi-stage screening, the result obtained by the screening in the previous stage is used as the object to be screened in the next stage; A target text is generated according to the target table fragment and the target user requirement, wherein the target text includes data corresponding to the newly added field.

2. The method according to claim 1, characterized in that Rewriting the user requirements according to the table data to obtain target user requirements includes: Segmenting the table data to obtain a plurality of table segments, wherein the number of the table segments is the same as the number of rows of the table data; Determine a key field in a plurality of the table fragments, wherein the key field is a field containing header data of the table data; The user requirement is rewritten into the target user requirement according to the key field.

3. The method according to claim 2, characterized in that Rewriting the user requirement into the target user requirement according to the key field includes: Fill the key fields and the user requirements into a rewriting template, wherein the rewriting template is used to merge the user requirements with the key fields, and the rewriting template at least includes: a first to-be-filled field for filling the user requirements, and a second to-be-filled field for filling the key fields; determine the rewriting template filled with the key fields and the user requirements as the target user requirements.

4. The method according to claim 1, characterized in that: According to the target user's needs, a multi-stage screening is performed in multiple table segments to obtain a target table segment, including: Determining a first semantic similarity between each of the table segments and the target user requirement; Performing a first-stage screening on the plurality of table segments according to the first semantic similarity to obtain a first screening result; Determine a second semantic similarity between each of the table segments in the first screening result and the user requirement by using a similarity prediction model; The first screening result is subjected to a second stage of screening according to the second semantic similarity to obtain a second screening result, wherein the table segment included in the second screening result is the target table segment.

5. The method according to claim 4, characterized in that Determining a first semantic similarity between each of the table segments and the target user requirement includes: Determine a plurality of third semantic similarities corresponding to each of the table segments, wherein the plurality of third semantic similarities are semantic similarities between the table segment and the target user requirement calculated using different semantic similarity algorithms; Determine a weight value corresponding to each of the third semantic similarities, wherein the weight value is used to indicate the importance of a semantic similarity algorithm for generating the third semantic similarity among the plurality of semantic similarity algorithms, and the importance is positively correlated with the weight value; For each of the third semantic similarities, determine the product of each of the third semantic similarities and the weight value corresponding to the third semantic similarity to obtain a plurality of the products; Adding a plurality of the products together, to obtain the first semantic similarity.

6. The method according to claim 4, characterized in that Performing a first-stage screening on the plurality of table segments according to the first semantic similarity to obtain a first screening result includes: Sorting all the table segments constituting the table data in descending order according to the first semantic similarity to obtain a first sorting result; determining the first M table segments in the first sorting result as the first screening result, where M is an integer; Performing a second stage of screening on the first screening result according to the second semantic similarity to obtain a second screening result, including: The plurality of table fragments in the first screening result are sorted in descending order according to the second semantic similarity to obtain a second sorting result; and the first K table fragments in the second sorting result are determined as the second screening result, where K is an integer and is less than M.

7. The method according to claim 1, characterized in that Generating a target text according to the target table fragment and the target user requirement includes: Fill the target table fragment and the target user demand into a fusion template, wherein the fusion template is used to merge the target user demand with the target table fragment, and the fusion template at least includes: a third field to be filled in for filling the target user demand, and a fourth field to be filled in for filling the target table fragment.

8. A data processing device, characterized in that: include: An acquisition module, used for acquiring input information, wherein the input information includes: table data and user requirements, wherein the user requirements are used for indicating requirements satisfied by the text generated according to the table data; A rewriting module, used for rewriting the user requirements according to the table data to obtain target user requirements, wherein the target user requirements include a newly added field generated according to the table data; A screening module, configured to perform multi-stage screening in a plurality of table segments according to the target user's needs to obtain a target table segment, wherein the plurality of table segments are obtained by segmenting the table data, and in the multi-stage screening, the result obtained by the screening in the previous stage is used as the object to be screened in the next stage; A generating module is used to generate a target text according to the target table fragment and the target user's needs, wherein the target text includes data corresponding to the newly added field.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, wherein the data processing method according to any one of claims 1 to 7 is executed by running the computer program on the device where the non-volatile storage medium is located.

10. An electronic device comprising a memory and a processor, characterized in that: The memory stores a computer program, and the processor is configured to execute the data processing method according to any one of claims 1 to 7 through the computer program.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the data processing method described in any one of claims 1 to 7 are implemented.

Images