Data processing method and device, computer equipment and storage medium

Abstract

The invention relates to a data processing method and device, computer equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: acquiring long text data and task instruction data; segmenting the long text data to obtain a plurality of text segments; performing semantic compression on the text segment to obtain a plurality of text abstracts; matching the plurality of text abstracts based on the task instruction data to obtain target text data; and inputting the task instruction data and the target text data into the large language model to obtain a final output result. According to the method, the long text data is segmented and semantically compressed, so that the information abstract is extracted, and in the using process, the information abstract can be extracted from the vector database according to needs, and a large language model can be called and guided on the premise that an original model structure or parameters do not need to be modified. Understanding, screening and backtracking behaviors of human being during large-scale text data processing are simulated, so that the response efficiency and accuracy of a large language model in a long text processing task are improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the development of Artificial Intelligence (AI) technology, Large Language Models (LLMs) have emerged, used for natural language processing tasks, possessing context understanding and generation capabilities. However, existing LLMs based on the Transformer architecture have a limited maximum text length that can be processed at one time due to the limitation of the context window. Traditional techniques mainly expand the readable range of large language models by increasing the context window during LLM training or optimizing the model structure (such as sparse attention), but these methods require complex fine-tuning or changes to the model structure, making them rather cumbersome. Summary of the Invention

[0003] Therefore, it is necessary to provide a data processing method, apparatus, computer equipment, and storage medium that does not require complex fine-tuning or changes to the model structure of a large language model to address the aforementioned technical problems.

[0004] Firstly, this application provides a data processing method. The method includes: acquiring long text data and task instruction data; segmenting the long text data to obtain multiple text segments; semantically compressing the text segments to obtain multiple text summaries; wherein the multiple text summaries are stored in a vector database; matching the multiple text summaries based on the task instruction data to obtain target text data; and inputting the task instruction data and the target text data into a large language model to obtain a final output result.

[0005] In one embodiment, the method further includes: updating the vector database based on the feedback result data in response to the feedback result data.

[0006] In one embodiment, the step of segmenting the long text data to obtain multiple text segments includes: performing preliminary segmentation of the long text data based on paragraph identifiers to obtain multiple basic segments; inputting the basic segments into the large language model for content topic clustering and context boundary recognition to obtain multiple semantic segments; and obtaining multiple text segments based on the multiple semantic segments.

[0007] In one embodiment, obtaining multiple text segments based on multiple semantic segments includes: determining the segmentation granularity based on the task type of the task instruction data; and performing segmentation and combination processing on the multiple semantic segments based on the segmentation granularity to obtain multiple text segments.

[0008] In one embodiment, semantic compression of the text segment is performed to obtain multiple text summaries, including: inputting the text segment and the task instruction data into the large language model for summary extraction to obtain multiple summary data; and performing redundancy filtering on the multiple summary data to obtain the text summary.

[0009] In one embodiment, redundancy filtering is performed on multiple summary data to obtain the text summary, including: inputting the summary data into the large language model for redundancy filtering to obtain the text summary.

[0010] In one embodiment, matching multiple text summaries based on the task instruction data to obtain target text data includes: performing vector transformation on the task instruction data to obtain a task instruction vector; and performing similarity matching on the summary vectors of the text summaries based on the task instruction vector to obtain the target text data.

[0011] Secondly, this application also provides a data processing apparatus. The apparatus includes: a data acquisition module for acquiring long text data and task instruction data; a segmentation processing module for segmenting the long text data to obtain multiple text segments; a semantic compression module for semantically compressing the text segments to obtain multiple text summaries; wherein the multiple text summaries are stored in a vector database; a summary matching module for matching the multiple text summaries based on the task instruction data to obtain target text data; and a result calculation module for inputting the task instruction data and the target text data into a large language model to obtain a final output result.

[0012] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the above-described method.

[0013] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0014] The aforementioned data processing methods, apparatus, computer equipment, and storage media, when processing long text data, first segment the long text data to obtain multiple shorter text segments, and then perform semantic compression on the text segments to extract key information. This results in multiple text summaries, which are stored in a vector database for easy retrieval later. When a corresponding natural language processing task needs to be executed, the multiple text summaries are first matched against the task instruction data to obtain target text data related to the task instruction data. Then, the task instruction data and target text data are input into a large language model to obtain the final output. This application achieves information summary extraction by segmenting and semantically compressing long text data. During use, it can be called from the vector database as needed, guiding a pre-trained large language model to simulate human understanding, filtering, and backtracking behavior when processing large-scale text data without modifying the original model structure or parameters, thereby improving the response efficiency and accuracy of the large language model in long text processing tasks. Furthermore, since it does not require modification of the original model structure or parameters, it is not limited by hardware resources or architecture, making it easy to deploy. Attached Figure Description

[0015] Figure 1 This is a diagram illustrating the application environment of a data processing method in one embodiment.

[0016] Figure 2 This is a flowchart illustrating a data processing method in one embodiment;

[0017] Figure 3 This is a flowchart illustrating the data processing method in another embodiment;

[0018] Figure 4 This is a flowchart illustrating the process of obtaining a text segment in one embodiment;

[0019] Figure 5 This is a flowchart illustrating the process of obtaining a text segment in another embodiment;

[0020] Figure 6 This is a flowchart illustrating the process of obtaining a text summary in one embodiment;

[0021] Figure 7 This is a schematic diagram of the process for obtaining target text data in one embodiment;

[0022] Figure 8 This is a schematic diagram of a data processing device module in one embodiment;

[0023] Figure 9 This is an internal structural diagram of a computer device in one embodiment;

[0024] Figure 10This is a diagram of the internal structure of a computer device in another embodiment. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0026] The data processing method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0027] In some embodiments, the data processing method can be executed by server 104. Server 104 can acquire long text data and task instruction data through a communication network, then segment the long text data to obtain multiple text segments. After semantic compression of the text segments, multiple text summaries are obtained. Then, the multiple text summaries are matched based on the task instruction data to obtain the target text data. Finally, the task instruction data and target text data are input into a large language model to obtain the final output result. In some other embodiments, the data processing method can also be executed by terminal 102 alone, or by terminal 102 and server 104 jointly. In practical applications, the executing entity of the data processing method can be flexibly adjusted according to the specific application scenario.

[0028] The data processing method provided in this application can be executed by an agent configured in terminal 102 or server 104. An agent is an important concept in artificial intelligence and computer science, referring to an autonomous system that achieves a goal through perception and action in a given environment. An agent possesses three functions: perception, reasoning, and action. This allows it to make decisions based on perceived environmental information and influence or change the environment through certain actions. It is not merely a passive tool but a tool with a degree of initiative and adaptability, designed to mimic certain intelligent behaviors of humans or animals to achieve more efficient and intelligent operation.

[0029] In one embodiment, such as Figure 2 As shown, a data processing method is provided. Taking the application of this method in a data processing system as an example, the method includes the following steps:

[0030] Step S100: Obtain long text data and task instruction data.

[0031] Specifically, when processing data, the data processing system first acquires the long text data and task instruction data input by the user. The task instruction data corresponds to the long text data and is used to characterize the processing tasks that need to be performed based on the long text data. For specific examples, the long text data can be long work order data, fault log data, and configuration change record data; the task instruction data is used to instruct on fault diagnosis based on this data. The long text data can also be construction / maintenance procedure document data; the task instruction data is used to instruct on extracting key processes based on this data. The long text data can be equipment manual data; the task instruction data is used to instruct on summarizing and interactive Q&A based on this data. The long text data in this application can be in the millions of words range, and it can vary according to specific usage needs.

[0032] Step S200: The long text data is segmented to obtain multiple text segments.

[0033] Specifically, after the data processing system obtains long text data, it first segments the long text data to obtain multiple shorter text segments. When performing segmentation, methods such as fixed-length segmentation (e.g., fixed number of characters or lines), natural segmentation (e.g., line breaks, headings, formatting marks), semantic segmentation, or mixed segmentation can be used.

[0034] Step S300: Semantic compression is performed on the text segment to obtain multiple text summaries.

[0035] Specifically, after obtaining multiple text segments, the data processing system performs semantic compression on each segment to obtain corresponding text summaries. During semantic compression, extractive summarization methods can be used to extract keywords, or generative summarization methods can be used to generate new sentences. Semantic compression further reduces data volume and storage redundancy. After obtaining multiple text summaries, the data processing system vectorizes them and stores them in a vector database with an index structure for subsequent retrieval. For example, the text summaries can be stored in a base layer as long-term memory, and relevant information can be dynamically loaded from the base layer into the task layer when needed to support the current task processing.

[0036] Step S400: Match multiple text summaries based on task instruction data to obtain target text data.

[0037] Specifically, after the data processing system obtains the task instruction data, it first matches multiple stored text summaries against the task instruction data to obtain the target text data that meets the requirements. During the matching process, vector similarity matching can be used directly to extract text summaries with similarity reaching a threshold and use them as the target text data; alternatively, the task instruction data can be semantically expanded before matching in a vector database to obtain the target text data.

[0038] Step S500: Input the task instruction data and target text data into the large language model to obtain the final output result.

[0039] Specifically, after the data processing system obtains the target text data, it inputs both the task instruction data and the target text data into a trained large language model. The large language model then understands the task instruction data and combines it with the target text data to obtain the final output result.

[0040] The aforementioned data processing method extracts information summaries by segmenting and semantically compressing long text data. During use, it can be invoked on demand from a vector database to guide a pre-trained large-scale language model. This simulates human understanding, filtering, and backtracking behavior when processing large-scale text data without modifying the original model structure or parameters, thereby improving the response efficiency and accuracy of large language models in long text processing tasks. Furthermore, because it does not require modification of the original model structure or parameters, it is also less constrained by hardware resources or architecture, making it easy to deploy.

[0041] In one embodiment, such as Figure 3 As shown, the data processing method also includes:

[0042] Step S600: In response to the feedback result data, update the vector database based on the feedback result data.

[0043] Specifically, after receiving the final output, users provide feedback based on its content, inputting feedback data. Upon receiving this feedback data, the data processing system updates the vector database to optimize subsequent output. For example, if the feedback data modifies the final output (such as correcting errors or adding information), the system will correct errors or add new information to the vector database and rebuild the indexes, making the system more suitable for specific domain needs and improving its professionalism and reliability.

[0044] In one embodiment, such as Figure 4 As shown, step S200, which involves segmenting long text data to obtain multiple text segments, includes:

[0045] Step S210: Perform preliminary segmentation of the long text data based on paragraph identifiers to obtain multiple basic segments.

[0046] Specifically, in this embodiment, a hybrid segmentation approach is used during segmentation. First, the long text data is initially segmented based on paragraph identifiers (such as line breaks, headings, formatting marks, etc.) to obtain multiple basic segments. It is understood that the long text data is pre-processed data, and irrelevant characters (such as special symbols, extra spaces) and non-semantic content (such as headers / footers, repeated paragraphs, advertising information, etc.) can be removed using a rule engine or simple pattern matching.

[0047] Step S220: Input the basic segments into the large language model for content topic clustering and context boundary recognition to obtain multiple semantic segments.

[0048] Specifically, after the data processing system obtains the basic segments, it inputs these segments into a large language model for content-topic clustering. This groupes basic segments with similar topics into the same cluster. Then, the large language model analyzes the semantic coherence between sentences (such as breaks in causal relationships and topic shifts) to locate context boundaries, and uses these boundaries for further segmentation, resulting in multiple semantic segments. Within the resulting semantic segments, the semantic content of the same semantic segment is similar.

[0049] Step S230: Obtain multiple text segments based on multiple semantic segments.

[0050] Specifically, after the data processing system obtains multiple semantic segments, it adjusts these segments according to the granularity of the segmentation (such as merging related chapters) to obtain multiple text segments. This embodiment adopts a semantic boundary-driven dynamic segmentation method, which adaptively segments long text data in conjunction with the context, thereby ensuring that each text segment has complete semantic information and improving granularity controllability and logical coherence.

[0051] In one embodiment, such as Figure 5 As shown, in step S230, multiple text segments are obtained based on multiple semantic segments, including:

[0052] Step S231: Determine the segmentation granularity based on the task type of the task instruction data.

[0053] Specifically, in this embodiment, when obtaining multiple text segments based on multiple semantic segments, the task type (such as reasoning, summarizing, question answering, etc.) is first determined by parsing the task instruction data. Then, the optimal segmentation granularity is determined based on the determined task type to ensure that the segmentation results highly match the task objectives. It can be understood that one task type corresponds to one segmentation granularity. For example, when the task type is determined to be question answering, each question needs to correspond to an independent text segment; in this case, the segmentation granularity is fine-grained. When the task type is determined to be summarizing, related semantic segments need to be merged to avoid summary fragmentation; in this case, the segmentation granularity is medium-grained.

[0054] Step S232: Based on the segmentation granularity, multiple semantic segments are segmented and combined to obtain multiple text segments.

[0055] Specifically, after determining the segmentation granularity, the data processing system will perform segmentation and combination processing (such as splitting, merging, and retaining) on ​​multiple semantic segments according to the segmentation granularity, thereby obtaining multiple processed text segments that conform to the task type. This embodiment segments by combining task type, maximizing task adaptability while ensuring semantic integrity, and providing high-quality input for subsequent language processing tasks.

[0056] In one embodiment, such as Figure 6 As shown, in step S300, semantic compression is performed on the text segment to obtain multiple text summaries, including:

[0057] Step S310: Input the text segment and task instruction data into the large language model for summary extraction to obtain multiple summary data.

[0058] Specifically, in this embodiment, semantic compression is performed by combining a large language model. The data processing system simultaneously inputs text segments and task instruction data into the large language model for summary extraction, guiding the summary extraction process with task relevance to obtain multiple summary data. By combining the task instruction data, the obtained summary data can be made more relevant to the current task.

[0059] Step S320: Redundancy filtering is performed on multiple summary data to obtain a text summary.

[0060] Specifically, after obtaining multiple summary data, the data processing system performs redundancy filtering on these summaries for further compression, resulting in a filtered text summary. Redundancy filtering can be performed using methods based on semantic similarity, keywords, or preset rules to remove duplicate and redundant summary data, yielding a more concise text summary.

[0061] In one embodiment, step S320, which involves performing redundancy filtering on multiple summary data to obtain a text summary, includes: inputting the summary data into a large language model for redundancy filtering to obtain a text summary.

[0062] Specifically, in this embodiment, when performing redundancy filtering on multiple summary data, the summary data is input into a large language model, and redundancy filtering is achieved through semantic similarity calculation, information redundancy identification, and global optimization reasoning, thereby dynamically adjusting the summary structure (such as merging similar topic segments, deleting redundant connecting words, refining core information, etc.) to obtain an optimized text summary.

[0063] In one embodiment, such as Figure 7 As shown, in step S400, multiple text summaries are matched based on the task instruction data to obtain target text data, including:

[0064] Step S410: Perform vector conversion on the task instruction data to obtain the task instruction vector.

[0065] Specifically, in this embodiment, during data matching, the task instruction data is first transformed into a vector to obtain a task instruction vector. This vector transformation can be performed using a vectorization model, which can be fine-tuned using domain data to improve the semantic understanding of domain-specific terminology. It is understood that the vector transformation in this embodiment can be the same as the method for vectorizing text summaries, resulting in vectors with the same dimension.

[0066] Step S420: Perform similarity matching on the summary vector of the text summary based on the task instruction vector to obtain the target text data.

[0067] Specifically, after obtaining the task instruction vector, the data processing system calculates the similarity between the task instruction vector and the summary vector to perform similarity matching, thereby obtaining the summary vector that meets the screening results (such as being greater than the similarity threshold), and using the corresponding text summary as the target text data.

[0068] The data processing method of this application is described in detail below with a specific embodiment. After the data processing system obtains long text data and task instruction data input by the user, it performs preliminary segmentation of the long text data based on paragraph identifiers to obtain multiple basic segments. Then, the basic segments are input into a large language model for content topic clustering and context boundary recognition to obtain multiple semantic segments. Then, the segmentation granularity is determined based on the task type of the task instruction data, and the multiple semantic segments are segmented and combined based on the segmentation granularity to obtain multiple text segments. Next, the text segments and task instruction data are input into the large language model for summary extraction to obtain multiple summary data. The summary data is then input into the large language model for redundancy filtering to obtain a text summary. Then, the task instruction data is vectorized to obtain task instruction vectors, and similarity matching is performed on the summary vectors of the text summary based on the task instruction vectors to obtain target text data. Then, the task instruction data and target text data are input into the large language model to obtain the final output result. Afterwards, when the user inputs feedback result data, the system responds to the feedback result data and updates the vector database based on the feedback result data.

[0069] The data processing method in this embodiment can significantly enhance the context awareness capability of large language models. It can transform long texts that were originally difficult to understand as a whole into structured knowledge through segmentation and compression. It can effectively avoid illusions and comprehension biases by filtering redundant interference through modular memory structures. It can improve the accuracy and stability of task response. Through semantic compression, vectorized indexing, task-driven retrieval, and interactive feedback mechanisms, it can significantly improve the ability to perform efficient semantic retrieval and task response in ultra-long contexts, enabling large language models to have stronger context understanding, higher retrieval accuracy, and higher response relevance.

[0070] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0071] Based on the same inventive concept, this application also provides a data processing apparatus for implementing the data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data processing apparatus embodiments provided below can be found in the limitations of the data processing method described above, and will not be repeated here.

[0072] In one embodiment, such as Figure 8 As shown, a data processing device is provided, including: a data acquisition module, a segmentation processing module, a semantic compression module, a summary matching module, and a result calculation module, wherein:

[0073] The data acquisition module is used to acquire long text data and task instruction data;

[0074] The segmentation module is used to segment long text data into multiple text segments.

[0075] The semantic compression module is used to perform semantic compression on text segments to obtain multiple text summaries; these multiple text summaries are stored in a vector database.

[0076] The summary matching module is used to match multiple text summaries based on task instruction data to obtain the target text data;

[0077] The results calculation module is used to input task instruction data and target text data into the large language model to obtain the final output results.

[0078] In one embodiment, the data processing apparatus further includes a result feedback module, configured to update the vector database based on the feedback result data in response to the feedback result data.

[0079] In one embodiment, the segmentation module is further configured to perform preliminary segmentation of long text data based on paragraph identifiers to obtain multiple basic segments; input the basic segments into a large language model for content topic clustering and context boundary recognition to obtain multiple semantic segments; and obtain multiple text segments based on the multiple semantic segments.

[0080] In one embodiment, the segmentation module is further configured to determine the segmentation granularity based on the task type of the task instruction data; and to perform segmentation and combination processing on multiple semantic segments based on the segmentation granularity to obtain multiple text segments.

[0081] In one embodiment, the semantic compression module is further configured to input text segments and task instruction data into a large language model for summary extraction to obtain multiple summary data; and to perform redundancy filtering on the multiple summary data to obtain a text summary.

[0082] In one embodiment, the semantic compression module is also used to input the summary data into a large language model for redundancy filtering to obtain a text summary.

[0083] In one embodiment, the summary matching module is further configured to perform vector conversion on the task instruction data to obtain a task instruction vector; and perform similarity matching on the summary vector of the text summary based on the task instruction vector to obtain the target text data.

[0084] Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0085] In one embodiment, when the above modules are embedded in a computer device in hardware form, they can be interconnected via a data bus and control signal lines to achieve parallel operation and task collaboration. The modules can interact with each other via APIs (Application Programming Interfaces) to build a highly scalable data processing platform.

[0086] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 9 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a data processing method.

[0087] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 10As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a data processing method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0088] Those skilled in the art will understand that Figure 9 or Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0089] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0090] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described method embodiments.

[0091] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0092] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0093] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data processing method, characterized in that, The method includes: Retrieve long text data and task instruction data; The long text data is segmented to obtain multiple text segments; The text segment is semantically compressed to obtain multiple text summaries; wherein the multiple text summaries are stored in a vector database; Based on the task instruction data, multiple text summaries are matched to obtain target text data; The task instruction data and the target text data are input into the large language model to obtain the final output result.

2. The data processing method according to claim 1, characterized in that, The method further includes: In response to the feedback result data, the vector database is updated based on the feedback result data.

3. The data processing method according to claim 1, characterized in that, The step of segmenting the long text data to obtain multiple text segments includes: The long text data is initially segmented based on paragraph identifiers to obtain multiple basic segments; The basic segments are input into the large language model for content topic clustering and context boundary recognition to obtain multiple semantic segments. Multiple text segments are obtained based on multiple semantic segments.

4. The data processing method according to claim 3, characterized in that, Multiple text segments are obtained based on the multiple semantic segments, including: The segmentation granularity is determined based on the task type of the task instruction data. Based on the segmentation granularity, multiple semantic segments are segmented and combined to obtain multiple text segments.

5. The data processing method according to claim 1, characterized in that, Semantic compression is performed on the text segment to obtain multiple text summaries, including: The text segment and the task instruction data are input into the large language model for summary extraction to obtain multiple summary data. Redundancy filtering is performed on multiple summaries to obtain the text summary.

6. The data processing method according to claim 5, characterized in that, Redundancy filtering is performed on multiple summaries to obtain the text summary, including: The summary data is input into the large language model for redundancy filtering to obtain the text summary.

7. The data processing method according to claim 1, characterized in that, Based on the task instruction data, multiple text summaries are matched to obtain target text data, including: The task instruction data is vectorized to obtain the task instruction vector; The target text data is obtained by performing similarity matching on the summary vector of the text summary based on the task instruction vector.

8. A data processing apparatus, characterized in that, The device includes: The data acquisition module is used to acquire long text data and task instruction data; The segmentation module is used to segment the long text data to obtain multiple text segments; A semantic compression module is used to perform semantic compression on the text segment to obtain multiple text summaries; wherein the multiple text summaries are stored in a vector database; The summary matching module is used to match multiple text summaries based on the task instruction data to obtain target text data; The result calculation module is used to input the task instruction data and the target text data into the large language model to obtain the final output result.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images