Corpus processing method and device

Abstract

The embodiment of the invention provides a corpus processing method and device, and relates to the technical field of artificial intelligence, the corpus processing method comprises the following steps: storing the content of an original corpus according to a storage specification and an entity identifier to obtain one or more first storage entities, each first storage entity corresponding to a first identifier; performing corpus preprocessing based on each first storage entity to obtain one or more second storage entities corresponding to different second identifiers, and recording a first association relationship between each second identifier in the different second identifiers and the first identifier of the corresponding first storage entity, each second storage entity is used for indicating the content of the pre-training corpus corresponding to the original corpus; and under the condition that the content of the stored original corpus or the pre-training corpus is changed, according to the first association relationship, performing updating corresponding to the change on the content of the original corpus or the content of the pre-training corpus associated with the change content in the first corpus. According to the scheme, the corpus quality can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a corpus processing method and apparatus. Background Technology

[0002] With the development of technology and demand, artificial intelligence is gradually shifting from "model-centric" to "data-centric," with data mainly including corpora. High-quality corpora can improve the accuracy of models. Therefore, in model training, the requirements for corpus quality are becoming increasingly higher, and the demand for corpus quantity is also increasing.

[0003] In related technologies, the main process involves acquiring raw corpora through data collection, cleaning the raw corpora to obtain pre-training corpora, and then annotating the pre-training corpora to obtain annotated corpora for training the model. For example, to acquire pre-training corpora, online data resources can be exported, converted in format, logged, and copied to obtain raw corpora. The content of the raw corpora is then extracted and uploaded to a data cleaning platform. The data cleaning platform performs corpus preprocessing: corpus extraction and corpus cleaning, to obtain the pre-training corpora.

[0004] However, in practical applications, online data resources are prone to change over time, and the original corpus changes accordingly. The pre-training corpus may also undergo corrections and modifications. In this case, if the content of the original corpus and the pre-training corpus is not synchronized, the pre-training corpus becomes untraceable, thereby reducing the credibility and security of both the pre-training corpus and the original corpus, leading to problems with low corpus quality. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a corpus processing method and apparatus. In this corpus processing method, by establishing a correlation between a first storage entity and a first identifier indicating the content of the original corpus, and a second identifier indicating the content of a second storage entity indicating the content of the pre-training corpus, bidirectional synchronization of content between the original corpus and the pre-training corpus is achieved, improving the reliability and security of both the pre-training and original corpora, thereby enhancing the quality of both.

[0006] In a first aspect, embodiments of this application provide a corpus processing method, the method comprising: storing the content of an original corpus according to storage specifications and entity identifiers to obtain one or more first storage entities, each first storage entity corresponding to a first identifier; performing corpus preprocessing based on each first storage entity to obtain one or more second storage entities each corresponding to a different second identifier, and recording a first association relationship between each second identifier in the different second identifiers and the first identifier of the corresponding first storage entity, each of the one or more second storage entities being used to indicate the content of pre-training corpus corresponding to the original corpus; and updating the content of the original corpus or the content of the pre-training corpus associated with the changed content in the first corpus according to the first association relationship, the first corpus including the original corpus or the pre-training corpus.

[0007] In this embodiment, a first storage entity corresponding to the original corpus and a second storage entity corresponding to the pre-training corpus are respectively marked by an associated first identifier and a second identifier, ensuring that the content of the original corpus and the content of the pre-training corpus are related. Thus, when the content of either corpus changes and is stored, the content of the associated corpus can be updated synchronously, achieving bidirectional synchronization between the original corpus and the pre-training corpus. This ensures that the original corpus and the pre-training corpus are of the same origin, updatable, and traceable, thereby improving the reliability and security of the pre-training corpus and enhancing its quality.

[0008] According to the first aspect, the content of the original corpus includes text content, and the storage specification includes one or more editable text blocks; according to the storage specification and entity identifiers, the content of the original corpus is stored to obtain one or more first storage entities, each first storage entity corresponding to a first identifier, including: dividing the content of the original corpus according to the storage specification and extracting the divided content to obtain multiple storage entities; marking each of the multiple storage entities with a different entity identifier to obtain multiple first storage entities with different first identifiers respectively; performing corpus preprocessing based on each first storage entity to obtain one or more second storage entities with different second identifiers respectively, including: performing corpus extraction and corpus cleaning on each of the multiple first storage entities to obtain second storage entities corresponding to the first storage entities; marking each second storage entity with a different second identifier to obtain multiple second storage entities with different second identifiers respectively.

[0009] In this embodiment of the application, when the content of the original corpus is text content, the storage specification includes one or more editable text blocks. Based on this, there is no format conversion between the original corpus and the smallest editable block of the original content. Before the smallest editable block of the original content is converted into the smallest editable block of the pre-training corpus, the required corpus content is extracted from the smallest editable block by the corpus extraction tool without any intermediate format conversion. This achieves the maximum preservation and restoration of the original corpus information and further improves the quality of the pre-training corpus.

[0010] According to the first aspect, or any implementation of the first aspect above, when the content of the stored first corpus changes, the content of the original corpus or the content of the pre-training corpus associated with the changed content in the first corpus is updated according to the first association relationship, including: when the content of the stored first corpus changes, determining the first identifier of the first updated storage entity with the changed content based on the editable text block that forms the changed content, wherein the first corpus includes the original corpus; performing corpus extraction and corpus cleaning on the first updated storage entity to obtain a second updated storage entity corresponding to the first identifier of the first updated storage entity; and updating the second storage entity marked by the second identifier corresponding to the first identifier of the first updated storage entity to the second updated storage entity corresponding to the first identifier of the first updated storage entity according to the first association relationship.

[0011] In this embodiment of the application, when the content of the original corpus is text, the storage specification includes one or more editable text blocks. Thus, the range of the first storage entity that needs to be updated can be determined according to the editable text blocks that form the changed content. In this way, corpus preprocessing can be performed directly based on the first storage entity that needs to be updated and its corresponding first identifier, quickly obtaining the updated second storage entity, thereby improving the update efficiency of the pre-training corpus, which in turn improves the processing efficiency of the pre-training corpus.

[0012] According to the first aspect, or any implementation of the first aspect above, when the content of the stored first corpus changes, the content of the original corpus or the content of the pre-training corpus associated with the changed content in the first corpus is updated according to the first association relationship, including: when the content of the stored first corpus changes, determining the second identifier of the second updated storage entity with the changed content based on the editable text block that forms the changed content, wherein the first corpus includes the pre-training corpus; and updating the editable text block in the second storage entity marked by the first identifier corresponding to the second identifier of the second updated storage entity to the editable text block in the second updated storage entity according to the first association relationship.

[0013] In this embodiment of the application, when the content of the original corpus is text content, the storage specification includes one or more editable text blocks. Thus, the range of the second storage entity that needs to be updated can be determined according to the editable text blocks that form the changed content. In this way, the corresponding editable text blocks in the first storage entity can be directly updated based on the second storage entity that needs to be updated and the corresponding second identifier, quickly obtaining the updated first storage entity, thereby improving the updating efficiency of the original corpus, which in turn improves the processing efficiency of the original corpus.

[0014] According to the first aspect, or any implementation of the first aspect above, after recording the first association relationship between each second identifier in different second identifiers and the first identifier of the corresponding first storage entity, the method further includes: according to the first association relationship, outputting the content of the stored original corpus and the content of the pre-training corpus corresponding to the original corpus to a visualization device for display; and / or, according to the first association relationship, obtaining the comparison information between the content of the stored original corpus and the content of the pre-training corpus corresponding to the original corpus, and outputting the comparison information to a visualization device for display.

[0015] This application's embodiments improve the ease with which users can manage, such as modify and adjust, the original corpus and / or pre-training corpus by visualizing them, thereby further improving corpus quality and processing efficiency. Furthermore, visualization of comparison information between related corpora can improve the accuracy of corpus management, further enhancing corpus quality and acquisition efficiency.

[0016] According to the first aspect, or any implementation of the first aspect above, the comparison information is used to indicate one or more of the same content, relatively deleted content, and relatively added content between the content of the stored original corpus and the pre-training corpus corresponding to the original corpus.

[0017] This application embodiment provides diverse corpus analysis perspectives by comparing various information between different corpora, which can further improve the accuracy and efficiency of corpus management, thereby further improving corpus quality and processing efficiency.

[0018] According to the first aspect, or any implementation of the first aspect above, after recording the first association relationship between each second identifier in different second identifiers and the first identifier of the corresponding first storage entity, the method further includes: performing corpus annotation processing based on each of the one or more second storage entities to obtain one or more fine-tuned corpora corresponding to different third identifiers; recording the second association relationship between each third identifier in different third identifiers and the second identifier of the corresponding second storage entity; and, in the case of a change in the content of the stored second corpus, updating the content of the pre-training corpus or the content of the fine-tuned corpus associated with the changed content in the second corpus according to the second association relationship, wherein the second corpus includes the pre-training corpus or any fine-tuned corpus in one or more fine-tuned corpora.

[0019] In this embodiment, the second storage entity corresponding to the pre-training corpus and the content of the corresponding fine-tuning corpus are marked respectively by associated second and third identifiers, ensuring that the content of the pre-training corpus and the content of the fine-tuning corpus are related. Thus, when the content of the pre-training corpus changes, the content of the associated fine-tuning corpus can be updated synchronously, achieving bidirectional synchronization between the pre-training corpus and the fine-tuning corpus. This ensures that the pre-training corpus and the fine-tuning corpus are of the same origin, updatable, and traceable, thereby improving the reliability and security of the pre-training corpus and the fine-tuning corpus, and ultimately improving their quality.

[0020] According to the first aspect, or any implementation of the first aspect above, when the content of the stored second corpus changes, the content of the pre-training corpus or the content of the fine-tuned corpus associated with the changed content in the second corpus is updated according to the second association relationship, corresponding to the change. This includes: when the content of the stored second corpus changes, obtaining updated content associated with the changed content in the second corpus based on the changed content in the second corpus, the prompt, and the pre-trained fine-tuning model; the prompt is used to instruct the fine-tuning model to output content conforming to the form of the pre-training corpus or content conforming to the form of the fine-tuning corpus; the fine-tuning model is a model trained using sample corpus, the adjustment labels of the sample corpus, and the prompt corresponding to the adjustment target; and updating the content of the pre-training corpus or the content of the fine-tuned corpus associated with the changed content in the second corpus to the updated content associated with the changed content in the second corpus according to the second association relationship.

[0021] In this embodiment, a pre-trained fine-tuning model is used to generate pre-trained corpora or fine-tuning corpora, thereby further improving the convenience of bidirectional updates between pre-trained corpora and fine-tuning corpora, reducing the acquisition cost of fine-tuning corpora and pre-trained corpora, and facilitating online batch acquisition of fine-tuning corpora.

[0022] Secondly, embodiments of this application provide a corpus processing apparatus, comprising: a corpus storage module, configured to store the content of original corpus according to storage specifications and entity identifiers to obtain one or more first storage entities, each first storage entity corresponding to a first identifier; a corpus association module, configured to perform corpus preprocessing based on each first storage entity to obtain one or more second storage entities respectively corresponding to different second identifiers, and record a first association relationship between each second identifier in the different second identifiers and the first identifier of the corresponding first storage entity, each of the one or more second storage entities being used to indicate the content of pre-training corpus corresponding to the original corpus; and a corpus synchronization module, configured to update the content of the original corpus or the content of the pre-training corpus associated with the changed content in the first corpus according to the first association relationship when the content of the stored first corpus changes, wherein the first corpus includes the original corpus or the pre-training corpus.

[0023] According to the second aspect, the content of the original corpus includes text content, and the storage specification includes one or more editable text blocks; the corpus storage module is specifically used for: dividing the content of the original corpus according to the storage specification, and extracting the divided content to obtain multiple storage entities; marking each storage entity in the multiple storage entities with different entity identifiers to obtain multiple first storage entities corresponding to different first identifiers; the corpus association module is specifically used for: performing corpus extraction and corpus cleaning on each of the multiple first storage entities to obtain second storage entities corresponding to the first storage entities; marking each second storage entity with different second identifiers to obtain multiple second storage entities corresponding to different second identifiers.

[0024] According to the second aspect, or any implementation of the second aspect above, the corpus synchronization module is specifically used for: when the content of the stored first corpus changes, determining the first identifier of the first updated storage entity whose content has changed based on the editable text block that forms the changed content, wherein the first corpus includes the original corpus; performing corpus extraction and corpus cleaning on the first updated storage entity to obtain a second updated storage entity corresponding to the first identifier of the first updated storage entity; and updating the second storage entity marked by the second identifier corresponding to the first identifier of the first updated storage entity to the second updated storage entity corresponding to the first identifier of the first updated storage entity according to the first association relationship.

[0025] According to the second aspect, or any implementation of the second aspect above, the corpus synchronization module is specifically used for: when the content of the stored first corpus changes, determining the second identifier of the second updated storage entity whose content has changed based on the editable text block that forms the changed content, wherein the first corpus includes pre-training corpus; and updating the editable text block in the second storage entity marked by the first identifier corresponding to the second identifier of the second updated storage entity to the editable text block in the second updated storage entity according to the first association relationship.

[0026] According to the second aspect, or any implementation of the second aspect above, the device further includes a visualization module, used for: outputting the content of the stored original corpus and the content of the pre-training corpus corresponding to the original corpus to the visualization device for display according to the first association relationship; and / or, obtaining the comparison information between the content of the stored original corpus and the content of the pre-training corpus corresponding to the original corpus according to the first association relationship, and outputting the comparison information to the visualization device for display.

[0027] According to the second aspect, or any implementation of the second aspect above, the comparison information is used to indicate one or more of the same content, relatively deleted content, and relatively added content between the content of the stored original corpus and the pre-training corpus corresponding to the original corpus.

[0028] According to the second aspect, or any implementation of the second aspect above, the corpus association module is further configured to: after recording the first association relationship between each second identifier in different second identifiers and the first identifier of the corresponding first storage entity, perform corpus annotation processing based on each of the one or more second storage entities to obtain one or more fine-tuned corpora corresponding to different third identifiers; record the second association relationship between each third identifier in different third identifiers and the second identifier of the corresponding second storage entity; the corpus synchronization module is further configured to: when the content of the stored second corpus changes, update the content of the pre-training corpus or the content of the fine-tuned corpus associated with the changed content in the second corpus according to the second association relationship, wherein the second corpus includes the pre-training corpus or any fine-tuned corpus in one or more fine-tuned corpora.

[0029] According to the second aspect, or any implementation of the second aspect above, the corpus synchronization module is specifically used for: when the content of the stored second corpus changes, obtaining updated content associated with the changed content in the second corpus based on the changed content, prompts, and a pre-trained fine-tuning model in the second corpus; the prompts are used to instruct the fine-tuning model to output content that conforms to the form of the pre-trained corpus or content that conforms to the form of the fine-tuned corpus; the fine-tuning model is a model trained using sample corpus, the adjustment labels of the sample corpus, and the prompts corresponding to the adjustment targets; and updating the content of the pre-trained corpus or the content of the fine-tuned corpus associated with the changed content in the second corpus to the updated content associated with the changed content in the second corpus according to the second association relationship.

[0030] The second aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the second aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0031] Thirdly, embodiments of this application provide a computing device, including: a processor and a memory; the processor and the memory are connected; the memory is used to store one or more programs; when one or more programs are executed by one or more processors, the one or more processors implement the method as described in the first aspect and any implementation thereof.

[0032] Fourthly, embodiments of this application provide a computer-readable medium for storing a computer program, the computer program including instructions for performing the method in the first aspect or any possible implementation of the first aspect.

[0033] Fifthly, embodiments of this application provide a computer program including instructions for performing the method in the first aspect or any possible implementation thereof.

[0034] Sixthly, embodiments of this application provide a chip including a processing circuit and transceiver pins. The transceiver pins and the processing circuit communicate with each other via an internal connection path. The processing circuit executes the method in the first aspect or any possible implementation of the first aspect to control the receiving pin to receive signals and to control the transmitting pin to transmit signals. Attached Figure Description

[0035] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a diagram illustrating one method of obtaining annotated corpora;

[0037] Figure 2 This is a diagram illustrating method two for obtaining annotated corpora;

[0038] Figure 3 This is a structural block diagram of a corpus processing platform provided in an embodiment of this application;

[0039] Figure 4 This is a schematic flowchart of a corpus processing method provided in an embodiment of this application;

[0040] Figure 5 This is an example diagram illustrating the implementation process of bidirectional updating and visualization of the original and pre-trained corpora provided in the embodiments of this application;

[0041] Figure 6 This is an example diagram illustrating the implementation process of bidirectional updating of pre-trained corpus and fine-tuned corpus provided in the embodiments of this application;

[0042] Figure 7 This is a schematic diagram illustrating the application process of a corpus processing method provided in an embodiment of this application;

[0043] Figure 8 This is a structural block diagram of a corpus processing device provided in an embodiment of this application;

[0044] Figure 9 This is a structural block diagram of a computing device provided in an embodiment of this application;

[0045] Figure 10 This is a structural block diagram of a computing device cluster provided in an embodiment of this application. Detailed Implementation

[0046] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0047] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0048] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first target object" and "second target object," etc., are used to distinguish different target objects, not to describe a specific order of target objects.

[0049] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0050] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0051] To facilitate understanding of this embodiment, some technical terms and background technologies involved in this embodiment will be introduced first:

[0052] Original content: refers to the original content of various documents, such as the content of manually written Word or XML documents. That is, the content of the pre-training corpus before cleaning is called original content or original corpus.

[0053] Pre-training corpus: This is the collection of text data used to train a large language model. These corpora can come from various resources containing text data, such as news articles, social media posts, and e-books. By processing and cleaning the content of these resources, i.e., the raw corpus, a dataset suitable for model training, i.e., the pre-training corpus, can be obtained.

[0054] Annotated corpus: This is the corpus used for supervised training of the model. Unlike pre-training corpus, labeled corpus consists of data that has been manually labeled, categorized, or answered. The labels, categories, or answers serve as learning points for the large model to align with human understanding.

[0055] A prompt is the input text used to trigger a response from an AI model, indicating the form in which the model should respond. The purpose of using a prompt is to guide a large language model to respond in a specific way (e.g., speech, image, text, text paradigm, etc.) as indicated by the prompt, thereby meeting the user's needs or completing a specific task.

[0056] Longest common subsequence (LCS): A sequence S is called the longest common subsequence of the known sequences if it is a subsequence of two or more known sequences and is the longest of all sequences that meet this condition. It can also be called the maximum common subsequence.

[0057] Supervised fine-tuning (SFT): A commonly used fine-tuning method that uses labeled data to train the model, optimizes the model parameters, and ensures that the model can learn the features of the target task.

[0058] In related technologies, the acquisition of pre-training corpus may include: processing available online materials (such as online documents, e-books, e-journals, newspapers, books, scripts, etc.) through data crawling, OCR image recognition, etc., to obtain the content of the above online materials in formats such as txt, word, or pdf; uploading the obtained content to a corpus extraction and cleaning platform; extracting all the information; and removing useless or unqualified information from the extracted information according to cleaning rules to obtain pre-training corpus.

[0059] For example, Figure 1 This is a diagram illustrating method one for obtaining annotated corpora. For example... Figure 1 As shown, the original corpus is uploaded to the annotation platform for manual annotation and review. ① represents the process of synchronizing the operation log to the data cleaning platform. In this process, the data transfer between the digital writing platform and the data cleaning platform is unidirectional, manual, and the data format is inconsistent. This results in the original writing content on the writing platform becoming incompatible with the original content after cleaning, making it impossible to synchronize updates from the cleaned content to the original writing content. Similarly, updates to the original writing content cannot be accurately updated to the corresponding pre-training corpus in the corpus platform. ③ represents the process of manually cleaning the corpus on the data cleaning platform. This process involves data format changes during the conversion from the original corpus to the pre-training corpus, which can easily lead to information loss. ⑤ represents all operation procedures on the data cleaning platform. This process lacks a reverse synchronization mechanism for content updates.

[0060] For example, Figure 2 This is a diagram illustrating method two for obtaining annotated corpora. For example... Figure 2 As shown, the process involves uploading the original corpus to an annotation platform, where it is automatically annotated, then manually corrected and reviewed. ① represents the process of importing the digitized content into the data cleaning platform as a Word document; in this process, the original corpus and pre-training corpus cannot be synchronized bidirectionally. ② represents the process of writers updating their writing content on the writing platform; this process is prone to information loss during the conversion from original to pre-training corpus. ④ represents the process of data engineers manually reviewing the corpus on the data cleaning platform. In this process, the fine-tuned corpus and pre-training corpus cannot be synchronized bidirectionally, resulting in poor corpus quality. Furthermore, the manual workload is large, and it is impossible to obtain annotated corpus in batches.

[0061] In summary, in corpus processing, the main problem with obtaining pre-training corpus is that the content of the original corpus and the pre-training corpus is not synchronized, which makes the pre-training corpus untraceable. This reduces the credibility and security of both the pre-training corpus and the original corpus, resulting in low corpus quality.

[0062] This application provides a corpus processing method to solve the above-mentioned problems. The method uses an associated first identifier and a second identifier to mark a first storage entity corresponding to the original corpus and a second storage entity corresponding to the pre-training corpus, respectively, ensuring that the content of the original corpus and the content of the pre-training corpus are related. Thus, when the content of either corpus changes and is stored, the content of the associated corpus can be updated synchronously, achieving bidirectional synchronization between the original corpus and the pre-training corpus. This ensures that the original corpus and the pre-training corpus are of the same origin, updatable, and traceable, thereby improving the reliability and security of the pre-training corpus and enhancing its quality.

[0063] Before describing the technical solutions of the embodiments of this application, the operating platform of the corpus processing method of the embodiments of this application will first be described with reference to the accompanying drawings. The embodiments of this application can be applied to a corpus processing platform. In the specific implementation of the embodiments of this application, the corpus processing platform may be, for example, a computer, a server, or a computing device cluster composed of multiple computing devices. For example, Figure 3 This is one of the structural block diagrams of a corpus processing platform provided in an embodiment of this application. For example... Figure 3As shown, the corpus processing platform may include a corpus processing device for processing the raw corpus to generate pre-training corpus and / or fine-tuned corpus. The fine-tuned corpus is similar to the labeled corpus, except that it can be obtained through a pre-trained fine-tuning model. This model is trained using sample corpus, adjustment labels on the sample corpus, and prompts corresponding to the adjustment targets. The adjustment targets may include, for example, extracting specific content, classification, labeling, or generating knowledge graphs. The corpus processing device may specifically include a corpus cleaning unit for preprocessing the raw corpus, such as corpus extraction and corpus cleaning, to obtain the pre-training corpus.

[0064] In one example, the corpus processing device may further include a corpus annotation unit and / or a corpus processing information visualization unit. The corpus annotation unit is used to annotate the pre-training corpus to obtain a fine-tuned corpus. The corpus processing information visualization unit is used to visualize the corpus processing information, which may include, for example, the correlation between the original corpus and the pre-training corpus, content comparison information, etc.

[0065] In practical applications, the units in the corpus processing device can be applied as plug-ins on any content generation platform.

[0066] In one example, the corpus processing platform may also include a raw corpus production device to provide a raw corpus production operation area for writing, reviewing, and revising. In this case, it is equivalent to merging the raw corpus production platform and the corpus processing device into a single platform, so that the corpus processing does not require exporting, importing, or format conversion, which helps improve the quality of the corpus.

[0067] It should be understood that, Figure 3 The platform shown is just one example of a corpus processing platform, and a corpus processing platform may have more or fewer components than shown in the figure, may combine two or more components, or may have different component configurations. Figure 1 The various components shown can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0068] The following is combined Figures 4 to 7 The corpus processing method provided in the embodiments of this application will be described in detail.

[0069] For example, Figure 4 This is a flowchart illustrating a corpus processing method provided in an embodiment of this application. For example... Figure 4 As shown, the method may include:

[0070] S401, based on the storage specifications and entity identifiers, store the content of the original corpus to obtain one or more first storage entities, each of which corresponds to a first identifier.

[0071] Storage specifications are used to indicate the content length or content range of a storage entity. For example, Figure 5 This is an example diagram illustrating the implementation process of bidirectional updating and visualization of the original and pre-trained corpora provided in the embodiments of this application. For example... Figure 5 As shown, for the original corpus created by the user, that is... Figure 5 The original data in the corpus, based on its logical expression and the storage method of the original system, stores the content of the original corpus that meets the storage specifications as an entity and assigns it a unique ID (row_data_entity_id), which is also the first identifier. The storage specifications can be visual display content, such as the entire corpus content, the smallest visual content, the largest visual content, or storable content, such as the smallest storable content, the largest storable content, etc. In one example, the stored entity can be the smallest stored entity: a sentence or multiple sentences. The first stored entity is also the smallest stored entity in the original corpus. For example, if the original corpus content is: P1: We are the sea of ​​flowers of our motherland; P2: Embrace the era with youth, then the smallest stored entity of the original corpus is: p1 + p2.

[0072] S402, perform corpus preprocessing based on each first storage entity to obtain one or more second storage entities that correspond to different second identifiers, and record the first association relationship between each second identifier in the different second identifiers and the first identifier of the corresponding first storage entity. Each of the one or more second storage entities is used to indicate the content of the pre-training corpus corresponding to the original corpus.

[0073] See also Figure 5 During the generation of the original corpus, the minimum storage entity (i.e., the second storage entity) of the pre-training corpus, corresponding to the storage entity `row_data_entity_id` of the original corpus, is generated simultaneously and assigned a unique ID (`corpus_entity_id`), i.e., the second identifier. `row_data_entity_id` and `corpus_entity_id` are in one-to-one correspondence, and their relationship is recorded. For example, in the pre-training corpus: D1: We are a sea of ​​flowers; D2: Youth embraces the era; minimum storage entity Entity: d1+d2. In this case, the correspondence is P1-D1; P2-D2.

[0074] In one optional implementation, the minimum storage entity of the original corpus can be decomposed into several minimum editable blocks, i.e., several editable text blocks. Each minimum editable block of the original content has a unique ID (mini_row_data_block_id). Since the minimum storage entity of the original corpus consists of multiple minimum editable blocks, in terms of storage relationships, one row_data_entity_id can contain multiple mini_row_data_block_ids. Assuming the minimum storage entity is a sentence or multiple sentences, then the minimum editable block is a sentence. Based on this, a one-to-one correspondence is established between the minimum editable blocks of the two, and the original data is extracted and cleaned according to the minimum editable blocks to obtain the pre-training corpus. Therefore, each minimum editable block of the original content will have a corresponding minimum editable block of the pre-training corpus. Similar to the minimum editable blocks of the original content, each minimum editable block of the pre-training corpus also maps to a unique ID (mini_corpus_data_block_id). The minimum storage entity of the pre-training corpus can be decomposed into several minimum editable blocks of the pre-training corpus.

[0075] It is understood that the editable text block can be the largest editable block, such as the entire original corpus. The specific settings of the editable text block can be set according to application requirements, and this application embodiment does not limit this.

[0076] See also Figure 5 As can be seen from the changes in the form of data during the extraction and cleaning of pre-training corpus, in this embodiment, there is no format conversion between the original data and the smallest editable block of the original content. Before the smallest editable block of the original content is converted to the smallest editable block of the pre-training corpus, the required corpus content is extracted from the smallest editable block by the corpus extraction tool without any intermediate format conversion. The unique associated ID is retained, and irreversible information loss caused by format conversion is avoided, thus achieving the maximum preservation and restoration of the original data information.

[0077] S403, if the content of the stored first corpus changes, according to the first association relationship, update the content of the original corpus or the content of the pre-training corpus associated with the changed content in the first corpus in accordance with the change. The first corpus includes the original corpus or the pre-training corpus.

[0078] For example, a one-to-one association between the original data and the pre-training corpus is established from the storage entity to the smallest content editing entity. When the original data changes and the content creator saves the original data, the related entities and the smallest editable block of the pre-training corpus are synchronously extracted and cleaned based on the unique ID association. After the synchronously updated pre-training corpus is saved, it is displayed on the page, thus achieving the update and synchronization from the original data to the pre-training corpus. Similarly, when the pre-training corpus is edited and modified by the creator, triggering a save, the updated content using the pre-training corpus is also synchronously updated in reverse according to the scope of the smallest editable block, based on the corresponding unique ID association. This two-way update mechanism enables the visualization and traceability of the original content and the pre-training corpus.

[0079] In one optional implementation, after associating the corpus, the corpus processing platform can output the content of the stored original corpus and the content of the pre-training corpus corresponding to the original corpus to a visualization device for display based on the first association relationship; and / or, based on the first association relationship, obtain the comparison information between the content of the stored original corpus and the content of the pre-training corpus corresponding to the original corpus, and output the comparison information to a visualization device for display.

[0080] For example, the corpus processing apparatus provided in this application embodiment can provide a corpus dashboard embedded in the data production source page. This allows the producer of the original data during corpus production (i.e., processing) to view the pre-trained corpus generated from the corresponding data in real time. The producer of the original data can then judge the correctness of the generated pre-trained corpus information. The visualization of the original corpus and the corresponding pre-trained corpus improves the convenience for users to manage, such as modify and adjust, the original corpus and / or pre-trained corpus, thereby further improving corpus quality and processing efficiency. Furthermore, the visualization of comparison information between related corpora can improve the accuracy of corpus management, thereby further improving corpus quality and acquisition efficiency.

[0081] In one alternative implementation, the comparison information is used to indicate one or more of the same content, relatively deleted content, and relatively added content between the content of the stored original corpus and the pre-training corpus corresponding to the original corpus.

[0082] For example, the original corpus and the pre-training corpus are treated as two string objects, two text segments. Each character is counted as a length, and both characters are encoded separately. Each character has a unique encoded index value in each string object, representing its specific position within the string object. Based on this, the maximum common subsequence algorithm is used to calculate the maximum common content between the stored original corpus and the corresponding pre-training corpus. Relative deletions and additions are calculated through string comparison. A recursive algorithm is used to continuously perform the above operation on every pair of strings in the two objects being compared until all comparisons of the two objects are completed. In one example, a prefix sum algorithm can be used to calculate the index range of the common parts of the two strings in each string. In another example, the comparison results can be merged according to a certain order, such as "deletion, addition, common subsequence," which may also include the location indices of relatively deleted and relatively added content in the merged content. In one example, the comparison results obtained above are visualized using visualization page rendering technology.

[0083] In one alternative implementation, for the above... Figure 2 In process ②, there is a problem of poor corpus quality due to the inability to achieve bidirectional synchronization between the fine-tuned corpus and the pre-training corpus. In this embodiment, the fine-tuned corpus is generated from the pre-training corpus using a method similar to the bidirectional synchronization between the original corpus and the pre-training corpus. Regarding the above... Figure 2 The fourth step in the process involves a large amount of manual work and makes it impossible to obtain labeled corpora in batches. This embodiment of the application addresses this by using a pre-trained fine-tuning model to process the pre-trained corpus to obtain fine-tuned corpora, thereby reducing manual work and ensuring batch acquisition of fine-tuned corpora. The following is a combination of... Figure 6 The process of fine-tuning the corpus is explained in detail.

[0084] For example, Figure 6 This is an example diagram illustrating the implementation process of bidirectional updating of pre-trained corpora and fine-tuned corpora provided in the embodiments of this application. For example... Figure 6 As shown, a fine-tuning model specifically designed for generating fine-tuned corpora is pre-trained based on a deep learning model and SFT fine-tuning techniques. Then, using the fine-tuning model and a prompt, the model can generate as many initial drafts of the fine-tuned corpora as possible based on the input content and the prompt's requirements. These initial drafts can be used directly as fine-tuned corpora, or they can be used after manual review and modification.

[0085] After generating the fine-tuned corpus, if the fine-tuned corpus changes, the corpus processing platform can reconstruct the fine-tuned corpus into a declarative sentence using the aforementioned fine-tuning model. Based on the recorded source pre-training corpus ID and string length index range of the annotated corpus, the regenerated declarative sentence replaces the text specified in the original pre-training corpus, thus updating the content of the fine-tuned corpus to the pre-training corpus. At this time, the prompt is used to indicate the format or form of the pre-training corpus, such as "a declarative sentence describing the text".

[0086] In one optional implementation, when a user triggers an update or modification (i.e., a change) to the fine-tuning corpus, the content update and synchronization of the fine-tuning corpus to the pre-training corpus can be triggered automatically or by the user's choice. The synchronization method may include: the corpus processing platform, based on the association relationships recorded when storing the corpus, retrieves the content corresponding to the minimum editable block ID (i.e., the third identifier) ​​of the pre-training corpus associated with the fine-tuning corpus, along with related instructions (i.e., prompts indicating the content format of the pre-training corpus), inputs these instructions into the deep learning model (i.e., the fine-tuning model), regenerates the minimum editable block of the pre-training corpus corresponding to the changed content, and replaces the original minimum editable block content in the pre-training corpus with the newly generated corpus based on the ID.

[0087] In one optional implementation, similar to the reverse update from the fine-tuned corpus to the pre-training corpus described above, when the pre-training corpus changes, the corpus processing platform obtains the ID of the smallest editable block of the pre-training corpus corresponding to the changed content, queries the IDs of all fine-tuned corpora associated with that ID, inputs the content of the latest smallest editable block of the pre-training corpus and the prompt used to indicate the form of the fine-tuned corpus content into the fine-tuning model, replaces the stored original fine-tuned corpus content with the latest fine-tuned corpus content output by the model, and keeps the ID of the fine-tuned corpus unchanged, thus completing the forward synchronization from the pre-training corpus to the fine-tuned corpus.

[0088] Furthermore, fine-tuned corpora are generated using pre-trained corpora that have been cleaned and manually verified, rather than directly using raw data (similar to...). Figure 2 and Figure 1 In the case shown in the figure, the introduction of non-compliant information can be reduced, thereby further improving the quality of the fine-tuning corpus.

[0089] For ease of understanding, the corpus processing procedures provided in the various embodiments of this application are described in a comprehensive manner below. For example, Figure 7 This is a schematic diagram illustrating the application process of a corpus processing method provided in an embodiment of this application. For example... Figure 7As shown, a visualization area for synchronously displaying pre-trained corpora is embedded on the upstream data generation platform. This allows original corpus producers to simultaneously view the content-generated pre-trained corpora during the data production process, enabling visual traceability of the corpus. Simultaneously, the pre-trained corpus can be viewed during the original content generation process, allowing the original content generator to assess the correctness and accuracy of the pre-trained corpus content. Secondly, the pre-trained corpus visualization area integrates fine-tuning corpus generation capabilities. Based on the pre-trained corpus, a draft of the fine-tuned corpus is automatically generated. This draft is synchronously displayed in the corpus visualization area, allowing original corpus producers to view it and manually modify and confirm the content expression and accuracy of the fine-tuned corpus based on their own thought process, ultimately forming a high-quality, manually-generated corpus. In this example, the corpus generation method, generation process, and generation ontology have been rearranged and optimized, and a fine-tuning model has been introduced. Figure 7 The deep learning model shown reduces the human input required to generate high-quality human-generated corpora. The human creators, i.e., the original data producers, only need to write the original content on the platform. Their writing triggers an interaction between computation and the large model, generating pre-trained corpora, fine-tuning the corpora, and then displaying them.

[0090] Understandable, Figure 7 The automated processing steps shown can all be executed using a fixed-time or manually triggered mechanism. In specific applications, the corpus processing method provided in this application can fine-tune the corpus in the form of a knowledge graph. This application's embodiments process the corpus content itself; therefore, it is not limited by the storage format of the original written content, i.e., the original corpus source, and can be applied to corpus sources of various formats, as well as those such as… Figure 2 and Figure 3 The scenarios shown are where the digital writing platform (the source of the original corpus) and the data cleaning platform are independent of each other or require data format conversion.

[0091] This application also provides a corpus processing apparatus. Figure 8 This is a structural block diagram of a corpus processing device provided in an embodiment of this application. Figure 8 As shown, the device includes:

[0092] The corpus storage module is used to store the content of the original corpus according to the storage specifications and entity identifiers, and obtain one or more first storage entities, each of which has a corresponding first identifier;

[0093] The corpus association module is used to perform corpus preprocessing based on each first storage entity to obtain one or more second storage entities that correspond to different second identifiers, and to record the first association relationship between each second identifier and the first identifier of the corresponding first storage entity. Each of the one or more second storage entities is used to indicate the content of the pre-training corpus corresponding to the original corpus.

[0094] The corpus synchronization module is used to update the content of the original corpus or the pre-training corpus associated with the changed content in the first corpus according to the first association relationship when the content of the stored first corpus changes. The first corpus includes the original corpus or the pre-training corpus.

[0095] The corpus storage module, corpus association module, and corpus synchronization module can all be implemented in software or hardware. For example, the implementation of the corpus storage module will be described below. Similarly, the implementation of the corpus association module and the corpus synchronization module can refer to the implementation of the corpus storage module.

[0096] As an example of a software functional unit, a corpus storage module may include code running on computing instances. These computing instances may include at least one of a physical host (computing device), a virtual machine, or a container. Furthermore, the aforementioned computing instances may be one or more. For example, a corpus storage module may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ comprising one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0097] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0098] As an example of a hardware functional unit, a corpus storage module may include at least one computing device, such as a server. Alternatively, the corpus storage module may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The aforementioned PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0099] The corpus storage module includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the corpus storage module can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the corpus storage module can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0100] It should be noted that, in other embodiments, the corpus storage module can be used to execute any step in the corpus processing method, the corpus association module can be used to execute any step in the corpus processing method, and the corpus synchronization module can be used to execute any step in the corpus processing method. The steps implemented by the corpus storage module, the corpus association module, and the corpus synchronization module can be specified as needed. By implementing different steps in the corpus processing method through the corpus storage module, the corpus association module, and the corpus synchronization module, all functions of the corpus processing device can be realized.

[0101] This application also provides a computing device 100. Figure 9 This is a structural block diagram of a computing device provided in an embodiment of this application. For example... Figure 9 As shown, the computing device 100 includes a bus 102, a processor 104, a memory 106, and a communication interface 108. The processor 104, the memory 106, and the communication interface 108 communicate with each other via the bus 102. The computing device 100 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 100.

[0102] Bus 102 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 9 The bus 104 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 104 may include a path for transmitting information between various components of the computing device 100 (e.g., memory 106, processor 104, communication interface 108).

[0103] The processor 104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0104] Memory 106 may include volatile memory, such as random access memory (RAM). Processor 104 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0105] The memory 106 stores executable program code, and the processor 104 executes the executable program code to implement the functions of the aforementioned corpus storage module, corpus association module, and corpus synchronization module, thereby realizing the corpus processing method. That is, the memory 106 stores instructions for executing the corpus processing method.

[0106] The communication interface 103 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 100 and other devices or communication networks.

[0107] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone. For example, Figure 10 This is a structural block diagram of a computing device cluster provided in an embodiment of this application. For example... Figure 10 As shown, the computing device cluster includes at least one computing device 100. The memory 106 of one or more computing devices 100 in the computing device cluster may store the same instructions for performing corpus processing methods.

[0108] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing the corpus processing method. In other words, a combination of one or more computing devices 100 can jointly execute the instructions for executing the corpus processing method.

[0109] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform a corpus processing method, or instruct the computing device to perform a corpus processing method.

[0110] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned related steps to implement the corpus processing method in the above embodiment.

[0111] In this embodiment, the computing device, computer storage medium, or computer program product is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding method provided above, and will not be repeated here.

[0112] Any content in the various embodiments of this application, as well as any content in the same embodiment, can be freely combined. Any combination of the above content is within the scope of this application.

[0113] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0114] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A corpus processing method, characterized in that, The method includes: Based on the storage specifications and entity identifiers, the content of the original corpus is stored to obtain one or more first storage entities, each of which corresponds to a first identifier. Based on each of the first storage entities, corpus preprocessing is performed to obtain one or more second storage entities that correspond to different second identifiers, and the first association relationship between each of the different second identifiers and the first identifier of the corresponding first storage entity is recorded. Each of the one or more second storage entities is used to indicate the content of the pre-training corpus corresponding to the original corpus. If the content of the stored first corpus changes, the content of the original corpus or the pre-training corpus associated with the changed content in the first corpus is updated according to the first association relationship to correspond to the change. The first corpus includes the original corpus or the pre-training corpus.

2. The method according to claim 1, characterized in that, The original corpus includes text content, and the storage specification includes one or more editable text blocks; The process involves storing the content of the original corpus according to storage specifications and entity identifiers, resulting in one or more first storage entities. Each first storage entity corresponds to a first identifier, including: The original corpus content is divided according to the storage specifications, and the divided content is extracted to obtain multiple storage entities; Each of the plurality of storage entities is labeled with a different entity identifier, resulting in a plurality of first storage entities each having a different first identifier; The step of preprocessing the corpus based on each of the first storage entities to obtain one or more second storage entities each corresponding to a different second identifier includes: For each of the plurality of first storage entities, corpus extraction and corpus cleaning are performed to obtain a second storage entity corresponding to the first storage entity; Each second storage entity is labeled with a different second identifier, resulting in multiple second storage entities each with a different second identifier.

3. The method according to claim 2, characterized in that, In the event of a change in the content of the stored first corpus, the method of updating the content of the original corpus or the pre-training corpus associated with the changed content in the first corpus according to the first association relationship includes: In the event of a change in the content of the stored first corpus, a first identifier of the first updated storage entity whose content has been changed is determined based on the editable text block that forms the changed content, wherein the first corpus includes the original corpus; The first updated storage entity is subjected to corpus extraction and corpus cleaning to obtain a second updated storage entity corresponding to the first identifier of the first updated storage entity. Based on the first association relationship, the second storage entity marked by the second identifier corresponding to the first identifier of the first updated storage entity is updated to the second updated storage entity corresponding to the first identifier of the first updated storage entity.

4. The method according to claim 2, characterized in that, In the event of a change in the content of the stored first corpus, the method of updating the content of the original corpus or the pre-training corpus associated with the changed content in the first corpus according to the first association relationship includes: In the event of a change in the content of the stored first corpus, a second identifier of the second updated storage entity with the changed content is determined based on the editable text block that forms the changed content, wherein the first corpus includes the pre-training corpus; Based on the first association relationship, the editable text block in the second storage entity marked by the first identifier corresponding to the second identifier of the second updated storage entity is updated to the editable text block in the second updated storage entity.

5. The method according to any one of claims 1 to 4, characterized in that, After recording the first association relationship between each of the different second identifiers and the first identifier of the corresponding first storage entity, the method further includes: Based on the first association relationship, the content of the stored original corpus and the content of the corresponding pre-training corpus are output to a visualization device for display; and / or, Based on the first association relationship, the comparison information between the content of the stored original corpus and the content of the pre-training corpus corresponding to the original corpus is obtained, and the comparison information is output to the visualization device for display.

6. The method according to claim 5, characterized in that, The comparison information is used to indicate one or more of the same content, relatively deleted content, and relatively added content between the content of the stored original corpus and the pre-training corpus corresponding to the original corpus.

7. The method according to any one of claims 1 to 6, characterized in that, After recording the first association relationship between each of the different second identifiers and the first identifier of the corresponding first storage entity, the method further includes: Based on each of the one or more second storage entities, corpus annotation processing is performed to obtain one or more fine-tuned corpora with different third identifiers respectively; Record the second association relationship between each of the different third identifiers and the second identifier of the corresponding second storage entity; If the content of the stored second corpus changes, the content of the pre-training corpus or the fine-tuning corpus associated with the changed content in the second corpus is updated according to the second association relationship to correspond to the change. The second corpus includes the pre-training corpus or any one of the one or more fine-tuning corpora.

8. The method according to claim 7, characterized in that, In the event of a change in the content of the stored second corpus, the method of updating the content of the pre-training corpus or the fine-tuned corpus associated with the changed content in the second corpus according to the second association relationship includes: When the content of the stored second corpus changes, based on the changed content in the second corpus, the prompt, and the pre-trained fine-tuning model, updated content associated with the changed content in the second corpus is obtained. The prompt is used to instruct the fine-tuning model to output content that conforms to the form of the pre-trained corpus or content that conforms to the form of the fine-tuning corpus. The fine-tuning model is a model trained using sample corpus, the adjustment labels of the sample corpus, and the prompt corresponding to the adjustment target. Based on the second association relationship, the content of the pre-training corpus or the fine-tuned corpus associated with the changed content in the second corpus is updated to the updated content associated with the changed content in the second corpus.

9. A corpus processing device, characterized in that, The device includes: The corpus storage module is used to store the content of the original corpus according to the storage specifications and entity identifiers, and obtain one or more first storage entities, each of which has a corresponding first identifier; The corpus association module is used to perform corpus preprocessing based on each of the first storage entities to obtain one or more second storage entities that correspond to different second identifiers, and to record the first association relationship between each of the different second identifiers and the first identifier of the corresponding first storage entity. Each of the one or more second storage entities is used to indicate the content of the pre-training corpus corresponding to the original corpus. The corpus synchronization module is used to update the content of the original corpus or the pre-training corpus associated with the changed content in the first corpus according to the first association relationship when the content of the stored first corpus changes. The first corpus includes the original corpus or the pre-training corpus.

10. The apparatus according to claim 9, characterized in that, The original corpus includes text content, and the storage specification includes one or more editable text blocks; The corpus storage module is specifically used for: The original corpus content is divided according to the storage specifications, and the divided content is extracted to obtain multiple storage entities; Each of the plurality of storage entities is labeled with a different entity identifier, resulting in a plurality of first storage entities each having a different first identifier; The corpus association module is specifically used for: For each of the plurality of first storage entities, corpus extraction and corpus cleaning are performed to obtain a second storage entity corresponding to the first storage entity; Each second storage entity is labeled with a different second identifier, resulting in multiple second storage entities each with a different second identifier.

11. The apparatus according to claim 10, characterized in that, The corpus synchronization module is specifically used for: In the event of a change in the content of the stored first corpus, a first identifier of the first updated storage entity whose content has been changed is determined based on the editable text block that forms the changed content, wherein the first corpus includes the original corpus; The first updated storage entity is subjected to corpus extraction and corpus cleaning to obtain a second updated storage entity corresponding to the first identifier of the first updated storage entity. Based on the first association relationship, the second storage entity marked by the second identifier corresponding to the first identifier of the first updated storage entity is updated to the second updated storage entity corresponding to the first identifier of the first updated storage entity.

12. The apparatus according to claim 10, characterized in that, The corpus synchronization module is specifically used for: In the event of a change in the content of the stored first corpus, a second identifier of the second updated storage entity with the changed content is determined based on the editable text block that forms the changed content, wherein the first corpus includes the pre-training corpus; Based on the first association relationship, the editable text block in the second storage entity marked by the first identifier corresponding to the second identifier of the second updated storage entity is updated to the editable text block in the second updated storage entity.

13. The apparatus according to any one of claims 9 to 12, characterized in that, The device also includes a visualization module for: Based on the first association relationship, the content of the stored original corpus and the content of the corresponding pre-training corpus are output to a visualization device for display; and / or, Based on the first association relationship, the comparison information between the content of the stored original corpus and the content of the pre-training corpus corresponding to the original corpus is obtained, and the comparison information is output to the visualization device for display.

14. The apparatus according to claim 13, characterized in that, The comparison information is used to indicate one or more of the same content, relatively deleted content, and relatively added content between the content of the stored original corpus and the pre-training corpus corresponding to the original corpus.

15. The apparatus according to any one of claims 9 to 14, characterized in that, The corpus association module is also used for: After recording the first association relationship between each second identifier in the different second identifiers and the first identifier of the corresponding first storage entity, corpus annotation processing is performed based on each of the one or more second storage entities to obtain one or more fine-tuned corpora corresponding to different third identifiers; and the second association relationship between each third identifier in the different third identifiers and the second identifier of the corresponding second storage entity is recorded. The corpus synchronization module is also used for: If the content of the stored second corpus changes, the content of the pre-training corpus or the fine-tuning corpus associated with the changed content in the second corpus is updated according to the second association relationship to correspond to the change. The second corpus includes the pre-training corpus or any one of the one or more fine-tuning corpora.

16. The apparatus according to claim 15, characterized in that, The corpus synchronization module is specifically used for: When the content of the stored second corpus changes, based on the changed content in the second corpus, the prompt, and the pre-trained fine-tuning model, updated content associated with the changed content in the second corpus is obtained. The prompt is used to instruct the fine-tuning model to output content that conforms to the form of the pre-trained corpus or content that conforms to the form of the fine-tuning corpus. The fine-tuning model is a model trained using sample corpus, the adjustment labels of the sample corpus, and the prompt corresponding to the adjustment target. Based on the second association relationship, the content of the pre-training corpus or the fine-tuned corpus associated with the changed content in the second corpus is updated to the updated content associated with the changed content in the second corpus.

17. A computing device, characterized in that, include: Processor and memory; The processor and the memory are connected; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that, The method includes a computer program, characterized in that, when the computer program is run on a computing device, it causes the computing device to perform the method as described in any one of claims 1 to 8.

19. A chip, characterized in that, The device includes one or more interface circuits and one or more processors; the interface circuits are configured to receive signals from the memory of a computing device and send the signals to the processors, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the computing device performs the method according to any one of claims 1 to 8.

20. A computer program product, characterized in that, The method includes a computer program that, when executed by a computing device, causes the computing device to perform the method according to any one of claims 1 to 8.

Images