Text processing method and device based on large model, electronic equipment and storage medium

Abstract

The invention provides a text processing method and device based on a large model, electronic equipment and a storage medium, relates to the field of artificial intelligence, and can be used for application scenes such as generative search, intelligent document editing, intelligent assistants, virtual assistants and intelligent e-commerce. According to the specific implementation scheme, an input text is segmented into a plurality of fragments; using a first large model to complement each fragment in the plurality of fragments, and then performing predetermined processing to obtain a processed text corresponding to each fragment; and obtaining a target text based on the processing text corresponding to each fragment. According to the embodiment of the invention, the preset processing speed of the input text can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to the fields of natural language processing and large models, and can be used in application scenarios such as generative search, intelligent document editing, intelligent assistants, virtual assistants, and intelligent e-commerce. Background Art

[0002] The application of artificial intelligence technology in various fields has developed rapidly. In recent years, it has performed well in natural language processing fields such as content generation and creation, intelligent document editing, and generative dialogue. When performing tasks such as text rewriting, polishing, and translation based on artificial intelligence technology, users may enter text with a very large number of characters. It takes a long time to process text data with a very large number of characters, which may cause timeout errors and reduced user experience. Summary of the invention

[0003] The present disclosure provides a text processing method, device, electronic device and storage medium based on a large model.

[0004] According to one aspect of the present disclosure, a text processing method based on a large model is provided, comprising:

[0005] Split the input text into multiple fragments;

[0006] Using the first large model, each of the multiple fragments is completed and then processed in a predetermined manner to obtain a processed text corresponding to each fragment;

[0007] Based on the processed text corresponding to each segment, a target text is obtained.

[0008] According to another aspect of the present disclosure, a text processing device based on a large model is provided, comprising:

[0009] The fragmentation module is used to split the input text into multiple fragments;

[0010] A processing module, used to use the first large model to complete each of the multiple fragments and then perform predetermined processing to obtain a processed text corresponding to each fragment;

[0011] The summarization module is used to obtain the target text based on the processed text corresponding to each segment.

[0012] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0016] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0017] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0018] In the technical solution of the disclosed embodiment, the input text is divided into multiple segments, and the first large model is used to first complete the segments and then perform predetermined processing, so that the first large model can perform predetermined processing on the content with complete semantics, so that the method of performing predetermined processing on the segments and then aggregating to obtain the target text can replace the predetermined processing on the complete input text, thereby improving the speed of the predetermined processing on the input text, reducing the processing time, and avoiding problems such as timeout errors and reduced user experience.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0021] Figure 1 It is a flowchart of a text processing method based on a large model provided in an embodiment of the present disclosure;

[0022] Figure 2 is a schematic diagram of the processing flow of each shard in an embodiment of the present disclosure;

[0023] Figure 3 is a schematic diagram of an application example of the text processing method based on a large model according to an embodiment of the present disclosure;

[0024] Figure 4 is a schematic diagram of another application example of the text processing method based on a large model according to an embodiment of the present disclosure;

[0025] Figure 5 is a schematic diagram of another application example of the text processing method based on a large model according to an embodiment of the present disclosure;

[0026] Figure 6is a structural schematic diagram of a text processing device based on a large model according to an embodiment of the present disclosure;

[0027] Figure 7 is a structural schematic diagram of a text processing device based on a large model according to another embodiment of the present disclosure;

[0028] Figure 8 It is a block diagram of an electronic device used to implement the large model-based text processing method of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0030] In order to facilitate understanding of the technical solution of the embodiment of the present disclosure, the application scenario of the text processing method based on the large model in the embodiment of the present disclosure is first introduced below. The text processing method based on the large model in the embodiment of the present disclosure can be used to perform predetermined processing on the input text. Among them, the predetermined processing is, for example, rewriting, polishing, editing, translation, information extraction or summary summarization for long texts. For example, the predetermined processing may include rewriting, and the user may provide input text, and the electronic device executing the method in the embodiment of the present disclosure may rewrite the target text (for example, the text rewritten according to the expression in the specific application scenario) based on the input text, which has the same or similar semantics as the input text and meets the preset standards, and returns the target text to the user. Specifically, in the embodiment of the present disclosure, rewriting may be achieved by dividing the input text into multiple segments, and supplementing and rewriting each segment respectively.

[0031] Figure 1 A flowchart of a large model-based text processing method provided for an embodiment of the present disclosure. The method may be based on a large model text processing device, which may be deployed in an electronic device. The electronic device may be, for example, a single or multi-machine terminal, server or other processing device. Among them, the terminal may be a user equipment (UE) such as a mobile device, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc.; the server may be a stand-alone server or a server cluster. In some possible implementations, the method may also be implemented by a processor calling computer-readable instructions stored in a memory. For example Figure 1As shown, the method may include the following steps S110 to S130.

[0032] S110: Divide the input text into multiple segments.

[0033] A segment can be understood as a part of the input text, that is, step S110 can be understood as dividing the input text into multiple parts, each part is a segment, and the segment can also be called a segment character, a block, etc. In practical applications, the segmentation operation can also be described as slicing, cutting, splitting, slicing, etc.

[0034] Optionally, the electronic device may divide the input text into multiple segments according to punctuation characters / symbols in the input text, or may divide the input text into multiple segments based on one or more of the length of the input text, the preset number of segments, and the preset character amount threshold. For example, the character amount of each segment is determined based on the length of the input text and the allocated number, and then the input text is divided into multiple segments according to the character amount. For example, if the character amount of each segment is 10,000, the input text is divided into multiple segments containing 10,000 characters.

[0035] S120. Using the first large model, complete each of the multiple fragments and perform predetermined processing to obtain a processed text corresponding to each fragment.

[0036] Among them, the large model refers to a neural network model with a large number of parameters and complex structure, which can process massive data and complete various complex tasks. The large model may include a large language model (LLM). Optionally, the first large model may be a large model that can handle a variety of complex tasks, or a large model that has been specially trained / fine-tuned to implement the processing of shards in the embodiments of the present disclosure.

[0037] Exemplarily, for each segment, or for any segment, the segment is completed, and it can be completed based on the context and semantics in the segment. It can be understood that when the input text is segmented into multiple segments, the sentences and words at the beginning and end of each segment may be truncated, thereby affecting the effect of the predetermined processing on the single segment. In the embodiment of the present disclosure, the segment is completed first and then the predetermined processing is performed, which can ensure the effect of executing the predetermined processing, thereby supporting the predetermined processing after the input text is segmented into segments.

[0038] Optionally, the first large model can be a large model for processing text based on the input segmented output, that is, the algorithm processing process of the large model is equivalent to completing the segmented and performing predetermined processing. Alternatively, the first large model can include two large models, one of which is used to complete the segmented, and the other is used to perform predetermined processing on the completed segmented; optionally, the two large models can be trained / fine-tuned separately using one or more pre-collected corpora or text data sets to improve the processing effect on the segmented.

[0039] Optionally, the predetermined processing may include rewriting, polishing, editing, translation, information extraction, or summarization and other processing for long texts.

[0040] S130: Obtain target text based on the processed text corresponding to each segment.

[0041] In one implementation, the processed texts can be spliced ​​in sequence according to the order of the segments, and the spliced ​​text is used as the target text. In another implementation, after the processed texts are spliced ​​in sequence, the spliced ​​text can be further processed, and the further processed text is used as the target text.

[0042] Optionally, the further processing may include rewriting, polishing, etc., that is, rewriting and polishing the text obtained by splicing multiple processed texts to obtain a target text. The further processing can be implemented by other models, such as a smaller and more efficient model, so as to improve the processing effect while controlling the processing time of long texts.

[0043] Optionally, the further processing may also include formatting, packaging, and the like of the spliced ​​text, which is not limited in this embodiment of the present disclosure.

[0044] The above method divides the input text into multiple segments, and uses the first large model to first complete the segments and then perform predetermined processing, so that the first large model can perform predetermined processing on content with complete semantics, so that the method of performing predetermined processing on segments and then aggregating to obtain the target text can replace the predetermined processing on the complete input text, thereby improving the speed of predetermined processing on the input text, reducing processing time, and avoiding problems such as timeout errors and decreased user experience.

[0045] Figure 2 FIG. 1 is a schematic diagram of the processing flow of each shard in the embodiment of the present disclosure. Figure 2 As shown, in some embodiments, the above step S120, using the first large model, completes each of the multiple slices and performs predetermined processing to obtain a processed text corresponding to each slice, including:

[0046] S210, splicing the to-be-processed slice among the multiple slices with at least one prompt word (prompt) to obtain a splicing result; wherein the prompt word is used to guide the first large model to perform a predetermined process after completing the to-be-processed slice;

[0047] S220: Input the splicing result into the first large model to obtain the processing text corresponding to the fragment to be processed.

[0048] The to-be-processed fragment is any one of the multiple fragments. Optionally, the fragments may be extracted one by one, and the extracted fragments may be used as the to-be-processed fragments, and the above-mentioned splicing and model processing may be performed to obtain the processed text, and then the next fragment may be extracted as a new to-be-processed fragment.

[0049] Optionally, the above prompt can also be used to prompt the first large model of the current processing role and scene requirements and task requirements. For example, if the predetermined processing is to rewrite the input text according to the preset standard A, the prompt words spliced ​​with the to-be-processed segment can be as follows:

[0050] "You will be given a piece of text (the original text), and you need to rewrite it and output a new text. If you think the original text does not make sense, the new text should be the same as the original. If it does make sense, your rewrite needs to follow these rules:

[0051] Some words may be truncated at the beginning and end of the text, so you may need to complete them based on context and semantics before rewriting.

[0052] Every sentence of the entire article needs to be rewritten. You must make sure it is similar to the previous expression, but the vocabulary and sentence expression need to be changed.

[0053] You can also choose to rewrite several sentences at the same time, revise them together, and change the order of expression, but the meaning expressed must be similar to the original sentence.

[0054] You must also ensure that your rewrite meets Standard A.

[0055] The length (number of words) of the new text you output must be close to the length (number of words) of the original text, without too much fluctuation.

[0056] Sometimes the original text may be very short and you need to give the new text based on the original text.

[0057] If the original text is empty, then the new text should also be empty."

[0058] According to the above embodiment, after the slice to be processed is spliced ​​with the prompt, the splicing result is input into the first large model, and the prompt guides the first large model to complete the slice to be processed and perform the predetermined processing, that is, the algorithm processing process of the first large model is equivalent to completing the slice and performing the predetermined processing. This method can ensure the effect of the predetermined processing of the slice, and there is no need to train / fine-tune the first large model additionally, which can reduce resource consumption.

[0059] In some embodiments, the at least one prompt word may include a custom prompt word and / or a role prompt word. The custom prompt word may be a prompt word input by a target object, and the target object may refer to a user or a developer / maintainer of a text processing device, etc. The custom prompt word may be used to describe scene requirements, task requirements, etc., thereby improving the flexibility of text processing, so that the method of the disclosed embodiment can achieve better results in a variety of scenes and tasks. The role prompt word may be a pre-configured prompt word, which is used to prompt the first large model to complete the input to-be-processed fragments and then perform predetermined processing.

[0060] Figure 3 FIG. 1 is a schematic diagram showing an application example of the text processing method based on a large model according to an embodiment of the present disclosure. Figure 3 As shown, this application example includes:

[0061] S310, dividing the input text into multiple segments, including segment 1, segment 2, segment n, etc.

[0062] S320. For each fragment, concatenate it with the custom prompt and the role prompt and input it into the first large model to obtain the processed text corresponding to the fragment output by the first large model after processing.

[0063] S330, returning the target text composed of the processed texts to the user in an SSE manner. Among them, SSE (Server-Sent Events) is a streaming output technology that presents the text to the user character by character, thereby providing a progressive presentation method, optimizing the conversation experience, and helping the user to track the model's thinking process and see how it gradually builds the answer, which helps to improve the transparency and explainability of the conversation.

[0064] In some embodiments, the at least one prompt word includes a role prompt word;

[0065] The method also includes:

[0066] Get the custom prompt word entered by the target object;

[0067] The custom prompt words are rewritten using the second largest model to obtain the role prompt words; wherein the second largest model is obtained by training with multiple texts related to the predetermined processing.

[0068] For example, the target object (such as a user, developer, maintainer, etc.) can input a custom prompt word to describe the task requirements, scene requirements, etc. The custom prompt word is rewritten by the second largest model to generate a role prompt word, and the role prompt word is used to splice with the slice to be processed. Among them, the second largest model can be a model specially trained for the predetermined processing and can more accurately express the task requirements and scene requirements of the predetermined processing.

[0069] According to the above embodiment, the custom prompt word input by the target object is rewritten to obtain the role prompt word, which can standardize the standard and form of the prompt word, thereby improving the guiding effect of the first large model, and further improving the effect of the predetermined processing of the input text.

[0070] In some embodiments, the text processing method based on the large model may further include:

[0071] A message queue corresponding to the input text is created, and multiple fragments are sequentially stored in the message queue; wherein the fragments in the message queue are sequentially extracted by consumers, and the consumers use the first large model to complete the extracted fragments and then perform predetermined processing.

[0072] Accordingly, based on the processed text corresponding to each fragment, the target text is obtained, including:

[0073] When there are no shards in the message queue, the singleton mode is used to encapsulate the processing text corresponding to each shard to obtain the target text.

[0074] The above embodiment uses a message queue to implement sequential processing of each shard and singleton mode encapsulation. Figure 4 This is a schematic diagram of an application example. This application example takes the predetermined processing as text rewriting as an example to present the above processing flow. Figure 4As shown, when the input text is sliced, multiple slices (slice 1, slice 2, slice 3, and slice n, etc.) can be used as producers, and a message queue corresponding to the input text is created, and each slice is placed in the message queue one by one. When there are a specified number of slices in the queue, the producer no longer pushes the slices to the message queue, and the consumers (including the module of the extraction task and the text rewriting module) extract the slices for completion and rewriting. The rewritten output processed text will be added to the SSE stream message and enter the singleton mode for encapsulation. After rewriting a slice, the text rewriting module notifies the module of the extraction task to consume the next slice, and this goes back and forth until the processing of the last slice is completed. When all slices are rewritten, the target text is encapsulated by the singleton mode, and SSE transmission is performed to return the target text to the user.

[0075] It can be understood that, using the above method, when the user provides another input text, a message queue corresponding to the new input text will be created and similar operations will be performed. That is, each fragment of the same input text will be processed by the message queue to ensure that the predetermined processing is executed in sequence, and finally encapsulated in a singleton mode. When the user enters different input texts at the same time, there will be no situation where the fragments of different input texts are interspersed. In this way, while effectively speeding up the speed of the predetermined processing, the consistency of each processed text is maintained, thereby ensuring the quality of the target text.

[0076] In some embodiments, a design to optimize user experience may be added to the above mechanism. Figure 4 As shown, when the first shard enters the message queue, the user can be notified that it is loading. When the rewrite is completed, the user can be notified that the task is completed, so that the user can understand the progress of the work. For example, if the user chooses to clear the text during the work process and cancels the request to process the input text, a pop-up window asking whether to clear the text can be displayed on the interactive interface. If the user chooses not to prompt again, the timestamp is recorded and saved, and the pop-up window will not be popped up repeatedly within a predetermined period of time. For another example, it can support synchronizing the output target text to the editor in the interactive interface for reprocessing to ensure the integrity of the user experience.

[0077] In some embodiments, the input text is segmented into a plurality of segments, including:

[0078] Based on the preset character count threshold, the input text is divided into multiple segments.

[0079] The preset number of characters is, for example, 10,000 or 20,000.

[0080] The above embodiment can ensure that the processing speed is effectively improved by presetting a fixed character quantity threshold and dividing the input text into segments whose character quantity does not exceed the threshold.

[0081] In some embodiments, the input text may be limited in number of words. Exemplarily, the word limit may be based on the number of characters or words. If the word limit is based on the number of words, the word count may be calculated according to the following rules: 1. The word count is calculated starting from the first input character; 2. When a space or line break appears, a new character is counted as a new word; except for spaces and line breaks, other special characters and punctuation marks are not counted as additional words; multiple spaces, multiple line breaks, and a combination of line breaks and spaces are only counted as one word. Using the number of words to limit the word count can better adapt to the predetermined processing tasks in most scenarios.

[0082] In some embodiments, the text processing method based on the large model further includes:

[0083] Based on the proportion of each language in the to-be-processed shards among the multiple shards, a language adopted by the processed text corresponding to the to-be-processed shards is determined.

[0084] Optionally, the language with the largest share may be used as the language used to process the text.

[0085] Exemplarily, when the segment to be processed is all in English, the processed text output by the first largest model is determined to be English; when the segment to be processed is all in Chinese, the processed text output by the first largest model is determined to be Chinese; when 90% of the content in the processing segment is in Chinese and 10% of the content is in English, the processed text output by the first largest model is determined to be Chinese.

[0086] Optionally, after determining the language used for processing the text, a restriction on the language may be added to the prompt word, or the above-mentioned language determination rule may be added to the prompt word so that the first large model outputs the processed text according to the above-mentioned rule.

[0087] According to the above embodiment, the language of the processed text is determined according to the proportion of each language in the segment, and different languages ​​can be supported for predetermined processing, thereby improving flexibility.

[0088] In some embodiments, the text processing method based on the large model further includes:

[0089] Render the target text based on multiple rendering plug-ins in the multimodal rendering engine;

[0090] Output the rendered target text through the interactive interface.

[0091] Exemplarily, the plurality of rendering plug-ins maliciously include one or more of a paragraph plug-in, a list plug-in, a table plug-in, a reference plug-in, a formula plug-in, a hyperlink plug-in, a code block plug-in, a data chart plug-in, a flowchart plug-in, and a mind map plug-in.

[0092] According to the above embodiment, a multimodal rendering engine can be used to render the target text, thereby improving the display effect of the target text and optimizing the user experience.

[0093] Figure 5 is a schematic diagram of an application example. Figure 5 As shown, a text processing device is deployed in an electronic device, which can provide services for various AIGC (Artificial Intelligence Generated Content) scenarios, such as intelligent creation, rewriting, summarization, intelligent insight, etc. Corresponding services are provided based on AIGC data services, AIGC business plug-ins and AIGC multimodal rendering engines. Among them, the AIGC data service can access the AIGC data server and output text in markdown format. AIGC business plug-ins include creation plug-ins, insight plug-ins and plug-ins for other scenarios. The AIGC multimodal rendering engine completes the rendering processing of the output text through core layer capabilities, external dependency base libraries, multimodal rendering plug-ins and interface layers.

[0094] Figure 6 is a schematic diagram of the structure of a text processing device based on a large model according to an embodiment of the present disclosure. Figure 6 As shown, the device comprises:

[0095] Segmentation module 610, used to segment the input text into multiple segments;

[0096] A processing module 620 is used to use the first large model to complete each of the multiple fragments and then perform a predetermined process to obtain a processed text corresponding to each fragment;

[0097] The summarizing module 630 is used to obtain the target text based on the processed text corresponding to each segment.

[0098] In some embodiments, the processing module 620 is used to:

[0099] Splicing the to-be-processed slices among the multiple slices with at least one prompt word to obtain a splicing result; wherein the prompt word is used to guide the first large model to perform a predetermined process after completing the to-be-processed slices;

[0100] The splicing result is input into the first large model to obtain the processing text corresponding to the slice to be processed.

[0101] In some embodiments, at least one prompt word includes a role prompt word. Figure 7 As shown, the text processing device also includes:

[0102] The prompt word acquisition module 710 is used to acquire the custom prompt word input by the target object;

[0103] The prompt word rewriting module 720 is used to rewrite the custom prompt word using the second largest model to obtain the role prompt word; wherein the second largest model is obtained by training with multiple texts related to the predetermined processing.

[0104] In some embodiments, Figure 7 As shown, the text processing device also includes:

[0105] The queue processing module 730 is used to create a message queue corresponding to the input text, and store multiple fragments in the message queue in sequence; wherein the fragments in the message queue are extracted in sequence by consumers, and the consumers use the first large model to complete the extracted fragments and then perform predetermined processing.

[0106] Accordingly, the aggregation module is used to encapsulate the processing text corresponding to each shard in a singleton mode to obtain the target text when no shard exists in the message queue.

[0107] In some embodiments, the sharding module 610 is used to:

[0108] Based on the preset character count threshold, the input text is divided into multiple segments.

[0109] In some embodiments, Figure 7 As shown, the text processing device also includes:

[0110] The language determination module 740 is used to determine the language used for processing text corresponding to the to-be-processed segment based on the proportion of each language in the to-be-processed segment among the multiple segments.

[0111] In some embodiments, Figure 7 As shown, the text processing device also includes:

[0112] A rendering module 750, configured to render the target text based on multiple rendering plug-ins in the multimodal rendering engine;

[0113] The output module 760 is used to output the rendered target text through an interactive interface.

[0114] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0115] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0116] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0117] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0118] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0119] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0120] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as a text processing method based on a large model. For example, in some embodiments, the text processing method based on a large model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the text processing method based on the large model described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the large model-based text processing method in any other appropriate manner (eg, by means of firmware).

[0121] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0122] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0123] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0124] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0125] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0126] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0127] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0128] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A text processing method based on a large model, comprising: Split the input text into multiple segments; Using the first large model, each of the multiple fragments is completed and then processed in a predetermined manner to obtain a processed text corresponding to each fragment; Based on the processed text corresponding to each segment, a target text is obtained.

2. The method according to claim 1, wherein: The method of using the first large model to complete each of the multiple fragments and then perform predetermined processing to obtain a processed text corresponding to each fragment includes: Splicing the to-be-processed slice among the multiple slices with at least one prompt word to obtain a splicing result; wherein the prompt word is used to guide the first large model to complete the to-be-processed slice and then perform a predetermined process; The splicing result is input into the first large model to obtain the processing text corresponding to the to-be-processed segment.

3. The method according to claim 2, wherein: The at least one prompt word includes a role prompt word; The method further comprises: Get the custom prompt word entered by the target object; The custom prompt word is rewritten using the second largest model to obtain the role prompt word; wherein the second largest model is obtained by training with multiple texts related to the predetermined process.

4. The method according to any one of claims 1 to 3, further comprising: Creating a message queue corresponding to the input text, and sequentially storing the multiple fragments in the message queue; wherein the fragments in the message queue are sequentially extracted by consumers, and the consumers are used to complete the extracted fragments using the first large model and then perform predetermined processing; The step of obtaining a target text based on the processed text corresponding to each of the fragments includes: When there is no shard in the message queue, the processing text corresponding to each shard is encapsulated in a singleton mode to obtain the target text.

5. The method according to any one of claims 1 to 4, wherein: The step of dividing the input text into multiple segments includes: Based on a preset character quantity threshold, the input text is divided into a plurality of segments.

6. The method according to any one of claims 1 to 5, further comprising: Based on the proportion of each language in the to-be-processed segment among the multiple segments, a language adopted by the processed text corresponding to the to-be-processed segment is determined.

7. The method according to any one of claims 1 to 6, further comprising: Rendering the target text based on multiple rendering plug-ins in a multimodal rendering engine; The rendered target text is outputted through an interactive interface.

8. A text processing device based on a large model, comprising: The fragmentation module is used to split the input text into multiple fragments; A processing module, used to use the first large model to complete each of the multiple fragments and then perform predetermined processing to obtain a processed text corresponding to each fragment; The summarization module is used to obtain the target text based on the processed text corresponding to each segment.

9. The device according to claim 8, wherein: The processing module is used for: Splicing the to-be-processed slice among the multiple slices with at least one prompt word to obtain a splicing result; wherein the prompt word is used to guide the first large model to complete the to-be-processed slice and then perform a predetermined process; The splicing result is input into the first large model to obtain the processing text corresponding to the to-be-processed segment.

10. The device according to claim 9, wherein: The at least one prompt word includes a role prompt word; The device also includes: A prompt word acquisition module is used to obtain the custom prompt word input by the target object; The prompt word rewriting module is used to rewrite the custom prompt word using the second largest model to obtain the role prompt word; wherein the second largest model is obtained by training with multiple texts related to the predetermined processing.

11. The device according to any one of claims 8 to 10, further comprising: A queue processing module, used for creating a message queue corresponding to the input text, and storing the plurality of fragments in the message queue in sequence; wherein the fragments in the message queue are sequentially extracted by consumers, and the consumers are used for completing the extracted fragments using the first large model and then performing predetermined processing; The aggregation module is used to: When there is no shard in the message queue, the processing text corresponding to each shard is encapsulated in a singleton mode to obtain the target text.

12. The device according to any one of claims 8 to 11, wherein: The sharding module is used to: Based on a preset character quantity threshold, the input text is divided into a plurality of segments.

13. The device according to any one of claims 8 to 12, further comprising: The language determination module is used to determine the language used for processing text corresponding to the to-be-processed segment based on the proportion of each language in the to-be-processed segment among the multiple segments.

14. The apparatus according to any one of claims 8 to 13, further comprising: A rendering module, used for rendering the target text based on multiple rendering plug-ins in a multimodal rendering engine; The output module is used to output the rendered target text through an interactive interface.

15. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

17. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images