Large model streaming writing auxiliary method and system based on Redis and finite-state machine
By combining Redis with a finite state machine, a large-scale streaming writing assistance method was developed, which solved the problems of content generation delay, citation tag parsing difficulties, and insufficient reference management in writing assistance systems. This enabled real-time writing and efficient citation management, improving user experience and writing efficiency.
Patent Information
- Application Number
- CN202511651021.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-12
- Publication Date
- 2026-02-27
AI Technical Summary
Existing writing assistance systems suffer from problems such as independent content generation and reference management, significant delays in long text generation, weak ability to maintain task status and session context, difficulty in real-time processing of citation tag parsing, and insufficient efficiency and accuracy in reference deduplication.
A large-scale streaming writing assistance method based on Redis and finite state machines is adopted. By generating streaming content, parsing citation tags in real time, managing references and persisting task states, and combining a three-stage deduplication algorithm, the method achieves real-time parsing and efficient management of citation tags.
It enables real-time content generation, dynamic parsing of citation tags, precise management of references, and stable storage of task status during long text writing, thereby improving writing efficiency, citation accuracy, and user experience.
Smart Images
Figure CN121580971A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of natural language processing and data storage technology, and in particular to a large-scale streaming writing assistance method and system based on Redis and finite state machines. Background Technology
[0002] With the rapid iteration of artificial intelligence technology, Large Language Models (LLMs) are increasingly widely used in text generation, providing core technological support for the development of writing assistance tools. Currently, automatic text generation systems have gradually penetrated multiple fields such as academic research, enterprise office work, and government processing, becoming an important tool for improving writing efficiency. However, existing writing assistance systems still have many technical shortcomings in practical applications, making it difficult to meet users' needs for efficient, accurate, and real-time writing:
[0003] Content generation and reference management are independent processes, and the citation format processing workflow is cumbersome. Existing systems typically generate the full text first, then require users to manually add reference citations, or rely on batch processing tools for format conversion. Because different academic journals and institutions have varying requirements for citation formats, manual proofreading and adjustment are time-consuming and laborious, and prone to errors such as incorrect citation numbers and inconsistent formats.
[0004] Long text generation suffers from significant delays, resulting in a poor user experience. When large models generate texts exceeding 10,000 words, they must complete full content calculations before returning results to the user, often taking several minutes or even longer. During this process, users cannot view the generation progress or the initial content outline in real time, making it difficult to adjust their writing direction based on intermediate results, which can easily lead to interruptions in the creative process.
[0005] The system suffers from weak capabilities in maintaining task status and session context. In multi-round writing scenarios, existing systems rely heavily on memory storage for task status, session history, and reference data, lacking a reliable persistence mechanism. When the system crashes, restarts, or experiences network interruptions, generated intermediate results, citation relationships, and writing progress are easily lost, requiring users to re-initiate tasks, severely impacting writing continuity.
[0006] Efficiency and accuracy in reference deduplication are difficult to balance. In academic writing, references come from a wide range of sources and are numerous. Existing systems often employ a single deduplication strategy, which fails to effectively identify semantically similar references. This results in either incomplete deduplication, leading to a large number of redundant citations, or excessive filtering, missing core and relevant literature, thus affecting writing quality.
[0007] There is a technical conflict between streaming generation and citation tag parsing. Existing streaming output systems can only push text fragments in real time and cannot parse structured citation tags embedded in the text stream in real time. If the citation tags are parsed after the entire text is generated, the real-time advantage of streaming output is lost; if the parsing is performed during streaming transmission, problems such as difficulty in identifying tag boundaries and complexity in dynamically adjusting citation numbers arise. Summary of the Invention
[0008] To address the problems in existing technologies, this invention provides a large-scale streaming writing assistance method and system based on Redis and finite state machines. This method solves the technical problems of separating content generation and reference management, difficulty in simultaneously handling long text streaming output and real-time citation tag parsing, unreliable task state maintenance, and insufficient reference deduplication accuracy in existing technologies. It enables real-time content generation, dynamic citation tag parsing, accurate reference management, and stable storage of task states during long text writing.
[0009] To achieve the above objectives, the present invention provides the following technical solution:
[0010] In one aspect, this invention proposes a large-scale streaming writing assistance method based on Redis and finite state machines, the method comprising:
[0011] Based on the content generation prompts, a large language model is invoked to generate streaming content and output a text stream.
[0012] The finite state machine is used to parse the citation markers in the text stream in real time.
[0013] Based on the citation results obtained from the analysis, the newly cited references are determined, and the newly cited references are deduplicated, numbered, and formatted.
[0014] The server sends an event protocol to push the parsed text stream to the client in real time.
[0015] Dynamically update and manage the reference list; after the current chapter is generated, persist the updated session history and reference status to the Redis cache database.
[0016] Optionally, before calling the large language model to generate streaming content based on the content generation prompt information and outputting the text stream, the method further includes: receiving writing task information, generating a globally unique task ID, and persisting the task status and initial data to a Redis cache database; wherein, the writing task information includes: writing requirements, article outline, and initial reference list;
[0017] Based on the writing task information, a parallel search process is initiated to obtain relevant reference data, integrate writing requirements, current chapter outline and relevant references, and construct content generation prompt information.
[0018] The task status persisted to the Redis cache database includes: task ID, writing requirements, current generation progress, and initial reference list;
[0019] The initial data persisted to the Redis cache database includes the original data corresponding to the writing requirements, article outline, and initial reference list.
[0020] Optionally, the relevant reference data is acquired based on a pre-set timeout mechanism for reference data; the timeout mechanism includes: if the reference data acquisition is not completed within the timeout threshold, the search is stopped, and the initial reference list is used to generate subsequent content.
[0021] Optionally, the real-time parsing of reference markers in the text stream using a finite state machine includes: the finite state machine performing state transitions including normal state, tag open state, tag content state, numeric content state, and tag closed state according to a predefined trigger character, so as to complete the identification, content extraction, and storage of reference markers;
[0022] The predefined trigger characters include: characters that trigger the transition from the normal state to the tag open state, character sequences that trigger the transition from the tag open state to the tag content state, characters that trigger the transition from the tag content state to the numeric content state, characters that trigger the transition from the numeric content state to the tag closed state, and character sequences that trigger the transition from the tag closed state to the normal state.
[0023] Optionally, the deduplication process for newly cited references includes:
[0024] Calculate the Hamming distance between the SimHash fingerprint of the document to be added and the SimHash fingerprint of the document already in the cache database, determine the magnitude of the Hamming distance, and filter the documents to be added that are within the preset Hamming distance threshold range as suspected duplicate candidate documents.
[0025] Calculate the TF-IDF vector cosine similarity between the document to be added and the suspected duplicate candidate documents, and filter the suspected duplicate candidate documents whose TF-IDF vector cosine similarity is within the first similarity threshold as high probability duplicate documents;
[0026] Calculate the cosine similarity of the semantic embedding vectors of the document to be added and the documents with high probability of duplication. Select the documents with high probability of duplication whose semantic embedding vector cosine similarity is within the second similarity threshold as the duplicate documents.
[0027] Optionally, the text stream data structure pushed to the client in real time includes: content type identifier, current chapter identifier, text fragments, and cumulative text content.
[0028] Optionally, it also includes:
[0029] In response to the request to generate a bibliography section, all cited references are sorted according to their display numbers, formatted and output according to a preset format, and the Redis cache data corresponding to the writing task is cleared.
[0030] In another aspect, the present invention proposes a large-model streaming writing assistance system based on Redis and finite state machines, for implementing the method described in any one of the first aspects, comprising:
[0031] The content generation module is used to call the large language model to generate streaming content based on the content generation prompt information and output a text stream;
[0032] The reference resolution module is used to parse reference markers in a text stream in real time using a finite state machine;
[0033] The reference management module is used to determine newly cited references based on the citation results obtained from the parsing, and to perform deduplication, numbering, and formatting of the newly cited references.
[0034] The streaming output module is used to push the parsed text stream to the client in real time using a server-sending event protocol;
[0035] The dynamic update module is used to dynamically update and manage the reference list; after the current chapter is generated, the updated session history and reference status are persisted to the Redis cache database.
[0036] A third aspect of the present invention provides an electronic device including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the large-model streaming writing assistance method based on Redis and finite state machines as described in any one of the first aspects.
[0037] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the large-model streaming writing assistance method based on Redis and finite state machines as described in the first aspect.
[0038] The present invention has the following beneficial effects:
[0039] This invention proposes a large-scale streaming writing assistance method and system based on Redis and finite state machines. Leveraging the character-by-character parsing capability of finite state machines, it completes citation mark recognition and number extraction simultaneously with the large model's output text stream, eliminating the need to wait for full text generation. Users can check the citation format in real time, enhancing the streaming interactive experience. It achieves synchronous processing of large-scale streaming text generation and citation mark parsing, avoiding real-time failures caused by "parsing after generation."
[0040] Furthermore, it defines a standardized streaming data structure to ensure that the client can accurately render text content and chapter structure;
[0041] This invention utilizes Redis persistent storage. Task IDs, writing progress, session history, and reference data are all stored in a distributed cache. Even in the event of a server restart or network interruption, the entire state can be quickly restored using the task ID, ensuring writing continuity. Persistent storage of task status, session history, and reference data is achieved through Redis caching, preventing data loss due to unexpected interruptions.
[0042] This invention employs a literature retrieval mechanism with timeout control and a three-stage progressive deduplication algorithm to improve the efficiency and accuracy of reference management. The timeout control mechanism avoids process stagnation caused by literature retrieval. The three-stage deduplication algorithm (SimHash coarse screening → TF-IDF judgment → semantic confirmation) balances speed and accuracy. It can exclude dissimilar documents in the coarse screening stage and accurately identify semantically duplicated documents in the subsequent stages. Unlike single deduplication methods with slower processing speed, this invention greatly improves the deduplication accuracy.
[0043] This invention achieves efficient content generation and reference management by integrating a large model, Redis cached data, finite state automata, streaming output mechanism, and reference manager. The system can parse citation tags in generated content in real time, dynamically maintain the reference list, and return results instantly via streaming, significantly improving the efficiency, accuracy, and user experience of long text writing.
[0044] The system of this invention adopts a modular design, supporting flexible replacement of core components (caching, transmission protocol) and improving system scalability. Specifically, the modular design allows for flexible replacement of core components; the cache management module can be replaced from Redis to Memcached, and the streaming transmission protocol can be replaced from SSE to WebSocket (supporting bidirectional interaction), without reconstructing the overall architecture, adapting to the technical requirements of different scenarios. Attached Figure Description
[0045] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0046] Figure 1 This is a flowchart illustrating the steps of a large-scale streaming writing assistance method based on Redis and finite state machines proposed in this invention.
[0047] Figure 2 This invention presents a flow diagram illustrating the execution process of a large-scale streaming writing assistance method based on Redis and finite state machines.
[0048] Figure 3 The flowchart of the application of the large-model streaming writing assistance method based on Redis and finite state machine proposed in this invention is shown below.
[0049] Figure 4 This is a flowchart of the reference deduplication mechanism proposed in this invention;
[0050] Figure 5 This is a block diagram of a large-scale streaming writing assistance system based on Redis and finite state machines proposed in this invention;
[0051] Figure 6 This is a schematic diagram of the structure of an electronic device used to implement the methods and system embodiments of this application. Detailed Implementation
[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0053] Example 1: As Figure 1 As shown, the first embodiment of this invention proposes a large-model streaming writing assistance method based on Redis and finite state machines. This method integrates Redis's state persistence capabilities, finite state machine's real-time parsing capabilities, large-model streaming generation capabilities, and a multi-stage progressive deduplication algorithm into a unified writing assistance scheme. It achieves simultaneous progress in content generation, citation resolution, and document management, filling the gap in existing technologies for citation processing and state maintenance in streaming scenarios. The method specifically includes the following steps:
[0054] Step S101: Based on the content generation prompt information, call the large language model to generate streaming content and output a text stream;
[0055] Step S102: Parse the citation markers in the text stream in real time using a finite state machine;
[0056] Step S103: Based on the citation results obtained from the parsing, determine the newly cited references, and perform deduplication, numbering, and formatting on the newly cited references;
[0057] Step S104: Using the server-sending event protocol, push the parsed text stream to the client in real time;
[0058] Step S105: Dynamically update and manage the reference list; after the current chapter is generated, persist the updated session history and reference status to the Redis cache database.
[0059] In step S101 above, before calling the large language model to generate streaming content based on the content generation prompt information and outputting the text stream, the method further includes: receiving writing task information, generating a globally unique task ID, and persisting the task status and initial data to the Redis cache database; wherein, the writing task information includes: writing requirements, article outline, and initial reference list;
[0060] Based on the writing task information, a parallel search process is initiated to obtain relevant reference data, integrate writing requirements, current chapter outline and relevant references, and construct content generation prompt information.
[0061] The task status persisted to the Redis cache database includes: task ID, writing requirements, current generation progress, and initial reference list;
[0062] The initial data persisted to the Redis cache database includes the original data corresponding to the writing requirements, article outline, and initial reference list.
[0063] In the above embodiments, the relevant reference data is acquired based on a pre-set timeout mechanism for reference data; the timeout mechanism for reference data includes: if the acquisition of reference data is not completed within the timeout threshold, the retrieval is stopped, and the initial reference list is used to generate subsequent content.
[0064] In step S102 above, the real-time parsing of reference markers in the text stream using a finite state machine includes: the finite state machine performing state transitions including normal state, tag open state, tag content state, numeric content state, and tag closed state according to a predefined trigger character, so as to complete the identification, content extraction, and storage of reference markers;
[0065] The predefined trigger characters include: characters that trigger the transition from the normal state to the tag open state, character sequences that trigger the transition from the tag open state to the tag content state, characters that trigger the transition from the tag content state to the numeric content state, characters that trigger the transition from the numeric content state to the tag closed state, and character sequences that trigger the transition from the tag closed state to the normal state.
[0066] In step S103 above, the deduplication process for newly cited references includes:
[0067] Calculate the Hamming distance between the SimHash fingerprint of the document to be added and the SimHash fingerprint of the document already in the cache database, determine the magnitude of the Hamming distance, and filter the documents to be added that are within the preset Hamming distance threshold range as suspected duplicate candidate documents.
[0068] Calculate the TF-IDF vector cosine similarity between the document to be added and the suspected duplicate candidate documents, and filter the suspected duplicate candidate documents whose TF-IDF vector cosine similarity is within the first similarity threshold as high probability duplicate documents;
[0069] Calculate the cosine similarity of the semantic embedding vectors of the document to be added and the documents with high probability of duplication. Select the documents with high probability of duplication whose semantic embedding vector cosine similarity is within the second similarity threshold as the duplicate documents.
[0070] In step S104 above, the text stream data structure pushed to the client in real time includes: content type identifier, current chapter identifier, text fragments, and cumulative text content.
[0071] It also includes: responding to the reference chapter generation request, sorting all cited references according to their display numbers, formatting the output according to a preset format, and clearing the Redis cache data corresponding to the writing task.
[0072] To further illustrate the overall process of the method in Embodiment 1 above, as follows: Figure 2 As shown, this is achieved through the following steps:
[0073] S1: Task initialization. The system receives the writing requirements, article outline, and initial reference list input by the user through the client, then generates a globally unique task ID, and persists the basic information of the task (including ID, status, writing requirements, initial references, etc.) to the Redis cache database.
[0074] S2: Parallel reference search. Starts a parallel search process based on writing needs and outline titles, retrieves relevant references and caches them in Redis, sets a search timeout mechanism (60 seconds), and abandons the search after the timeout to use the initial references for subsequent steps.
[0075] S3: Construct prompt information. For the chapter being generated, obtain the writing requirements, the current chapter outline, and relevant references, and integrate this information into a structured content prompt, ready to be sent to the large language model.
[0076] S4: Streaming content generation and citation processing. The prompt information constructed in S3 is input into the large language model, which generates streaming response content. Each returned text fragment is processed in real time using a finite state machine, parsing and transforming the citation markers, and updating the reference mapping relationship. This step is the core of this invention, and its specific implementation is detailed in Section 2.
[0077] S5: Streaming transmission, using the Server Send Event (SSE) protocol, pushes text fragments processed in step S4 to the client in real time. The pushed data structure is encapsulated, including content type identifier, current chapter identifier, the text fragment itself, and the cumulative text content up to the current time, to support smooth rendering and display on the client.
[0078] S6: Dynamic Reference Management. During content generation, the reference management module is activated whenever a new reference is cited (parsed in step S4). This module first performs deduplication on the new reference; if it is a unique reference, it assigns it a unique display number and maintains the mapping relationship between it and the original reference identifier. Simultaneously, it updates the reference list and mapping table in Redis.
[0079] S7: State persistence. After the current chapter is generated, the updated session history and reference status, and other intermediate data are persisted to Redis again to ensure that the task state is saved.
[0080] S8: Task Completion. Upon receiving a request to generate the "References" section, the reference management module sorts all cited references according to their display numbers and generates a formatted reference list, returning it to the client. Finally, it clears the Redis cache associated with this task and releases resources.
[0081] 2. Specific implementation of finite state machine
[0082] The finite state machine module is the core parsing engine of the system, a finite state machine specifically designed for real-time streaming text parsing. It is responsible for accurately identifying and extracting citation markers embedded in the text during the text generation process (e.g., ...).<cite_symbol> [.....]< / cite_symbol> This process triggers the transition of the citation number. The state machine, through clearly defined states and character-based transition conditions, parses the citation mark results character by character, avoiding the latency and complexity associated with concatenating the streaming output into complete text before batch processing.
[0083] 2.1 State Definition
[0084] The state machine contains five core states:
[0085] (1) Normal state: This is the initial default state. In this state, the input plain text characters are directly output, and the system continuously monitors the input stream to find the start of the reference mark.
[0086] (2) Tag-on state: This state is entered when the character < is detected in the normal state. This state is responsible for identifying and confirming the starting tag of the reference mark.<cite_symbol> .
[0087] (3) Tag content status: When the complete starting tag is successfully recognized while the tag is in the open state.<cite_symbol> Then it enters this state. This state indicates that the system has entered the interior of a reference marker and is ready to extract the referenced content.
[0088] (4) Numeric Content State: This state is entered when the character [ is detected in the tag content state. This state is responsible for extracting and caching the reference number within the square brackets [ ].
[0089] (5) Tag Closed State: This state is entered when the character ] is detected in the numeric content state. This state is responsible for identifying and confirming the closing tag of the quotation mark.< / cite_symbol> .
[0090] 2.2 State Transition Logic
[0091] The complete workflow and precise logical judgment of a finite state machine are as follows: Figure 3 The process is shown below.
[0092] 2.3 Output and Action
[0093] This finite state machine not only transitions between states, but also triggers actions at specific nodes.
[0094] In the digital content state (STATE_IN_NUMS), the system caches consecutive numeric characters in a temporary buffer to form the original citation number. When the state machine successfully transitions from the tag-closed state (STATE_CLOSE_TAG) back to the normal state, it signifies that a complete citation tag has been parsed. At this point, the system immediately uses the original citation number in the cache to query the number mapping table maintained by the reference management module to obtain the corresponding display number, and immediately displays the original citation tag (such as...).<cite_symbol> [1]< / cite_symbol> Replace the formatted reference identifier in the output stream (e.g., [1]).
[0095] 3. Specific Implementation of the Three-Stage Reference Deduplication Mechanism
[0096] In the large language model streaming writing scenario described in this invention, reference data is characterized by its massive volume, high-speed arrival, and the need for immediate judgment. Conventional single deduplication methods struggle to achieve a balance between efficiency and accuracy. To address the shortcomings of single deduplication methods, a three-stage progressive deduplication algorithm is designed. This algorithm achieves layer-by-layer filtering from coarse to fine through a pipeline of SimHash->TF-IDF->semantic vectors. This specific combination strategy, along with thresholds validated through extensive experiments, combines SimHash (Hamming distance ≤ 3), TF-IDF similarity (≥ 0.85), and embedding vector similarity (≥ 0.9) to address the technical challenges of balancing processing speed and deduplication accuracy in the aforementioned scenario.
[0097] 3.1 The core process is as follows:
[0098] S1: A rapid coarse screening based on SimHash, utilizing the local sensitivity of SimHash to perform efficient and preliminary screening for duplication probability in a large number of documents.
[0099] Calculate the SimHash fingerprint S_new of the document to be added, and perform Hamming distance H(S_new, S_i) with the SimHash fingerprint S_i of existing documents in the cache database. If H(S_new, S_i) ≤ T1 (where the threshold T1 = 3), the document i is marked as a suspected duplicate candidate and proceeds to the next stage; otherwise, the document is considered brand new and can be directly added to the reference cache. This stage can quickly eliminate most dissimilar documents.
[0100] S2: Structural similarity determination based on TF-IDF cosine similarity. On the basis of preliminary screening, more accurate similarity judgment is made from the perspective of text structure and keyword weight.
[0101] For the suspected duplicate candidate documents identified in the first stage, calculate their TF-IDF vectors V_tfidf_new and V_tfidf_i, respectively, and calculate their cosine similarity Sim_TFIDF=cos(V_tfidf_new,V_tfidf_i). If Sim_TFIDF≥T2 (where the threshold T2=0.85), the two documents are considered highly similar in topic and key content, and are marked as high-probability duplicates, proceeding to the final confirmation stage; otherwise, they are determined to be non-duplicates and removed from the candidate list.
[0102] S3: Semantic consistency verification based on cosine similarity of text embedding vectors serves as the final checkpoint for deduplication decisions, performing final verification at a deep semantic level to ensure the accuracy of deduplication judgment.
[0103] A pre-trained semantic vector model (such as Bge-m3) is used to obtain the semantic embedding vectors E_new and E_i of the document to be added and the candidate document with high probability of duplication, and the cosine similarity Sim_Emb=cos(E_new,E_i) is calculated. If Sim_Emb≥T3 (where the threshold T3=0.90), then the document to be added and candidate document i are finally confirmed as duplicate documents; otherwise, they are determined to be non-duplicate documents.
[0104] 3.2 Deduplication mechanism process as follows Figure 4 .
[0105] Although the steps in the above embodiments are described in the above order, those skilled in the art will understand that in order to achieve the effect of this embodiment, different steps do not need to be executed in such an order. They can be executed simultaneously (in parallel) or in a reverse order. These simple variations are all within the protection scope of this invention.
[0106] Example 2: Based on the above technical concept, the second embodiment of the present invention proposes a virtual system corresponding to the method of the first embodiment, namely, a large-model streaming writing assistance system based on Redis and finite state machines. This system integrates the streaming generation capabilities of a large language model, the real-time parsing capabilities of a finite state machine, the persistent state management capabilities of Redis caching, and a multi-stage progressive reference deduplication algorithm to construct an integrated writing assistance system. This system can receive user writing tasks, acquire references in parallel, and while streaming text content, parse and convert citation tags in real time, dynamically manage the reference list, and push the processed text stream to the client in real time.
[0107] The following is combined Figure 5 The various parts of the second embodiment of the present invention will be described in detail below.
[0108] like Figure 5 The aforementioned large-scale streaming writing assistance system based on Redis and finite state machines, wherein the modules of the system work collaboratively according to the method described in Embodiment 1, includes:
[0109] Content generation module 210 is used to call a large language model to generate streaming content based on content generation prompts and output a text stream.
[0110] Reference resolution module 220 is used to parse reference markers in a text stream in real time using a finite state machine;
[0111] The reference management module 230 is used to determine newly cited references based on the citation results obtained from the parsing, and to perform deduplication, numbering and formatting of the newly cited references.
[0112] The streaming output module 240 is used to push the parsed text stream to the client in real time using a server-sending event protocol.
[0113] The dynamic update module 250 is used to dynamically update and manage the reference list; after the current chapter is generated, the updated session history and reference status are persisted to the Redis cache database.
[0114] Example 3: This embodiment of the invention also provides an electronic device and a computer-readable storage medium corresponding to Examples 1 and 2.
[0115] One of the electronic devices includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor performs the steps of any one of S101-S105 of a large-model streaming writing assistance method based on Redis and a finite state machine.
[0116] like Figure 6 As shown, the electronic device may include: at least one processor 31, at least one network interface 35, user interface 34, memory 36, and at least one communication bus 32.
[0117] The communication bus 32 is used to enable communication between these components.
[0118] The user interface 34 may include a display screen and a camera. Optionally, the user interface 34 may also include a standard wired interface and a wireless interface.
[0119] The network interface 35 may optionally include a standard wired interface or a wireless interface (such as a WIFI interface).
[0120] The processor 31 may include one or more processing cores. The processor 31 connects to various parts of the server using various interfaces and lines, and performs various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 36, and by calling data stored in the memory 36. Optionally, the processor 31 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 31 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 31 and may be implemented as a separate chip.
[0121] The memory 36 may include random access memory (RAM) or read-only memory. Optionally, the memory 36 may include a non-transitory computer-readable storage medium. The memory 36 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 36 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 36 may also be at least one storage device located remotely from the aforementioned processor 31. Figure 6 As shown, the memory 36, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application program for a large-scale streaming writing assistance method, system, electronic device, and storage medium based on Redis and a finite state machine.
[0122] exist Figure 3In the electronic device shown, the user interface 34 is mainly used to provide an interface for users to input data and obtain user input data; while the processor 31 can be used to call an application stored in the memory 36 that is a large model streaming writing assistance method based on Redis and finite state machine. When executed by one or more processors 31, the electronic device performs one or more of the methods described in steps S101-S104 of the above embodiment.
[0123] Those skilled in the art will clearly understand that the technical solutions of this application can be implemented using software and / or hardware. In this specification, "unit" and "module" refer to software and / or hardware capable of independently performing or cooperating with other components to perform specific functions. Hardware may include, for example, a Field-Programmable Gate Array (FPGA), an Integrated Circuit (IC), etc.
[0124] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any one of S101-S104 of a large-model streaming writing assistance method based on Redis and a finite state machine.
[0125] Specifically, the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.
[0126] The code of the computer program can be in the form of source code, object code, executable file, or some intermediate form.
[0127] Computer-readable storage media may include cache, high-speed random access memory (RAM), such as the common double data rate synchronous dynamic random access memory (DDR SDRAM), and may also include non-volatile memory (NVRAM), such as one or more read-only memory (ROM), disk storage devices, flash memory devices, or other non-volatile solid-state storage devices such as optical discs (CD-ROM, DVD-ROM), floppy disks, or data tapes.
[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.
Claims
1. A large-scale streaming writing assistance method based on Redis and finite state machines, characterized in that, The method includes: Based on the content generation prompts, a large language model is invoked to generate streaming content and output a text stream. The finite state machine is used to parse the citation markers in the text stream in real time. Based on the citation results obtained from the analysis, the newly cited references are determined, and the newly cited references are deduplicated, numbered, and formatted. The server sends an event protocol to push the parsed text stream to the client in real time. Dynamically update and manage the reference list; after the current chapter is generated, persist the updated session history and reference status to the Redis cache database.
2. The method according to claim 1, characterized in that, The process of generating streaming content based on content-generated prompts and calling a large language model to generate the text stream includes, before outputting the text stream: receiving writing task information, generating a globally unique task ID, and persisting the task status and initial data to a Redis cache database; wherein, the writing task information includes: writing requirements, article outline, and initial reference list; Based on the writing task information, a parallel search process is initiated to obtain relevant reference data, integrate writing requirements, current chapter outline and relevant references, and construct content generation prompt information. The task status persisted to the Redis cache database includes: task ID, writing requirements, current generation progress, and initial reference list; The initial data persisted to the Redis cache database includes the original data corresponding to the writing requirements, article outline, and initial reference list.
3. The method according to claim 2, characterized in that, The relevant reference data is acquired based on a pre-set timeout mechanism for reference data. The timeout mechanism includes: if the reference data acquisition is not completed within the timeout threshold, the search is stopped, and the initial reference list is used to generate subsequent content.
4. The method according to claim 1, characterized in that, The real-time parsing of reference markers in the text stream using a finite state machine includes: the finite state machine performing state transitions based on predefined trigger characters, including normal state, tag open state, tag content state, numeric content state, and tag closed state, to complete the identification, content extraction, and storage of reference markers; The predefined trigger characters include: characters that trigger the transition from the normal state to the tag open state, character sequences that trigger the transition from the tag open state to the tag content state, characters that trigger the transition from the tag content state to the numeric content state, characters that trigger the transition from the numeric content state to the tag closed state, and character sequences that trigger the transition from the tag closed state to the normal state.
5. The method according to claim 1, characterized in that, The deduplication process for newly cited references includes: Calculate the Hamming distance between the SimHash fingerprint of the document to be added and the SimHash fingerprint of the document already in the cache database, determine the magnitude of the Hamming distance, and filter the documents to be added that are within the preset Hamming distance threshold range as suspected duplicate candidate documents. Calculate the TF-IDF vector cosine similarity between the document to be added and the suspected duplicate candidate documents, and filter the suspected duplicate candidate documents whose TF-IDF vector cosine similarity is within the first similarity threshold as high probability duplicate documents; Calculate the cosine similarity of the semantic embedding vectors of the document to be added and the documents with high probability of duplication. Select the documents with high probability of duplication whose semantic embedding vector cosine similarity is within the second similarity threshold as the duplicate documents.
6. The method according to claim 1, characterized in that, The text stream data structure pushed to the client in real time includes: content type identifier, current chapter identifier, text fragments, and cumulative text content.
7. The method according to claim 1, characterized in that, Also includes: In response to the request to generate a bibliography section, all cited references are sorted according to their display numbers, formatted and output according to a preset format, and the Redis cache data corresponding to the writing task is cleared.
8. A large-scale streaming writing assistance system based on Redis and finite state machines, characterized in that, To implement the method of any one of claims 1-7, comprising: The content generation module is used to call the large language model to generate streaming content based on the content generation prompt information and output a text stream; The reference resolution module is used to parse reference markers in a text stream in real time using a finite state machine; The reference management module is used to determine newly cited references based on the citation results obtained from the parsing, and to perform deduplication, numbering, and formatting of the newly cited references. The streaming output module is used to push the parsed text stream to the client in real time using a server-sending event protocol; The dynamic update module is used to dynamically update and manage the reference list; after the current chapter is generated, the updated session history and reference status are persisted to the Redis cache database.
9. An electronic device, comprising a memory and a processor, characterized in that, The memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the large-model streaming writing assistance method based on Redis and finite state machines as described in any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the steps of any one of claims 1 to 7 for large-scale streaming writing assistance based on Redis and finite state machines.