Data processing method and electronic device

Abstract

This application discloses a data processing method and electronic device, relating to the field of large model technology. The data processing method includes: acquiring dialogue fragment records; parsing the dialogue fragment records to obtain slots including at least one predefined field, and storing the slots in a candidate memory pool; performing quality evaluation on the slots in the candidate memory pool to obtain a refinement score, and determining a processing decision for the slots based on the refinement score; the processing decision includes at least refinement and retention; for the slots whose processing decision is refinement, submitting the refinement task corresponding to the slot to an asynchronous task queue decoupled from the dialogue response link; for the refinement task located in the asynchronous task queue, causing the dialogue model to output a structured memory item based on the slot and the prompt fragment; and writing the structured memory item into a long-term memory graph.

Description

Technical Field

[0001] This application relates to the field of large model technology, and in particular to a data processing method and an electronic device. Background Technology

[0002] Large Language Models (LLMs), or simply large models, are models trained on massive amounts of text data. These models possess powerful language understanding and generation capabilities, able to understand, summarize, predict, and generate human language. They are widely used in chatbots, code generation, content creation, and translation. As intelligent agents, the long-term memory mechanism supporting multi-turn dialogue has become a key factor in improving the dialogue ability of large models. The key to the long-term memory mechanism lies in how to efficiently insert new dialogue content into long-term memory. However, in related technologies, frequently calling large models to complete insertions suffers from high insertion latency, slow response speed, high resource consumption, and high graph construction overhead.

[0003] Therefore, an improved data processing method is expected to solve at least one of the above problems. Summary of the Invention

[0004] This application provides a data processing method and an electronic device to at least solve at least one of the problems of high insertion latency, slow response speed, large resource consumption, and large map construction overhead in the related art under long-term memory mechanisms.

[0005] On one hand, a data processing method is provided, comprising: acquiring dialogue fragment records; parsing the dialogue fragment records to obtain slots including at least one predefined field, and storing the slots in a candidate memory pool; performing quality evaluation on the slots in the candidate memory pool to obtain a refinement score, and determining a processing decision for the slots based on the refinement score; the processing decision includes at least refinement and retention; for the slots whose processing decision is refinement, submitting the refinement task corresponding to the slot to an asynchronous task queue decoupled from the dialogue response link; for the refinement tasks located in the asynchronous task queue, having the dialogue model output a structured memory item based on the slot and the prompt fragment; and writing the structured memory item into a long-term memory graph.

[0006] On the other hand, an electronic device is provided, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the data processing method described above.

[0007] The data processing method provided in this application performs quality assessment and screening of slots in the candidate memory pool, triggering dialogue model refinement only when the refinement score is high. Most general content can be temporarily stored or filtered out without requiring processing by the dialogue model, avoiding unnecessary resource overhead. Attached Figure Description

[0008] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 This is a flowchart illustrating a data processing method according to some embodiments; Figure 2 This is a flowchart illustrating another data processing method provided according to some embodiments; Figure 3 This is a flowchart illustrating another data processing method provided according to some embodiments; Figure 4 This is a flowchart illustrating another data processing method provided according to some embodiments; Figure 5 This is a flowchart illustrating another data processing method provided according to some embodiments; Figure 6 This is a flowchart illustrating another data processing method provided according to some embodiments; Figure 7 This is a flowchart illustrating another data processing method provided according to some embodiments; Figure 8 This is a schematic diagram of the structure of a data processing apparatus according to some embodiments. Detailed Implementation

[0010] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0011] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0012] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0013] As intelligent agents, dialogue models rely heavily on long-term memory mechanisms to support multi-turn dialogues, which has become a key factor in enhancing their conversational capabilities. Related technologies have proposed solutions to extract and store key information from dialogues for subsequent retrieval, such as vector-indexed memory banks or knowledge graph-based memory graphs. The key to long-term memory mechanisms lies in efficiently inserting new dialogue content into long-term memory. This insertion process involves parsing, summarizing, or extracting new information and updating the storage structure, often requiring the use of the dialogue model. However, immediately calling the dialogue model to parse and insert new information whenever it is generated causes significant delays, failing to meet the user's responsiveness requirements in real-time interaction. In long-term dialogues, frequent insertion operations are necessary, each relying on the dialogue model, leading to high resource overhead. For knowledge graph-based memory structures, inserting new knowledge often requires entity extraction, relation extraction, and graph database updates, a complex and time-consuming process. Extracting knowledge using dialogue models also suffers from time consumption and error susceptibility. Furthermore, building and maintaining a sophisticated long-term memory graph significantly increases resource consumption in large-scale applications.

[0014] In view of this, embodiments of this application provide a data processing method, such as... Figure 1 As shown, Figure 1 This is a flowchart illustrating a dialogue model data processing method according to some embodiments. The data processing method includes steps S101, S102, S104, S105, S106 and S108.

[0015] In step S101, the dialogue segment record is obtained.

[0016] In some embodiments, dialogue fragment recordings may originate from a dialogue response link, for example.

[0017] In some embodiments, see Figure 2This is a flowchart illustrating another data processing method provided according to some embodiments, wherein obtaining dialogue fragment records includes steps S201 and S202.

[0018] In step S201, the original fragment text is obtained. The original fragment text includes user input information and / or the output information of the dialogue model.

[0019] In step S202, a dialogue fragment record is generated based on the original fragment text, and fields such as fragment identifier code, session identifier code, and timestamp are assigned to the dialogue fragment record. The dialogue fragment record includes at least one of the following: text, role (user / assistant), and timestamp (ts).

[0020] For example, fields such as segment identifier code, session identifier code, and timestamp are assigned to dialogue segment records for slot extraction and traceable location in subsequent steps.

[0021] In step S102, the dialogue fragment record is parsed to obtain a slot including at least one predefined field, and the slot is stored in the candidate memory pool.

[0022] The predefined fields include slot code, fragment identifier code, entity (subject / object), event (event), time (time), location (location), raw_span (raw_span), and extraction confidence (extract_conf).

[0023] In some embodiments, see Figure 3 This is a flowchart illustrating another data processing method provided according to some embodiments. Step S102 includes steps S301 and S302.

[0024] In step S301, the memorable information type of the dialogue segment record is determined based on a lightweight text classification model.

[0025] In step S302, based on a preset rule base corresponding to the type of memorable information, field information corresponding to predefined fields in the dialogue fragment record is extracted and the extraction confidence level is recorded to generate slots.

[0026] In cases where at least one of the following conditions is met: missing, uncertain, or ambiguous field information in the dialogue segment record, the extraction reliability is reduced.

[0027] In some embodiments, see Figure 3 This is a flowchart illustrating another data processing method provided according to some embodiments. Step S102 further includes steps S303 and S304.

[0028] In step S303, the dialogue fragment record is subjected to grammatical analysis to obtain the keywords, sentence structure and positional features of the dialogue fragment record; In step S304, the classification features of the dialogue segment record are determined based on keywords, sentence structure, and location features.

[0029] For example, steps S301 and S302 can be executed before or after steps S303 and S304. Similarly, steps S301 and S302 can also be executed in parallel with steps S303 and S304.

[0030] For example, the dialogue fragment is segmented into sentences and words, and classification features are constructed based on keywords, sentence structure, and location features. A lightweight text classification model outputs the memorable information type of the dialogue fragment based on keywords, sentence structure, and location features. Subsequently, the memory extraction processing program reads a preset rule base and matches the dialogue fragments one by one according to the matching conditions of the rules. When a match is successful, the corresponding field extraction action extracts and fills in fields such as people, time, location, and event phrases from the original text, and records the location of the original text evidence and the credibility of the extraction.

[0031] Among them, people and locations are identified through dictionary matching and contextual constraint rules, events are identified through verb phrase location rules, and time is converted into a unified time representation through time identification and normalization processing rules. When a field is missing, cannot be determined, or is ambiguous, the extraction confidence is reduced while retaining original text evidence for subsequent filtering and asynchronous refinement.

[0032] Through the slot-based rapid extraction mechanism, important content is captured and entered into the candidate memory pool the moment the dialogue occurs, without waiting for the dialogue model to summarize or for memory to be transferred step by step. This allows the candidate memory pool to be written into the main dialogue response chain with low latency, enabling the acquisition of key information provided by the user as soon as possible and significantly improving response speed.

[0033] In step S104, the slots in the candidate memory pool are quality-assessed to obtain a refining score, and a processing decision for the slots is determined based on the refining score. The processing decision includes at least refining and retaining.

[0034] In this embodiment, the refinement score consists of an integrity score (C), a conflict / duplication score (R), and an importance score (I). The integrity score (C) is determined based on the missing fields of the predefined slots and their normalization (e.g., whether the time field is parsable). The conflict / duplication score (R) is determined based on the candidate pool deduplication signature and the comparison results with the current values ​​of the same entity and attribute in long-term memory storage. The importance score (I) is determined based on event type whitelists / blacklists, user profile key fields (preferences / identity / long-term habits) mapping rules, and dialogue turn positions, etc.

[0035] In some embodiments, see Figure 4 This is a flowchart illustrating another data processing method provided according to some embodiments. Step S104 includes steps S401 to S403. The refining fraction includes a first refining fraction.

[0036] In step S401, the refining score characteristics are output based on the slot.

[0037] The refined scoring features include completeness features, extraction credibility features, duplication features, consistency features, and importance features.

[0038] In step S402, the first refining score is determined based on the refining scoring characteristics and preset weights.

[0039] In step S403, if the first refining score is greater than or equal to the first refining threshold, the processing decision for the slot is determined to be refining; if the first refining score is greater than or equal to the second refining threshold and less than the first refining threshold, the processing decision for the slot is determined to be retention.

[0040] For example, a set of refined scoring features is generated for each slot, and a first refined score (refine_score) is calculated accordingly. The refined scoring features include predefined field completeness features (signals for missing required fields and resolvable time fields), extraction confidence features (extract_conf), duplication features (number of times the same signature appears in the candidate memory pool and count of duplicates within the recent time window), consistency features (comparison results with the current values ​​of the same subject and attribute in long-term memory storage and conflict flags), and importance features (slot type and whether the event / relationship hits the memory field table). The evaluation module performs a weighted summation of the above features according to preset weights to obtain a first refinement score, refine_score. Based on the threshold and rule priority, it outputs a processing decision: when the first refinement score refine_score is greater than or equal to the first preset threshold and no explicit invalid rule is hit, the processing decision is refinement; when the first refinement score refine_score is greater than or equal to the second refinement threshold and less than the first refinement threshold, the processing decision is retention; when an invalid rule is hit or the first refinement score refine_score is less than the second refinement threshold, the output processing decision is discard.

[0041] In some embodiments, see Figure 5 This is a flowchart illustrating another data processing method provided according to some embodiments. Step S104 includes steps S501 to S503. The refining fraction includes a second refining fraction.

[0042] In step S501, the refining score characteristics are output based on the slot.

[0043] The refined scoring features include completeness features, extraction credibility features, duplication features, consistency features, and importance features.

[0044] In step S502, the second refining score is determined based on the refining scoring characteristics and the preset model.

[0045] In step S503, if the second refining fraction is greater than or equal to the third refining threshold, the processing decision for the slot is determined to be refining.

[0046] For example, the lightweight classification model outputs a second refinement score p_refine based on the refined score features. Here, the second refinement score p_refine is the probability value that the slot needs refinement. Then, a processing decision is output based on a third refinement threshold. If the second refinement score is greater than or equal to the third refinement threshold, the processing decision is to pass; otherwise, the processing decision is to keep or discard.

[0047] Lightweight classification models are obtained, for example, through distillation: using the validity results refined by the dialogue model or the results of human review as supervision signals, training samples (feature vectors, labels) are constructed and trained; inference is performed locally to meet low latency requirements.

[0048] For example, training samples (slot_features, y) are constructed using the refining results of the offline refining process or the results of manual review as supervision signals, where y is the label of "whether refining is needed / whether a valid long-term memory item is formed after refining". The refining scoring feature slot_features includes at least (a) field integrity features (field missing flag, time field resolvability), (b) conflict / duplication features (matching results with the same entity and attribute as the long-term memory, candidate pool deduplication count), (c) importance features (event type, key field hits in user profile), and (d) extraction confidence features. The distilled classifier is deployed in the memory evaluation and screening process for low-latency discrimination and can be adjusted online through a third refining threshold and weight parameters.

[0049] The data processing method provided in this application filters slots in the candidate memory pool, triggering dialogue model refinement only for high-value content. Most general content can be temporarily stored or filtered out without dialogue model processing, avoiding unnecessary computational overhead. This selective asynchronous refinement reduces unnecessary dialogue model calls in each round of dialogue, alleviating server load and reducing interaction latency, thus minimizing user-perceptible interaction waiting time.

[0050] In step S105, for slots whose processing decision is refining, the refining task corresponding to the slot is submitted to the task queue.

[0051] In some embodiments, step S105 specifically includes: when the processing decision is to refine, submitting the refinement task corresponding to the slot to the asynchronous task queue, and using the slot, and the original fragment text pointed to by the fragment identifier code of the dialogue fragment record corresponding to the slot as the input of the refinement task.

[0052] In some embodiments, step S105 further includes: recording at least one of the following: the number of repetitions of the refining task, the timeout policy, the receipt status, and the reason for failure.

[0053] For example, slots that pass the processing decision are converted into refining task IDs and written to an asynchronous queue, with concurrency and idempotency control implemented. The input to a refining task is limited to the original fragment text pointed to by the predefined fields of the slot and the fragment identifier (optionally with a small amount of local context), avoiding sending the entire history of the dialogue directly into the refining process, thereby controlling costs. The number of retries, timeout policy, receipt status, and failure reason for each refining task are recorded to ensure maintainability.

[0054] The data processing method provided in this application breaks down the originally concentrated parsing and writing pressure into controllable small tasks by writing slots into the candidate memory pool, filtering and diverting by quality assessment, and scheduling by asynchronous queues. This reduces the risk of batch processing peaks and task backlog when activity changes, and improves the throughput stability and reliability of long-term memory updates.

[0055] In step S106, for the refinement task located in the task queue, the dialogue model outputs a structured memory item based on the slot and the prompt fragment.

[0056] In some embodiments, see Figure 6 This is a flowchart illustrating another data processing method provided according to some embodiments. Step S106 includes steps S601 to S603.

[0057] In step S601, a prompt fragment is constructed based on the slot.

[0058] Among them, the prompt fragment is, for example, a range-controlled model input structure fragment constructed from a single candidate memory.

[0059] In some embodiments, the prompt fragment includes: predefined field information of the slot and the original fragment text corresponding to the slot.

[0060] In some embodiments, the prompt fragment also includes a local context window and refined output format constraint information related to the original fragment text corresponding to the slot.

[0061] In step S602, the dialogue model supplements the predefined fields of the slot based on the prompt fragment.

[0062] In step S603, the dialogue model outputs structured memory items based on the supplemented slots. These structured memory items include relation triples.

[0063] For example, a prompt fragment is constructed based on the slot, the dialogue model is invoked to complete the semantic information, and a structured memory item is output. To ensure its parsability and consistency, the output must meet a predefined format (such as JSON), with fields including a set of relation triples or attribute sets, as well as confidence information and the corresponding original fragment text. The dialogue model refinement step is not in the main dialogue response chain to avoid affecting user response time.

[0064] Long-term memory entries, refined through memory assessment and dialogue models, possess higher information concentration and reliability. Because only content with high refinement scores enters the refinement stage and is extracted into structured memory items such as summaries or knowledge triples by the dialogue model, the information ultimately written into the long-term memory map is more refined and of higher value. Furthermore, conflict detection and relationship fusion strategies are employed during map merging to ensure a more consistent structural expression and maintainability of the long-term memory map throughout its continuous evolution.

[0065] In step S108, the structured memory items are written into the long-term memory map.

[0066] In some embodiments, step S108 includes: aligning the entity of the structured memory item to a node in the long-term memory graph using any one of an alias table, standardization rules, or hash signatures. Alternatively, if the structured memory is not being written to the long-term memory graph for the first time, updating the weights and most recently mentioned timestamps of the edges corresponding to the structured memory item in the long-term memory graph. Alternatively, if the entity in the long-term memory graph differs from the entity in the structured memory item, retaining the entity in the long-term memory graph, writing the entity in the structured memory item, and recording conflict information.

[0067] For example, writing structured memory items into the long-term memory graph includes entity alignment and deduplication, relation upsert and edge weight maintenance, and conflict marking. Entity alignment and deduplication align entities to existing nodes using alias tables / standardization rules / hash signatures to avoid duplicate node creation. In relation upsert and edge weight maintenance, when the same relation triple appears repeatedly, edges are not created again; instead, the edge weight and the most recently mentioned timestamp (last_seen_ts) are updated. In conflict marking, when different values ​​appear for the same attribute of the same entity, a conflict record is written and version information is retained. The current effective value is determined based on timestamp, frequency, and confidence strategy, while historical values ​​are retained for retrospection and rollback.

[0068] In the dialogue model data processing method provided in this application embodiment, the dialogue model is further configured to generate output information based on long-term memory graph and user input information.

[0069] For example, during dialogue generation, the dialogue model data processing method provided in this application can provide the dialogue model with long-term memory related to the current context. Target nodes or triples are retrieved from the long-term memory graph and converted into cue fragments (such as "memory summaries / lists of key facts") that can be used by the dialogue model, enhancing the consistency of the dialogue model's output information without compromising the latency of the main dialogue.

[0070] In some embodiments, the dialogue model is also configured to generate output information based on a long-term memory graph, slots in a candidate memory pool, and user input.

[0071] In some embodiments, see Figure 7 This is a flowchart illustrating another data processing method provided according to some embodiments. The data processing method provided in this application embodiment further includes step S103.

[0072] In step S103, the slots in the candidate memory pool are temporarily stored and their lifecycles are managed.

[0073] In this embodiment, the maintenance status field state ∈ {NEW, SCORED, QUEUED, REFINED, DROPPED} for each slot in the candidate memory pool records information such as the first record timestamp first_seen_ts and the most recent record timestamp last_seen_ts.

[0074] In this embodiment, step S103 includes: merging and counting slots repeatedly stored in the candidate memory pool within a preset deduplication period, i.e., merging and counting repeated slots within a short period according to (subject, event, object, time) or its hash signature. Alternatively, if the expiration preset period is greater than or equal to the specified period, downgrading or deleting slots whose maintenance status field has not been promoted, for example, according to LRU (Least Recently Used) or Time To Live (TTL) policies. Alternatively, determining the input and output information associated with the slot based on the slot's fragment identifier code, i.e., associating the original fragment text through the fragment identifier code to support context supplementation and result verification in subsequent steps. In some embodiments, see [reference]. Figure 7 The data processing method provided in this application embodiment further includes step S107, which is, for example, before step S108.

[0075] In step S107, the structured memory items are preprocessed.

[0076] In this embodiment, step S107 includes validating the fields and / or types of the structured memory, such as schema validation. Alternatively, the relation names of the structured memory are mapped to a relation dictionary. Alternatively, entity normalization is performed on the structured memory.

[0077] In this embodiment, step S107 further includes resubmitting the refining task to the asynchronous task queue if the field and / or type of the structured memory item is validated and the validation fails, in order to prevent dirty data from directly entering the graph.

[0078] In some embodiments, steps S101 and S102 constitute a dialogue response chain, where important content is extracted into the candidate memory pool through slots at the moment the dialogue occurs, without waiting for the dialogue model to summarize or for memory to be transferred level by level. This allows the candidate memory to be written into the dialogue response chain with low latency, enabling the acquisition of key information provided by the user as soon as possible, and significantly improving the response speed of the data processing method to new knowledge.

[0079] Steps S103 to S108 constitute an asynchronous processing link, decoupling the slot quality assessment, refining, and graph update steps from the dialogue response link. The dialogue model is only called when necessary, and this process is executed asynchronously independently of the dialogue response link. This reduces the dependence on the continuous online processing of the dialogue model, breaks down the originally centralized parsing and writing pressure into controllable small tasks, reduces the risk of batch processing peaks and task backlog when switching session stages or changing activity, and improves the throughput stability and reliability of long-term memory updates.

[0080] In some embodiments, the data processing method provided in this application includes three stages: rapid slot insertion, evaluation and screening, and asynchronous refinement and map update. The rapid slot insertion stage includes steps S101 to S103, the evaluation and screening stage includes steps S104 and S105, and the asynchronous refinement and map update stage includes steps S106 to S108.

[0081] In the slot quick insertion phase, for each new user dialogue, this application embodiment introduces a slot quick insertion mechanism. The slot, as a structured placeholder for new information, contains one or more predefined fields to cover elements such as Who, When, What, and Where.

[0082] After the current round of dialogue occurs, the system does not immediately invoke the dialogue model. Instead, it first performs basic text processing on the dialogue fragment records and uses a lightweight text classification model, such as a Natural Language Processing (NLP) model, to determine the type of memorable information to which the fragment belongs. Subsequently, the memory extraction processing program reads the preset rule base corresponding to the memorable information type, matches each dialogue fragment record one by one, and performs field extraction. It locates and extracts fields such as people, events, time, and location from the original text to generate slot records. Taking the user input "Remind me to attend Alice's birthday party in a week" as an example, the classification result is "Schedule / Reminder". The rule base can extract the person = Alice and the event = birthday party from the original text, and convert "in a week" into a unified time representation through time recognition and normalization processing; at the same time, it records the location of the original text evidence and the extraction credibility.

[0083] When a field is missing, uncertain, or ambiguous, the extraction confidence of that slot is reduced while retaining the original evidence for subsequent evaluation and asynchronous refinement. Extracted slots are directly written to the candidate memory pool, essentially temporarily storing this potential long-term memory. To avoid high-frequency writes impacting system performance, the candidate memory pool supports caching and batch write strategies: multiple slots generated in a short period can be temporarily stored in memory or cache, and then written to persistent storage once batch conditions are met, awaiting subsequent evaluation and processing.

[0084] By quickly inserting into slots, new dialogue points can be recorded in milliseconds even without invoking the dialogue model, providing a foundation for real-time retrieval.

[0085] During the evaluation and screening phase, since the slots that are quickly inserted have not undergone in-depth semantic analysis, they may contain incomplete, inaccurate, or unimportant content. Therefore, an evaluation and screening phase is set up to score the quality of the slots in order to determine whether to trigger the subsequent refining steps.

[0086] The first refinement score considers multiple factors. First, field completeness: if a slot's key fields (such as people or events) are missing or ambiguous, the score decreases. Second, conflict detection: the system compares the new slot with existing entries in the long-term memory graph; if the same entity already exists and the information contradicts it, or if the content has recently been repeated, the new slot is considered low-value, and the score decreases. Third, importance assessment: the importance of the slot is determined based on the context of the conversation (e.g., information involving the user's long-term preferences has high weight, while casual conversation has low weight). Additionally, the priority of information explicitly emphasized by the user can be considered. For example, the comprehensive scoring formula is: First Refinement Score = w_c * Completeness + w_i * Importance – w_f * Conflict Repetition, where w_c, w_i, and w_f are weights that can be determined based on training.

[0087] When the refined score of a slot is higher than the first refinement threshold, it is considered likely to become high-quality long-term memory and needs to enter the asynchronous refinement process; otherwise, it is only retained in the candidate pool and not yet put into refinement. Through memory evaluation and screening, a large amount of trivial or invalid information will be filtered out and will not occupy the computing power resources of the dialogue model.

[0088] In some embodiments, a lightweight classifier can also be introduced in the evaluation and screening stage. Based on the slot, a second refined score p_refine is output, and it is fused with the first refined score refine_score to form a final screening decision. The fusion method is as follows: when the first refined score Refine_score is greater than or equal to the first refinement threshold, or the second refined score p_refine is greater than or equal to the third refinement threshold, the processing decision is to refine; when the clear invalid rule is hit, the processing decision is to discard; in other cases, the processing decision is to retain, and the slot is retained in the candidate pool and the evaluation result is recorded.

[0089] In the asynchronous refinement and graph update stage, for the slots whose processing decision is to refine, an asynchronous refinement process is started to transform them into a complete long-term memory graph. The asynchronous refinement is completed by the dialogue model, but it is not synchronized with the dialogue. Instead, it is processed in the background thread without blocking user interaction.

[0090] The refinement process includes: based on the elements provided by the slot, constructing a prompt segment for the dialogue model to supplement and improve relevant information and extract more explicit relationship triples. For example, if the slot records "Person = Alice, Event = Birthday Party, Time = May 2025", the refinement prompt segment can guide the dialogue model to return more complete knowledge: "Alice's birthday is in May every year, and the user promised to attend her birthday party in 2025." The refined result output by the dialogue model will be parsed into the standard knowledge graph format (such as <User>-<Attend>-<Alice Birthday Party@2025-5>), and then merged with the existing long-term memory graph.

[0091] The graph update strategy employs an incremental update model: if new knowledge involves existing entity nodes (e.g., "Alice" already exists in the graph), new relation edges are added to that node; if a new entity appears, a new node is created and connected to the network. Conflict detection is performed during the merging process. For nodes with contradictory content (e.g., the previous memory recorded "Alice's birthday is in April," while new knowledge indicates it's in May), a conflict marking mechanism is triggered, retaining the more reliable or latest record and marking the node with a version or inconsistency flag. Duplicate memories are deduplicated, preventing duplicate node additions. After the update, the new knowledge officially becomes part of the long-term memory graph, available for subsequent retrieval. Asynchronous refinement allows the massive graph construction task to be completed in the background without affecting the smoothness of the foreground dialogue. Simultaneously, the high-quality relation triples obtained through refinement ensure a clear structure and semantic accuracy in the long-term memory.

[0092] In some embodiments, the dialogue model data processing method provided in this application can be applied to multi-turn dialogue-intensive scenarios such as enterprise-level intelligent agent platforms, AI customer service, and legal Q&A. In these scenarios, the intelligent agent needs to continuously memorize a large amount of information provided by users (such as user preferences and historical questions). The data processing method provided in this application can maintain long-term memory in an efficient and low-cost manner, enabling the intelligent agent to accurately remember user information and maintain contextual coherence even after long-term interaction, significantly improving user experience and service quality.

[0093] In some embodiments, the dialogue model data processing method provided in this application can be used in a multi-agent collaborative memory-sharing platform as a knowledge injection mechanism. Different agents can inject their newly acquired knowledge into the global knowledge graph after structuring it through a shared candidate memory pool and refinement pipeline. By filtering and refining to improve memory quality, efficient knowledge synchronization and long-term memory sharing can be achieved in the agent cluster, providing support for the collaborative processing of complex tasks.

[0094] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0095] Embodiments of this application also provide a dialogue model data processing apparatus, such as... Figure 8 As shown, Figure 8 This is a schematic diagram of the structure of a dialogue model data processing device according to some embodiments.

[0096] The data processing device 100 includes a dialogue access component 110, a memory update component 120, and a memory storage component 130. The memory update component 120 includes, for example, a slot quick insertion sub-component 120a, an evaluation and filtering sub-component 120b, and an asynchronous refinement and graph update sub-component 120c.

[0097] The dialogue access component 110 includes a dialogue acquisition module 111, which is used to acquire dialogue fragment records.

[0098] The slot quick insertion sub-component 120a includes a slot extraction module 121 and a candidate memory pool management module 122. The slot extraction module 121 is used to parse dialogue fragment records to obtain slots that include at least one predefined field.

[0099] The candidate memory pool management module 122 is used to merge and count slots that are repeatedly stored in the candidate memory pool within a preset deduplication period. Alternatively, the candidate memory pool management module 122 is used to downgrade or delete slots whose maintenance status field has not been promoted if the expiration period is greater than or equal to a preset expiration period. Alternatively, the candidate memory pool management module 122 determines the input and output information related to the slot based on the slot's fragment identifier code. The evaluation and filtering sub-component 120b includes an evaluation and filtering module 123 and a refining task scheduling module 124. The evaluation and filtering module 123 is used to perform quality evaluation on the slots in the candidate memory pool to obtain a refining score, and determine the processing decision for the slot based on the refining score; the processing decision includes at least refining and retention.

[0100] In some embodiments, the candidate memory pool management module 122 is further configured to perform a quality assessment on a slot for which the processing decision is to be retained, in order to obtain a refinement score, when a subsequent dialogue arrives or the slot reaches the reassessment time window.

[0101] In this embodiment, the slot is refined when the refining fraction of the slot is greater than or equal to the first refining threshold, or when the refining fraction is greater than or equal to the third refining threshold.

[0102] In this embodiment, if the refining conditions are not met in the slot and the time window for discarding is reached, the slot is cleaned up.

[0103] The refining task scheduling module 124 is used to submit the refining task corresponding to the slot to the task queue for slots whose processing decision is refining.

[0104] The asynchronous refinement and graph update sub-component 120c includes a refinement execution module 125, a structured processing module 126, and a graph update module 127. The refinement execution module 125 is used to enable the dialogue model to output structured memory items based on the slots and prompt fragments for the refinement tasks located in the task queue.

[0105] The structured processing module 126 is used to preprocess the structured memory items.

[0106] The graph update module 127 is used to write preprocessed structured memory items into the long-term memory graph.

[0107] The memory storage component 130 includes a long-term memory graph storage module 131, which is used to persistently store data related to the long-term memory graph. This data includes, for example, graph storage (entity nodes, relation edges, attribute tables, conflict tables, version tables) and necessary index structures (entity ID index, relation index, time index, etc.) to support efficient retrieval and subsequent maintenance.

[0108] In some embodiments, the dialogue access component 110 further includes a memory retrieval module 112, which generates output information based on a long-term memory graph and the user's input information.

[0109] In the data processing device provided in this application embodiment, the dialogue acquisition module 111, slot extraction module 121, candidate memory pool management module 122 and memory retrieval module constitute a dialogue response link, for example. Important content is extracted into the candidate memory pool through slot extraction at the moment the dialogue occurs, without waiting for the dialogue model to summarize or for memory to be transferred level by level. This allows the candidate memory to be written into the dialogue response link with low latency, enabling the acquisition of key information provided by the user in the first time, which significantly improves the response speed of the data processing device 100 to new knowledge.

[0110] The evaluation and screening sub-component 120b and the asynchronous refining and graph update sub-component 120c constitute an asynchronous processing link, decoupling the slot quality evaluation and refining steps from the dialogue response link. The dialogue model is only called when necessary, and this process is executed asynchronously independently of the dialogue response link. This reduces the data processing device 100's dependence on the continuous online processing of the dialogue model, breaks down the originally centralized parsing and writing pressure into controllable small tasks, reduces the risk of batch processing peaks and task backlogs when switching session stages or changing activity levels, and improves the throughput stability and reliability of long-term memory updates.

[0111] For example, suppose a user mentions information of long-term value in a conversation, such as "I hold concerts with students every Friday." The slot extraction module identifies key information such as "every Friday," "students," and "concert" and immediately writes the entire sentence as a candidate memory into the candidate memory pool. The memory evaluation and filtering module scores the candidate, judging its relevance to the user's long-term preferences / identity, and assigns it a higher refinement score. When the refinement score is greater than the refinement threshold, the processing decision is to refine it. After receiving this content, the background dialogue model generates a structured memory item, such as "The user's profession is a music teacher, and organizes student concerts every Friday," and adds or updates nodes in the long-term memory graph accordingly: under the "user" node, it adds the attributes "profession: music teacher," the association "activity: concert," and the information association "cycle: every Friday," thereby continuously storing the user's personalized preferences. Subsequently, even after multiple rounds of conversation, the memory "the user is a music teacher and holds concerts every Friday" can still be quickly retrieved through the long-term memory graph. When users bring up music-related topics again, the data processing method can immediately recall previous memories (such as asking about the most recent concert), demonstrating a highly coherent long-term memory capability. It captures key information in real time and ensures the accuracy and durability of memory through asynchronous refinement and graph updates, providing users with a continuous interactive experience.

[0112] The dialogue model data processing method provided in this application forms a closed-loop long-term memory optimization scheme: acquiring dialogue fragment records, extracting slots and writing them into the candidate pool, evaluating and filtering, asynchronously refining high-scoring items, incrementally updating the long-term memory map, and periodically evaluating / eliminating existing candidates.

[0113] The data processing method is deployed as a corresponding data processing device within the existing dialogue architecture. The modules of this device are clearly decoupled, allowing for parallel and asynchronous collaborative work, exhibiting good fault tolerance and scalability, and achieving efficient management of long-term memory. The data and control flows between modules form a closed loop of "fast writing—scoring and filtering—asynchronous refinement—structured merging." Because the architecture reduces reliance on continuous online processing of the dialogue model (calling only when necessary and executing asynchronously), it is easier to scale horizontally to support multi-user, large-scale dialogue scenarios. The overall process is optimized to run efficiently on existing hardware, demonstrating high engineering feasibility and application promotion value.

[0114] For a description of the features in the embodiment corresponding to the data processing device 100, please refer to the relevant description in the embodiment corresponding to the data processing method, which will not be repeated here.

[0115] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described data processing method embodiments.

[0116] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above data processing method embodiments when it is run.

[0117] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0118] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above data processing method embodiments.

[0119] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above data processing method embodiments.

[0120] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0121] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be executed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microprocessor (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0122] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0123] The above provides a detailed description of a dialogue model data processing method and electronic device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A data processing method, characterized in that, include: Retrieve dialogue snippets; The dialogue fragment record is parsed to obtain a slot including at least one predefined field, and the slot is stored in the candidate memory pool. The slots in the candidate memory pool are quality-assessed to obtain a refining score, and a processing decision for the slots is determined based on the refining score; the processing decision includes at least refining and retaining. For the slot where the processing decision is to refine, the refinement task corresponding to the slot is submitted to an asynchronous task queue that is decoupled from the dialogue response link. For the refined task located in the asynchronous task queue, the dialogue model outputs a structured memory item based on the slot and the prompt fragment; The structured memory items are written into the long-term memory map.

2. The data processing method according to claim 1, characterized in that, The acquisition of the dialogue segment record specifically includes: Obtain the original fragment text, which includes the user's input information and / or the output information of the dialogue model; Based on the original text fragments, dialogue fragment records are generated, and fragment identifier codes, session identifier codes, and timestamps are assigned to the dialogue fragment records; wherein... The dialogue segment record includes at least one of the following: text, characters, and timestamps.

3. The data processing method according to claim 1, characterized in that, The process of parsing the dialogue fragment record to obtain a slot including at least one predefined field includes: The type of memorable information recorded in the dialogue segment is determined based on a lightweight text classification model; Based on a preset rule base corresponding to the memorizable information type, field information corresponding to the predefined fields in the dialogue segment records is extracted and the extraction confidence level is recorded to generate the slot; wherein, If at least one of the following conditions is met: the field information recorded in the dialogue segment is missing, cannot be determined, or is ambiguous, the extraction reliability is reduced.

4. The data processing method according to claim 1, characterized in that, The refining score includes a first refining score. The process of performing a quality assessment on the slots in the candidate memory pool to obtain a refining score, and determining the processing decision for the slot based on the refining score, includes: Based on the slot output refining score features, the refining score features include completeness features, extraction credibility features, duplication features, consistency features, and importance features; The first refining score is determined based on the refining scoring features and preset weights; If the first refining score is greater than or equal to the first refining threshold, the processing decision for the slot is determined to be refining; If the first refining score is greater than or equal to the second refining threshold and less than the first refining threshold, the processing decision for the slot is determined to be to retain it.

5. The data processing method according to claim 1, characterized in that, The refining score includes a second refining score. The process of performing a quality assessment on the slots in the candidate memory pool to obtain a refining score, and determining the processing decision for the slot based on the refining score, includes: Based on the slot output refining score features, the refining score features include completeness features, extraction credibility features, duplication features, consistency features, and importance features; The second refining score is determined based on the refining scoring characteristics and the preset model; If the second refining score is greater than or equal to the third refining threshold, the processing decision for the slot is determined to be refining.

6. The data processing method according to claim 2, characterized in that, The step of submitting the refining task corresponding to the slot for which the processing decision is refining to the asynchronous task queue decoupled from the dialogue response link includes: If the processing decision is to refine, the refinement task corresponding to the slot is submitted to the asynchronous task queue, and the slot, and the original fragment text pointed to by the fragment identifier code of the dialogue fragment record corresponding to the slot, are used as the input of the refinement task; or, Record at least one of the following: number of repetitions of the refining task, timeout policy, receipt status, and reason for failure.

7. The data processing method according to claim 1, characterized in that, The step of writing the structured memory item into the long-term memory map includes: The entities of the structured memory items are aligned to the nodes of the long-term memory graph using any one of an alias table, standardization rules, or hash signatures. If the structured memory is not being written into the long-term memory graph for the first time, update the weights and most recently mentioned timestamps of the edges corresponding to the structured memory items in the long-term memory graph. If the entity in the long-term memory graph is different from the entity in the structured memory item, the entity in the long-term memory graph is retained, the entity in the structured memory item is written, and the conflict information is recorded.

8. The data processing method according to claim 1, characterized in that, Before writing the structured memory item into the long-term memory map, the method further includes: Validate the fields and / or types of the structured memory item; Map the relation names of the structured memory items to the relation dictionary; Entity normalization is performed on the structured memory item; If the validation of the fields and / or types of the structured memory item fails, the refining task will be resubmitted to the task queue.

9. The data processing method according to any one of claims 1-8, characterized in that, The dialogue model is configured to generate output information based on the long-term memory graph and the slots in the candidate memory pool.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the data processing method as described in any one of claims 1 to 9.

Images