Methods, apparatuses, electronic devices, media, and products for processing contextual data

Abstract

This disclosure relates to the field of large language model technology, and discloses a method, apparatus, electronic device, medium, and product for processing context data. The method includes: acquiring multimodal context data corresponding to a query task; encoding the multimodal context data to obtain target memory elements corresponding to the multimodal context data, whereby the target memory elements represent metadata information corresponding to the multimodal context data; prioritizing the target memory elements to obtain their corresponding priorities; and storing the target memory elements corresponding to the multimodal context data based on their priorities. By implementing the technical solution of this disclosure, using target memory elements as indexes to store multimodal context data reduces the context window length, and storing target memory elements according to priority ensures the storage of effective multimodal context data. This effectively solves the reasoning forgetting problem caused by the limited context window in large language models, and improves the model's reasoning efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of large language model technology, specifically to methods, apparatuses, electronic devices, media, and products for processing contextual data. Background Technology

[0002] AI agents based on Large Language Models (LLMs) are limited by a fixed token length (e.g., 4K, 32K, 128K, etc.). Once the dialogue history or task document exceeds this length, the LLM model must "forget" the earliest information, resulting in context truncation and impacting its inference efficiency. Furthermore, within a limited context window, not all information is equally important; irrelevant details, outdated instructions, or casual conversation can be mixed with core task information, diluting the LLM model's attention to critical information and leading to poor inference efficiency. Summary of the Invention

[0003] This disclosure provides a method, apparatus, electronic device, medium, and product for processing contextual data to address the problem of poor model inference efficiency.

[0004] In a first aspect, this disclosure provides a method for processing context data, including: obtaining multimodal context data corresponding to a query task; encoding the multimodal context data to obtain a target memory element corresponding to the multimodal context data, wherein the target memory element represents metadata information corresponding to the multimodal context data; prioritizing the target memory element to obtain a priority corresponding to the target memory element; and storing the target memory element corresponding to the multimodal context data based on the priority.

[0005] Secondly, this disclosure provides a context data processing apparatus, comprising: an acquisition module for acquiring multimodal context data corresponding to a query task; an encoding module for encoding the multimodal context data to obtain a target memory element corresponding to the multimodal context data, wherein the target memory element represents metadata information corresponding to the multimodal context data; a priority determination module for prioritizing the target memory element to obtain a priority corresponding to the target memory element; and a context storage module for storing the target memory element corresponding to the multimodal context data based on the priority.

[0006] Thirdly, this disclosure provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the context data processing method of the first aspect or any corresponding embodiment described above.

[0007] Fourthly, this disclosure provides a computer-readable storage medium storing computer instructions for causing a computer to perform the context data processing method of the first aspect or any corresponding embodiment described above.

[0008] Fifthly, this disclosure provides a computer program product, including computer instructions for causing a computer to execute the context data processing method of the first aspect or any corresponding embodiment described above.

[0009] The method, apparatus, electronic device, medium, and product for processing context data disclosed herein encode multimodal context data to obtain target memory elements corresponding to the multimodal context data, and use these target memory elements to represent the metadata information of the multimodal context data. Priority evaluation is performed on the target memory elements, and the target memory elements corresponding to the multimodal context data are stored according to the priority evaluation results. Therefore, by using the target memory elements as indexes to store multimodal context data, the context window length is reduced, effectively solving the reasoning forgetting problem caused by the limited context window in large language models, thereby improving model reasoning efficiency. Furthermore, by storing target memory elements according to priority to store effective multimodal context data, model reasoning efficiency is further improved. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the specific embodiments of this disclosure or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this disclosure;

[0012] Figure 2 This is a schematic flowchart of a first method for processing context data according to an embodiment of the present disclosure; Figure 3 This is a schematic diagram of the generation process of the target memory element according to an embodiment of the present disclosure; Figure 4 This is a second flowchart illustrating a method for processing context data according to an embodiment of the present disclosure; Figure 5 This is a schematic diagram of a third method for processing context data according to an embodiment of the present disclosure; Figure 6 This is a schematic diagram of the memory element recall process according to an embodiment of the present disclosure; Figure 7This is a system architecture block diagram of a conventional intelligent agent according to embodiments of the present disclosure; Figure 8 This is a system structure block diagram of an intelligent agent according to an embodiment of the present disclosure; Figure 9 This is a structural block diagram of a context data processing apparatus according to embodiments of the present disclosure; Figure 10 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this disclosure. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0014] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0015] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0016] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0017] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0018] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0019] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this disclosure, "a plurality of" means two or more, unless otherwise expressly specified.

[0020] LLM models are limited by a fixed token length. Once the dialogue history or task document exceeds this length, the model must "forget" the earliest information, resulting in context truncation. This causes LLM-based AI agents to repeat answers to previously solved questions or forget key constraints raised by the user at the beginning of the dialogue, exhibiting "amnesia." In other words, AI agents struggle to handle extremely long documents, complex projects, or long-term, deep dialogues.

[0021] Because not all information is equally important within a limited context window, irrelevant details, outdated instructions, or casual conversation can be mixed with core task information, leading to a decline in the model's reasoning quality. The model may be misled by irrelevant information, produce off-topic responses, or even "illusion" of non-existent facts.

[0022] Most current context windows use a "first-in, first-out" (FIFO) sliding window mechanism, which makes it difficult to intelligently determine which information is high-value and needs to be retained, and which is low-value and can be forgotten. This may lead to the forgetting of key user preferences, while retaining irrelevant chatter, resulting in low memory management efficiency.

[0023] Moreover, processing longer contexts requires more computing resources (GPU power) and time. The computational complexity of the model is typically related to the square or higher power of the context length, and the cost increases dramatically with length. This limits the deployment of AI agents in resource-constrained environments and makes providing ultra-long context services to each user extremely expensive and difficult to scale.

[0024] Based on this, the technical solution disclosed herein encodes context information into memory elements to reduce the length of the context window, performs multi-dimensional priority scoring on memory elements, and eliminates invalid information according to priority. This effectively solves the problems of inference forgetting caused by the AI ​​Agent within a limited context window and poor inference efficiency caused by too much invalid context information, thereby improving the model's inference quality, reducing token consumption, and saving computational costs.

[0025] As one optional application scenario of this disclosure embodiment, such as Figure 1As shown, electronic device 110 has application 101 installed, and application 101 deploys an AI agent based on an LLM model. User 130 can interact with application 101 through electronic device 110 and / or access device of electronic device 110.

[0026] For example, application 101 can be any application that provides question-and-answer related services. For instance, application 101 could be a question-and-answer interactive application, such as a text-to-text application, an image-to-text application, etc. Figure 1 In the application scenario shown, if application 101 is active, electronic device 110 can display the interface 102 of application 101. Interface 102 may include various pages that application 101 can provide, such as interactive pages, settings pages, query pages, etc.

[0027] In some embodiments, electronic device 110 is communicatively connected to server 120 to provide services to application 101. Electronic device 110 may be a mobile terminal, fixed terminal, or portable terminal, etc., including but not limited to mobile phones, desktop computers, laptop computers, multimedia tablets, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices, or any combination thereof. In some embodiments, electronic device 110 may also support any type of interface, and server 120 may be various types of computing systems or servers capable of providing computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, etc.

[0028] It should be noted that, Figure 1 This is merely an example of an application scenario and does not limit the scope of protection of this disclosure.

[0029] The embodiments of this disclosure will now be described with reference to the accompanying drawings. It should be understood that the pages shown in the drawings are merely examples, and various page designs are possible in practice. The various graphic elements on the page may have different arrangements and different visual representations, one or more elements may be omitted or replaced, and one or more other elements may also be present; no limitations are imposed on the embodiments described in this disclosure. Furthermore, the embodiments are primarily described below with reference to electronic device 110. It should be understood that the actions described with respect to electronic device 110 can be performed by application 101 on electronic device 110, or can be performed by application 101 in conjunction with its server (e.g., server 120).

[0030] According to an embodiment of this disclosure, a method for processing context data is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0031] This embodiment provides a method for processing context data, which can be used in the aforementioned electronic devices, such as mobile phones, tablets, and computers. Figure 2 This is a flowchart of a method for processing context data according to embodiments of this disclosure, such as... Figure 2 As shown, the process includes the following steps: Step S201: Obtain the multimodal context data corresponding to the query task.

[0032] Multimodal context data refers to the context data corresponding to a user-triggered query task. This multimodal context data may include user instructions and document paragraphs in the text modality, speech segments and ambient sounds in the audio modality, and images, interfaces, and videos in the visual modality.

[0033] like Figure 3 As shown, the intelligent agent receives a query task triggered by the application's interactive page, perceives and decomposes the query task, and obtains the query request carried by the query task. The intelligent agent includes an Agent module, a dynamic working memory module, and a storage module.

[0034] Specifically, the Agent module searches for user-generated conversations within the application based on query requests. Recent conversations within these conversations are stored in short-term memory. Then, it extracts multimodal context data corresponding to the current query task from the conversations using different memory strategies. These memory strategies can include semantic memory strategies, summarization strategies, and user preference strategies. Semantic memory strategies extract facts and knowledge from historical conversations; summarization strategies extract conversation summaries corresponding to each historical conversation to extract key content; and user preference strategies capture information such as the current user's preferences, style, and repeated selections.

[0035] Step S202: Encode the multimodal context data to obtain the target memory element corresponding to the multimodal context data, wherein the target memory element represents the metadata information corresponding to the multimodal context data.

[0036] The target memory element is a structured, high-dimensional vector representation of multimodal context data, possessing semantic consistency and searchability. Specifically, the target memory element contains metadata information such as content, timestamp, and source corresponding to the multimodal context data. For different modalities with the same semantics, the encoded target memory elements are close in distance in the vector space, and the target memory elements support similarity matching or structured queries to support memory element retrieval.

[0037] The agent invokes the dynamic working memory module to map the modal features of the multimodal context data to a unified semantic space. In this semantic space, semantic encoding is performed according to a unified and structured encoding method to compress the multimodal context data and obtain the target memory element corresponding to the multimodal context data.

[0038] Step S203: Priority evaluation is performed on the target memory cell to obtain the priority corresponding to the target memory cell.

[0039] Priority assessment refers to evaluating the effectiveness and importance of the information carried by the target memory cell, obtaining a corresponding score, and using this score to characterize the priority of the target memory cell.

[0040] Specifically, the agent invokes the dynamic working memory module to perform a priority evaluation of the target memory cell. If the score corresponding to the priority evaluation is higher than the first preset score, it indicates that the priority level is high, that is, the information carried by the target memory cell is valid information. If the score corresponding to the priority evaluation is lower than the second preset score, it indicates that the priority level is low, that is, the information carried by the target memory cell is invalid information. If the score corresponding to the priority evaluation is higher than the second preset score but lower than the first preset score, it indicates that the priority level is medium. In this case, the information carried by the target memory cell is not processed for the time being.

[0041] Step S204: Based on priority, store the target memory cell corresponding to the multimodal context data.

[0042] Using target memory cells as indexes, and combining a hierarchical storage strategy and a dynamic update mechanism, high-value target memory cells are retained according to priority within limited storage resources. This allows the multimodal context data corresponding to these high-value target memory cells to be stored in the appropriate storage modules using short-term memory. Simultaneously, low-value target memory cells are evicted, and the multimodal context data corresponding to them is deleted.

[0043] The context data processing method provided in this embodiment encodes multimodal context data to obtain target memory units corresponding to the multimodal context data, using these target memory units to represent the metadata information of the multimodal context data. Priority evaluation is performed on the target memory units, and they are stored according to the priority evaluation results. Therefore, by using target memory units as indexes to store multimodal context data, the context window length is reduced, effectively solving the reasoning forgetting problem caused by the limited context window in large language models, thereby improving model reasoning efficiency. Furthermore, storing target memory units according to priority to store effective multimodal context data further improves model reasoning efficiency.

[0044] This embodiment provides a method for processing context data, which can be used in the aforementioned electronic devices, such as mobile phones, tablets, and computers. Figure 4 This is a flowchart of a method for processing context data according to embodiments of this disclosure, such as... Figure 4 As shown, the process includes the following steps: Step S301: Obtain the multimodal context data corresponding to the query task. For details, please refer to the relevant descriptions of the steps in the embodiments shown above, which will not be repeated here.

[0045] Step S302: Encode the multimodal context data to obtain the target memory element corresponding to the multimodal context data. The target memory element represents the metadata information corresponding to the multimodal context data.

[0046] Specifically, step S302 includes: Step S3021: Extract the first text data and non-text data from the multimodal context data.

[0047] The first text data refers to the text modal data within the multimodal context data; the non-text data refers to the non-text modal data within the multimodal context data, such as voice data and image data. Specifically, the multimodal context data corresponding to the query task is input into the agent, and the agent module within the agent parses the multimodal context data to extract the first text data and non-text data carried within it.

[0048] Step S3022: Convert the non-text data into second text data.

[0049] The second text data is a textual description of the non-text data. The agent-driven module understands the core semantics, structure, or features of the non-text data, and uses natural language to describe the non-text data, thus obtaining the corresponding second text data.

[0050] In some alternative implementations, non-text data includes at least one of audio data, image data, and video data.

[0051] For the conversion of audio data into second text data, it can identify speech audio such as human voices and non-speech audio such as environmental sounds in the audio data. Automatic speech recognition (ASR) is used to convert speech audio into corresponding text descriptions (i.e., second text data). Through audio event monitoring and audio description models, the content of non-speech audio is understood and converted into corresponding text descriptions (i.e., second text data).

[0052] For the conversion of image data to second text data, the image data is converted into the corresponding text description (i.e., second text data) by recognizing the image content.

[0053] For the conversion from video data to second text data, the spatial content and temporal dynamics corresponding to the video data are identified, and image descriptions are performed on the keyframes of the video data to generate frame-level text. Speech transcription or environmental sound description is performed on the audio track to generate audio text. Then, the frame-level text and audio text are fused in chronological order to obtain the global descriptive text corresponding to the video data (i.e., the second text data).

[0054] Step S3023: Based on a preset structured encoding method, the first text data and the second text data are encoded to obtain the first code corresponding to the first text data and the second code corresponding to the second text data.

[0055] The preset structured encoding method is a pre-defined text encoding method, such as BERT encoding. After the conversion of the second text data is completed, the multimodal context data is converted into unimodal text data. At this point, the preset structured encoding method is called to encode the first and second text data respectively, obtaining the first code corresponding to the first text data and the second code corresponding to the second text data.

[0056] Step S3024: Based on the fusion result of the first encoding and the second encoding, the target memory element is obtained.

[0057] The first and second codes are encoded and fused according to the order of the first and second text data in the multimodal context data. The fused encoding vector, the text description of each modality, and the metadata information are combined to generate a standardized and structured target memory element.

[0058] In another optional implementation, multimodal context data is parsed, and the original inputs such as text, speech, and image data within the multimodal context data are processed separately to complete preprocessing such as speech-to-text and image frame extraction. Then, the BERT model is used to extract high-dimensional feature vectors for the text modality, the Wav2Vec2.0 model to extract high-dimensional feature vectors for the speech modality, and the CLIP model to extract high-dimensional feature vectors for the image modality. Subsequently, the features of different modalities are mapped to a unified vector space for cross-modal alignment, fusing the features of different models into a single vector in the unified vector space. This fused single vector, along with the text descriptions and metadata of each modality, generates a standardized target memory element.

[0059] Step S303: Priority evaluation is performed on the target memory cell to obtain its corresponding priority. For details, please refer to the relevant descriptions of the steps in the embodiments shown above; they will not be repeated here.

[0060] Step S304: Based on priority, store the target memory element corresponding to the multimodal context data. For details, please refer to the relevant descriptions of the steps in the embodiments shown above; they will not be repeated here.

[0061] The context data processing method provided in this embodiment identifies text data and non-text data in multimodal context data, converts non-text data into text data, and then encodes it. This achieves unified structured encoding for multimodal context data, ensures standardized generation of target memory elements, and uses target memory elements as indexes for multimodal context data to store more information in limited storage resources, thereby improving the quality of subsequent model inference.

[0062] This embodiment provides a method for processing context data, which can be used in the aforementioned electronic devices, such as mobile phones, tablets, and computers. Figure 5 This is a flowchart of a method for processing context data according to embodiments of this disclosure, such as... Figure 5 As shown, the process includes the following steps: Step S401: Obtain the multimodal context data corresponding to the query task. For details, please refer to the relevant descriptions of the steps in the embodiments shown above; they will not be repeated here.

[0063] Step S402: Encode the multimodal context data to obtain the target memory element corresponding to the multimodal context data. The target memory element represents the metadata information corresponding to the multimodal context data. For details, please refer to the relevant descriptions of the corresponding steps in the embodiments shown above, which will not be repeated here.

[0064] Step S403: Priority evaluation is performed on the target memory cell to obtain the priority corresponding to the target memory cell.

[0065] Specifically, step S403 includes: Step S4031: Based on the difference between the target memory element and the memory information in the preset database, determine the first priority evaluation result.

[0066] The pre-defined database is used to store multimodal context data corresponding to target memory elements in a long-term memory mode. This long-term memory mode is used to extract structured key information from the dialogue, retain knowledge across multiple sessions, and enable the agent module to learn information such as user preferences, facts, and summaries.

[0067] The first priority evaluation result is used to characterize the novelty of the target memory element relative to the preset database. This novelty is characterized by differences. Specifically, the target memory element is compared with persistent long-term memories in the preset database to determine the differences between the target memory element and long-term memories, such as content differences and feature differences.

[0068] By combining the differences between the target memory units and long-term memory, the target memory units are scored to obtain the first priority evaluation result of the target memory units under the novelty dimension. Specifically, the greater the difference, the more novel the information represented by the target memory unit is; the smaller the difference, the less novel the information represented by the target memory unit is.

[0069] Step S4032: Determine the second priority evaluation result based on the feedback information corresponding to the target memory element.

[0070] Feedback information is used to characterize positive feedback generated by the user for the target memory element. For example, the user explicitly indicates that they remember the multimodal context data corresponding to the target memory element, or gives positive feedback to the multimodal context data corresponding to the target memory element.

[0071] The second priority evaluation result represents the feedback generated for the target memory element. Specifically, it queries the feedback information that matches the target memory element to determine whether the feedback information corresponding to the target memory element is positive feedback, such as the user explicitly indicating that they have saved it.

[0072] By combining the target memory element with its corresponding feedback information, the target memory element is scored to obtain the second priority evaluation result of the target memory element under the user feedback dimension. Specifically, the more positive feedback information, the more important the information represented by the target memory element is; the less positive feedback, the less important the information represented by the target memory element is.

[0073] Step S4033: Based on the memory strength information represented by the target memory element, determine the third priority evaluation result.

[0074] Memory strength information is used to characterize the memory strength of the content contained in the target memory cell. For example, the multimodal context data corresponding to the target memory cell contains memory content with strong emotions (happiness, anger).

[0075] The third priority assessment result indicates the memory strength of the content contained in the target memory element. Specifically, the multimodal context data corresponding to the target memory element is parsed to determine whether the parsed multimodal context data contains content with high memory strength, such as memories of being very happy or very angry.

[0076] By combining the target memory units with their corresponding memory strength information, the target memory units are scored to obtain the third priority evaluation result under the memory strength dimension. Specifically, the more content with high memory strength, the more important the information represented by the target memory unit; the fewer content with high memory strength, the less important the information represented by the target memory unit.

[0077] Step S4034: Determine the fourth priority evaluation result based on the correlation between the target memory element and the query task.

[0078] Relevance is used to characterize the content relevance between the target memory element and the current query task. This relevance can be characterized based on the spatial similarity of feature vectors and the semantic matching degree of structured text. Specifically, the spatial similarity based on feature vectors can be determined by cosine similarity, Euclidean distance, dot product similarity, etc.; the semantic matching degree based on structured text can be determined by keyword matching, structured metadata, and cosine similarity of sentence embedding vectors, etc.

[0079] The fourth priority evaluation result indicates the degree of correlation between the target memory and the query task. Specifically, by parsing the multimodal context data corresponding to the target memory, the parsed multimodal context data is matched with the query task to determine the task relevance between the target memory and the query task.

[0080] By combining the relevance between the target memory unit and the query task, the target memory unit is scored, resulting in a fourth priority evaluation result under the task relevance dimension. Specifically, the higher the relevance, the more effective the information represented by the target memory unit; the lower the relevance, the less effective the information represented by the target memory unit.

[0081] Step S4035: Determine the priority based on the weighted fusion result of the first priority evaluation result, the second priority evaluation result, the third priority evaluation result, and the fourth priority evaluation result.

[0082] The scores representing the first priority assessment result, the scores representing the second priority assessment result, the scores representing the third priority assessment result, and the scores representing the fourth priority assessment result are weighted and merged according to their respective weights to obtain a weighted score result, which represents the priority level.

[0083] Step S404: Based on priority, store the target memory cell corresponding to the multimodal context data.

[0084] Specifically, step S404 includes: Step S4041: Adjust the context buffer space of the target memory cell based on the data length corresponding to the target memory cell.

[0085] The data length is the maximum data length of the target memory cell, which can be determined by combining the three dimensions of the target memory cell's vector feature length, structured metadata length, and text description length; the context buffer space is a buffer that stores all target memory cells corresponding to multimodal context data in multi-turn sessions, and this context buffer space is a buffer of variable length.

[0086] Specifically, the data length of the target memory cell is determined by analyzing its length across multiple dimensions, including vector features, structured metadata, and text description. The context buffer space is then adjusted based on this data length to ensure it can cover all the metadata within the target memory cell.

[0087] In a specific example, the length of the context buffer space can be equal to the sum of the data lengths of all target memories, so that the context buffer space can be automatically adjusted based on the data lengths of different target memories.

[0088] Step S4042: Based on priority, store the target memory cell using the context buffer space.

[0089] Based on the priority of the target memory element, different target memory elements are prioritized in the context buffer space, and then stored in the context buffer space in short-term memory mode according to the priority ranking result. The short-term memory mode is used to record the most recent rounds of dialogue within a session, ensuring that the agent module can remember the multimodal context data of the current dialogue.

[0090] In some optional implementations, the above method further includes: Step a1: In response to the context buffer space reaching a preset value, the first memory cell with priority level 1 and the second memory cell with priority level 2 are obtained, where the second priority level is greater than the first priority level.

[0091] Step a2: Delete the first memory cell and generate the summary information of the second memory cell.

[0092] Step a3: Store the summary information in a preset database.

[0093] The default value is the maximum storage space of the context buffer space; the second level is greater than the first level, indicating that the effectiveness and importance of the first memory cell are higher than that of the second memory cell.

[0094] When the context buffer space reaches a preset value, indicating that the context buffer space is full, the first memory cell needs to be deleted to ensure that the context buffer space can continue to store valid information. For the second memory cell, a summary information corresponding to the second memory cell will be generated and stored in the preset database in long-term memory mode.

[0095] By deleting or storing memory elements according to priority levels, invalid messages are eliminated, avoiding excessive invalid context information that leads to poor model inference efficiency. This reduces token consumption during model inference, saves inference computation costs, and expands the deployment scenarios of intelligent agents.

[0096] In some optional implementations, the above method further includes: Step b1: Obtain the task instructions triggered by the target object.

[0097] Step b2: Based on the query request carried by the task instruction, recall at least one target memory cell that matches the query request.

[0098] Step b3: Based on the multimodal context data corresponding to each target memory element, generate the task response content corresponding to the task instruction.

[0099] The task instruction is a command triggered by the target object through the application's interactive page. The agent module (Agent) perceives and decomposes the task instruction to obtain the query request carried within it. Further, this query request is sent to the dynamic working memory module, instructing it to access the context buffer space in the storage module and retrieve at least one target memory element matching the query request from the context buffer space, such as... Figure 6 As shown.

[0100] As described above, target memory units are generated by encoding multimodal context data. Here, after recalling each target memory unit, the multimodal context data corresponding to each target memory unit can be parsed and injected into the dialogue context corresponding to the current task instruction, so that the agent can call the large language model to perform reasoning and obtain the corresponding task response content.

[0101] By recalling the target memory elements that match the query request carried by the task instruction, and generating the task response content corresponding to the task instruction based on the multimodal context data corresponding to each target memory element, the problem of forgetting in large model reasoning caused by the limited context window is effectively solved, and the accuracy of task response content generation is improved.

[0102] In some alternative implementations, step b3 above includes: Step b31: Determine the priority ranking result of each target memory cell based on the priority of each target memory cell.

[0103] Step b32: Based on the priority sorting result and the multimodal context data corresponding to the target memory element, generate the task response content corresponding to the task instruction.

[0104] As described above, the priority of target memory units is determined based on multiple dimensions such as information novelty, user feedback, memory strength, and task relevance. Therefore, when recalling multiple target memory units, they can be sorted and filtered according to priority. Figure 6 As shown, based on the priority ranking results, the target memory element that best matches the query request is obtained. The multimodal context data corresponding to the filtered target memory element is injected into the dialogue context corresponding to the current task instruction, so that the agent can call the large language model to perform inference and obtain the corresponding task response content.

[0105] By using the priority sorting results and the multimodal context data corresponding to the target memory element, the task response content corresponding to the task instruction is generated, thereby effectively avoiding the problem of poor inference efficiency caused by too much invalid context information, improving the model inference quality, and thus improving the accuracy of task response content generation.

[0106] The context data processing method provided in this embodiment evaluates the priority of target memory units from multiple dimensions, including information difference, feedback information, memory strength information, and relevance, to determine the priority of the target memory units. This priority characterizes the importance and effectiveness of the target memory units, enabling the elimination of invalid information and the storage of valid information according to priority, thus improving the efficiency of memory management for target memory units. By adjusting the context buffer space in conjunction with the data length of the target memory units, a variable context buffer space is achieved, reducing the token overhead cost of a fixed buffer space. This allows large language model inference to support deep dialogue processing, avoids context truncation, and further improves the model's inference efficiency and quality for deep dialogue.

[0107] As a specific application embodiment of this disclosure, the above-mentioned context information processing method will be described in conjunction with a specific application scenario, such as... Figure 7 As shown, the intelligent agent is equipped with an agent module, an agent storage module, and an automatic memory retrieval module.

[0108] The Agent Memory module provides two modes: short-term memory and long-term memory. Short-term memory records the most recent rounds of dialogue within a session, ensuring that the Agent can "remember" the context of the current conversation. Long-term memory extracts structured key information from the dialogue, preserving knowledge across multiple sessions, enabling the Agent to "learn" information such as user preferences, facts, and summaries.

[0109] Specifically, regarding the storage of short-term and long-term memory, short-term memory is used to store the original interaction events as immediate context, while long-term memory is used to store summary knowledge extracted from the interaction events, which is then persisted in the vector database. Regarding the use of short-term and long-term memory, the Agent Memory storage module provides a retrieval API interface, allowing applications to extract relevant memories and inject them into the dialogue context when invoking LLM inference to obtain the corresponding response content.

[0110] The automatic memory retrieval module is responsible for extracting summary knowledge from short-term contexts for long-term storage based on different memory strategies. For example, it uses semantic memory strategies to extract facts and knowledge from conversations for future retrieval; it uses summarization strategies to generate conversation summaries for each session, extracting the main content; and it uses user preference strategies to capture information such as user preferences, styles, and repeated selections.

[0111] However, adopt Figure 7 For the AI ​​Agent shown, the limited context window leads to forgetfulness in reasoning of large language models, and the presence of too much invalid information in the context can easily cause poor reasoning efficiency of large models.

[0112] Based on this, a dynamic working memory module is deployed in the AI ​​Agent to implement memory element encoding, dynamic context windows, priority scoring, and memory element eviction and compression, such as... Figure 8 As shown, context information can be uniformly encoded into memory elements through a dynamic working memory module, retaining only the index to reduce the length of the context window, performing multi-dimensional priority scoring on memory elements, and eliminating invalid information according to the priority score, thereby improving the inference quality of large models.

[0113] Specifically, for memory encoding, multimodal input (text, speech, image) is received and encoded into a unified, structured "target memory element". Each target memory element contains metadata information such as content, timestamp, source, and sentiment tag.

[0114] For dynamic context windows, the token overhead of fixed windows is reduced by providing a variable-length context buffer.

[0115] Priority scoring is based on multiple dimensions to assess the importance and priority of target memory elements, including information novelty (difference from information in long-term memory), feedback information (whether the user explicitly states "remember this" or provides positive feedback), memory strength (memories containing strong emotions such as happiness or anger), and task relevance (the degree of association with the task being performed by the current agent module).

[0116] Regarding the eviction and compression of memory cells, when the context buffer is nearly full, memory cells with low priority scores are actively "forgotten" or "compressed." For memory cell information with high priority scores, a summary is generated and prepared for storage in long-term memory.

[0117] This embodiment also provides a context data processing device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0118] This embodiment provides a context data processing device, such as... Figure 9 As shown, it includes: The acquisition module 501 is used to acquire the multimodal context data corresponding to the query task.

[0119] The encoding module 502 is used to encode the multimodal context data to obtain the target memory element corresponding to the multimodal context data. The target memory element represents the metadata information corresponding to the multimodal context data.

[0120] The priority determination module 503 is used to evaluate the priority of the target memory cell and obtain the priority corresponding to the target memory cell.

[0121] The context storage module 504 is used to store the target memory cells corresponding to the multimodal context data based on priority.

[0122] In some alternative implementations, the encoding module 502 includes: The data extraction unit is used to extract the first text data and non-text data from the multimodal context data.

[0123] The conversion unit is used to convert non-text data into second text data.

[0124] The text encoding unit is used to encode the first text data and the second text data based on a preset structured encoding method to obtain the first code corresponding to the first text data and the second code corresponding to the second text data.

[0125] The encoding fusion unit is used to obtain the target memory element based on the fusion result of the first encoding and the second encoding.

[0126] In some alternative implementations, non-text data includes at least one of audio data, image data, and video data.

[0127] In some alternative implementations, the priority determination module 503 includes: The first evaluation unit is used to determine the first priority evaluation result based on the difference between the target memory element and the memory information in the preset database.

[0128] The second evaluation unit is used to determine the second priority evaluation result based on the feedback information corresponding to the target memory element.

[0129] The third evaluation unit is used to determine the third priority evaluation result based on the memory strength information represented by the target memory element.

[0130] The fourth evaluation unit is used to determine the fourth priority evaluation result based on the correlation between the target memory element and the query task.

[0131] The weighted fusion unit is used to determine the priority based on the weighted fusion result of the first priority evaluation result, the second priority evaluation result, the third priority evaluation result, and the fourth priority evaluation result.

[0132] In some alternative implementations, the context storage module 504 includes: The adjustment unit is used to adjust the context buffer space of the target memory cell based on the data length corresponding to the target memory cell.

[0133] The first storage unit is used to store the target memory cell based on priority and utilizing the context buffer space.

[0134] In some alternative implementations, the context storage module 504 further includes: The priority determination unit is used to obtain the first memory cell with the first priority level and the second memory cell with the second priority level in response to the context buffer space reaching a preset value, wherein the second priority level is greater than the first priority level.

[0135] The memory cell processing unit is used to delete the first memory cell and generate summary information for the second memory cell.

[0136] The second storage unit is used to store summary information into a preset database.

[0137] In some alternative embodiments, the above-described apparatus further includes: The instruction acquisition module is used to acquire task instructions triggered by the target object.

[0138] The recall module is used to recall at least one target memory cell that matches the query request carried by the task instruction.

[0139] The response module is used to generate task response content corresponding to task instructions based on the multimodal context data corresponding to each target memory element.

[0140] In some alternative implementations, the response module includes: The sorting unit is used to determine the priority sorting result of each target memory cell based on the priority corresponding to each target memory cell.

[0141] The content generation unit is used to generate task response content corresponding to task instructions based on priority sorting results and multimodal context data corresponding to the target memory element.

[0142] The context data processing apparatus provided in this disclosure can execute the context data processing method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of the method. By encoding the multimodal context data, target memory elements corresponding to the multimodal context data are obtained, and these target memory elements represent the metadata information of the multimodal context data. Priority evaluation is performed on the target memory elements, and the target memory elements corresponding to the multimodal context data are stored according to the priority evaluation results. Therefore, by using the target memory elements as indexes to store multimodal context data, the context window length is reduced, effectively solving the reasoning forgetting problem caused by the limited context window in large language models, thereby improving model reasoning efficiency. By storing target memory elements according to priority to store effective multimodal context data, model reasoning efficiency is further improved.

[0143] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0144] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure.

[0145] The following is a detailed reference. Figure 10This diagram illustrates a suitable structural schematic for implementing an electronic device according to embodiments of the present disclosure. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 601, which can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) 602 or a program loaded from memory 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the electronic device. The processor 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0146] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 10 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0147] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a memory 608, or installed from a ROM 602. When the computer program is executed by the processor 601, it performs the functions defined in the context information processing method of embodiments of this disclosure.

[0148] Figure 10 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0149] This disclosure also provides a computer-readable storage medium in which the methods described in this disclosure can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium may also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the context information processing methods shown in the above embodiments.

[0150] A portion of this disclosure can be applied to computer program products, such as computer program instructions, which, when executed by a computer, can invoke or provide methods and / or technical solutions according to this disclosure through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, and installation package files. Accordingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions; the computer compiling the instructions and then executing the corresponding compiled program; the computer reading and executing the instructions; or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0151] Although embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for processing context data, characterized in that, The method includes: Retrieve the multimodal context data corresponding to the query task; The multimodal context data is encoded to obtain the target memory element corresponding to the multimodal context data, and the target memory element represents the metadata information corresponding to the multimodal context data; Priority evaluation is performed on the target memory element to obtain the priority corresponding to the target memory element; Based on the priority, the target memory element corresponding to the multimodal context data is stored.

2. The method according to claim 1, characterized in that, The step of encoding the multimodal context data to obtain the target memory corresponding to the multimodal context data includes: Extract the first text data and non-text data from the multimodal context data; Convert the non-text data into second text data; Based on a preset structured encoding method, the first text data and the second text data are encoded to obtain a first code corresponding to the first text data and a second code corresponding to the second text data; The target memory element is obtained based on the fusion result of the first encoding and the second encoding.

3. The method according to claim 2, characterized in that, The non-text data includes at least one of audio data, image data, and video data.

4. The method according to any one of claims 1 to 3, characterized in that, The step of prioritizing the target memory cell to obtain its corresponding priority includes: Based on the difference between the target memory element and the memory information in the preset database, the first priority evaluation result is determined; Based on the feedback information corresponding to the target memory element, the second priority evaluation result is determined; Based on the memory intensity information represented by the target memory element, the third priority evaluation result is determined; Based on the correlation between the target memory element and the query task, the fourth priority evaluation result is determined; The priority is determined based on the weighted fusion result of the first priority evaluation result, the second priority evaluation result, the third priority evaluation result, and the fourth priority evaluation result.

5. The method according to claim 4, characterized in that, Based on the priority, the target memory element corresponding to the multimodal context data is stored, including: Based on the data length corresponding to the target memory cell, adjust the context buffer space of the target memory cell; Based on the priority, the target memory cell is stored in the context buffer space.

6. The method according to claim 5, characterized in that, Also includes: In response to the context buffer space reaching a preset value, the first memory cell with a first priority level and the second memory cell with a second priority level are obtained, wherein the second priority level is greater than the first priority level. Delete the first memory cell and generate summary information for the second memory cell; The summary information is stored in a preset database.

7. The method according to claim 1, characterized in that, Also includes: Retrieve the task instructions triggered by the target object; Based on the query request carried by the task instruction, at least one of the target memory elements that matches the query request is recalled; Based on the multimodal context data corresponding to each of the target memory elements, the task response content corresponding to the task instruction is generated.

8. The method according to claim 7, characterized in that, The step of generating the task response content corresponding to the task instruction based on the multimodal context data corresponding to each of the target memory elements includes: Based on the priority of each target memory element, the priority ranking result of each target memory element is determined; Based on the priority sorting result and the multimodal context data corresponding to the target memory element, the task response content corresponding to the task instruction is generated.

9. A context data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire the multimodal context data corresponding to the query task; An encoding module is used to encode the multimodal context data to obtain a target memory element corresponding to the multimodal context data, wherein the target memory element represents the metadata information corresponding to the multimodal context data; The priority determination module is used to evaluate the priority of the target memory element and obtain the priority corresponding to the target memory element. A context storage module is used to store the target memory element corresponding to the multimodal context data based on the priority.

10. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the method for processing context data according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the method for processing context data according to any one of claims 1 to 8.

12. A computer program product, characterized in that, Includes computer instructions for causing a computer to perform the method for processing context data according to any one of claims 1 to 8.

Images