Efficient conversational retrieval method and device based on memory-enhanced query intent understanding

Abstract

The application discloses an efficient dialogue retrieval method and device based on memory-enhanced query intention understanding. In the multi-round dialogue retrieval process, a fixed-length learnable memory vector group is introduced to cross-round progressive aggregation of user query intention representation. Meanwhile, a historical dialogue semantic key-value library is constructed. The similarity between the current memory state and the historical dialogue semantic key-value library is evaluated to selectively recall the forgotten historical semantics and reinforce the current memory, so as to suppress semantic drift and information forgetting. In addition, the candidate results obtained in the last round of retrieval are used as visual feedback to guide the retrieval of the current round of query representation, thereby enhancing the cross-round continuity and improving the retrieval accuracy. The application can guarantee the accuracy of multi-round retrieval, significantly reduce the repeated coding and reconstruction overhead of dialogue history, and realize efficient and stable dialogue retrieval.

Description

Technical Field

[0001] This invention relates to the field of video cross-modal retrieval technology, and in particular to an efficient conversational retrieval method and apparatus based on memory-enhanced query intent understanding. Background Technology

[0002] With the rapid growth of multimedia content on the internet and the widespread adoption of smart terminals, the demand for retrieving multimedia data such as images, videos, and documents is constantly increasing. In actual retrieval processes, users often struggle to accurately and completely express the attributes, categories, scenes, relationships, and other detailed constraints of the target object in a single round of input. Retrieval systems typically need to gradually understand the user's intent through multiple rounds of interaction, such as by asking for limiting conditions like "color / material / action / scene," or by continuously refining the query based on the user's further description of candidate results, thus forming a conversational retrieval process. The goal of conversational retrieval is to continuously update and accurately understand the user's intent in multiple rounds of interaction, making the candidate results output in each round closer to the user's actual needs, while ensuring the real-time performance and scalability of the system response. However, conversational retrieval differs significantly from traditional single-round retrieval. First, information within a conversation round is incrementally input, and users may introduce new constraints in subsequent rounds. Second, while information from early rounds often plays a crucial role in the final intent, its importance may be obscured by subsequent information. Therefore, conversational retrieval not only requires the model to have cross-round semantic understanding capabilities but also stability and high efficiency in long conversation scenarios. Existing conversational retrieval methods for understanding users' multi-turn query intent can be broadly categorized into two types: First, concatenating the current turn's text with historical turn's text and encoding it as a whole before matching it with candidate images; second, using existing large models to rewrite the multi-turn dialogue content and merge it into a complete query text to fit traditional retrieval frameworks. However, these approaches still have significant shortcomings. Concatenating dialogue history causes the input sequence to lengthen with each turn, leading to redundant computation, increased latency, and semantic dilution of long sequences. While query rewriting can shorten the text input, the introduction of large models during the rewriting process incurs additional computational overhead and may result in information omissions or semantic drift. Therefore, there is an urgent need for an efficient conversational retrieval method and device that can continuously model user intent with low computational overhead in multi-turn dialogues, selectively complete forgotten information, and effectively utilize historical retrieval results feedback to achieve query refinement. Summary of the Invention

[0003] The purpose of this invention is to address the problems of repeated encoding of historical text, large computational cost of reconstructing historical dialogue text in large models, and weak continuity across rounds in conversational retrieval, and to propose an efficient conversational retrieval method and device based on memory-enhanced query intent understanding.

[0004] The objective of this invention is achieved through the following technical solution: an efficient conversational retrieval method based on memory-enhanced query intent understanding, comprising: Construct a conversational retrieval network model, where each round of dialogue includes: Feature extraction is performed on the image to be retrieved, and feature encoding is performed on the text of the current retrieval round; Maintain a set of learnable memory features of fixed length, and use the memory features to interact and update the text features to obtain the memory features of the current dialogue round; The features of this round of dialogue are interactively encoded and aggregated with the updated memory features to obtain key features. The global representation vector is used as the value corresponding to the key, and the key-value pairs are stored in the historical dialogue semantic library. Retrieve historical representations that are semantically distant from the current memory from the historical dialogue semantic database, reintegrate them into the current memory, enhance the memory features, introduce the results of the previous round of retrieval, and refine the query representation of the current round. The search results for the current round are obtained by ranking the similarity based on the features of the image to be searched and the final query representation of the current round. The model is trained by inputting video and multi-turn dialogue text into the trained model to perform conversational retrieval.

[0005] Furthermore, the step of extracting features from the image to be retrieved and encoding features for the text in the current retrieval round includes: For a candidate image library consisting of several images, a pre-trained BLIP model is used to extract the feature vector set of each candidate image; For the current number During round-robin queries, given a query dialogue consisting of several words, a pre-trained BLIP model is used to extract a set of feature vectors for each word.

[0006] Furthermore, maintaining a fixed-length set of learnable memory features includes: A set of memory features is initialized in the first round of dialogue; The memory features and the first-round query text features are input into the transformer. The memory features are used as the query, and the query dialogue features are used as both the key and value. They are interactively encoded to obtain memory features with attached query text semantics. In subsequent rounds, the memory features from the previous round and the query text features from the current round are input into the transformer for interactive encoding.

[0007] Furthermore, the key-value pairs of the historical dialogue semantic library include: The semantic features of the attached query text are used as the query, and the text features of the words are used as both the key and value. These are then input into the transformer for interactive encoding to obtain the encoded key features. The weight vector is multiplied by the encoded key feature and passed through a Softmax layer to obtain the weight corresponding to each encoded key feature vector. The weighted aggregation is then used to obtain the final key feature of the current round. The global representation vector is the first and second digits of the word feature of the retrieved round dialogue, which serves as the value corresponding to the key feature of the current round.

[0008] Furthermore, the retrieval of historical representations from the historical dialogue semantic database that are semantically distant from the current memory includes: By using an attention pooling method with the same key features, the memory features of the current round are aggregated as a global representation of the current memory state; Calculate the similarity between the global representation of the current memory state and the key portion of all stored data in the historical dialogue semantic database; select the n stored data with the lowest similarity as semantic supplements; Meanwhile, for the selected semantic supplementary data Normalized recall weights are calculated based on the inverse relationship of similarity. ,Right now: in, This is a global representation of the current memory state. This refers to the key portion of the data stored in the historical dialogue semantic library.

[0009] Furthermore, the enhancement of memory features includes: For the selected semantic supplementary data The value part corresponding to each data item By recall weight The aggregated data forms the final forgotten information retrieval representation. The memory features of the current retrieval round serve as the query, and the forgotten information retrieval representation serves as both the key and value. This is achieved through cross-attention with the memory features of the current retrieval round. Interactive encoding is performed to obtain the enhanced memory features for the current retrieval round.

[0010] Furthermore, the refinement of the current query representation by incorporating the search results from the previous round includes: Take the first The top k image results in the final search ranking during the search round and features of all the images to be retrieved Select the corresponding image features to form the candidate image feature set for the previous retrieval round. ; Through a fully connected layer Projected onto the currently enhanced memory features The same dimensions, yield ; As a query, Simultaneously, these are used as both key and value inputs into the transformer for interactive encoding, resulting in the final memorized features. ; The final memory features The data is further aggregated into a global query vector for the current retrieval round using the same attention pooling method as the key features, and then projected onto the same embedding dimension as the image features to obtain the final query vector for the current retrieval round.

[0011] Furthermore, training the model includes: The model is trained end-to-end, and contrastive learning loss is used to align the final query text representation with the visual features of its corresponding image; in each retrieval round In this process, the model calculates a retrieval loss that makes the query representation of the current retrieval round closer to its matching image, while distancing it from the non-matching samples in the same mini-batch. in This represents the size of a small batch of data. or This represents the query text and images that are positive samples to each other in this batch. or This represents the query text and images that are negative samples to each other in this batch. Temperature coefficient; Final training loss This represents the average loss across all dialogue rounds.

[0012] On the other hand, a high-efficiency conversational retrieval device based on memory-enhanced query intent understanding is also provided, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements the high-efficiency conversational retrieval method based on memory-enhanced query intent understanding.

[0013] On the other hand, a computer-readable storage medium is also provided, on which a program is stored, which, when executed by a processor, implements the aforementioned efficient conversational retrieval method based on memory-enhanced query intent understanding.

[0014] The beneficial effects of this invention are: This invention constructs a persistent and lightweight semantic memory mechanism. During multi-turn dialogue retrieval, a fixed-length memory representation is used to accumulate and refine user query intent across turns. This allows the current query to integrate new semantics and maintain existing intent simply by interacting with existing memory, thus avoiding the high overhead and long-sequence dilution problems caused by repeatedly encoding or reconstructing the entire dialogue history. Furthermore, considering that increasing the number of dialogue turns can lead to semantic drift or partial forgetting, this invention further stores the semantics of each turn of dialogue in the form of historical context. Based on similarity, it assesses the degree of forgetting and selectively recalls historical semantics with high information content to strengthen the current memory state and restore gradually fading information. This invention addresses the limitations of existing methods that treat each round as an independent retrieval step and neglect the continuity of feedback between rounds. It introduces a result-driven visual feedback mechanism, utilizing visual semantic cues from the previous round's candidate results to guide the formation and refinement of the next round's query representation, making cross-round intent modeling more coherent and retrieval positioning more accurate. In summary, this invention effectively improves the accuracy and coherence of multi-round dialogue retrieval while maintaining a near-constant increase in overall computational cost, making it suitable for efficient retrieval applications in online interaction and large-scale candidate databases. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of an efficient conversational retrieval method based on memory-enhanced query intent understanding provided by the present invention.

[0016] Figure 2 This is a schematic diagram illustrating the computational efficiency in multi-turn dialogic image retrieval according to an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating a retrieval example according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an efficient conversational retrieval device based on memory-enhanced query intent understanding provided by the present invention. Detailed Implementation

[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0018] To address the common problems in conversational retrieval in real-world interactive scenarios, such as user intent being supplemented and easily changing in each round, increased processing overhead due to dialogue history growth, easy overwriting of early key information in long conversations, and lack of continuous utilization of result feedback in each round of retrieval, this invention proposes an efficient conversational retrieval method based on memory-enhanced query intent understanding. By maintaining a lightweight and persistent semantic state during the conversation and combining it with selective recall and result feedback guidance, stable modeling of user intent and efficient retrieval are achieved.

[0019] This implementation takes a single conversational retrieval session as its object, describing the data flow and calling relationships between modules according to the processing flow of "input acquisition, representation generation, state update, result output, and state maintenance." In each round of interaction, firstly, to avoid repeatedly concatenating and encoding all historical dialogues, which would cause redundant calculations and reduce real-time response capabilities, the system only receives the user's text input for the current round and simultaneously obtains the candidate results returned from the previous round of retrieval as usable feedback information. Subsequently, the text encoding module encodes the text of the current round to obtain the text feature representation used for subsequent interaction modeling. Furthermore, to continuously retain and accumulate cross-round intent information without significantly increasing the computational load with the number of dialogue rounds, the system maintains a set of fixed-length semantic memory units to carry long-term intent states during the dialogue process. At the same time, to solve the problem of early information decay and overwriting that may occur in long dialogues when relying solely on fixed-capacity memory, the system also maintains a historical dialogue semantic repository. In this repository, historical dialogue semantics are stored in the form of key-value pairs, where the key is used to match the relevance with the current memory state, and the value is used to provide the semantic content of the corresponding round for subsequent completion. Next, to achieve incremental updates that add new semantic information without disrupting accumulated intent, the semantic memory update module interacts with the semantic memory units and the current round's text features to generate an updated memory state. Subsequently, to promptly retrieve potentially weakened key information as dialogue rounds increase and semantic information is continuously added, the forgetting assessment and recall module selectively retrieves several historical semantic contents from the historical semantic repository as recall information based on the similarity between the current memory state and historical keys. Further, to effectively integrate the recalled historical semantics into the current state, thereby achieving memory supplementation and reinforcement, the recall information is fused with the current memory state to obtain a reinforced semantic memory state. Then, to overcome the problem that existing conversational retrieval often processes each round of retrieval in a fragmented manner and fails to fully utilize the visual and semantic cues contained in the previous round's candidate results, the visual feedback refinement module extracts and maps features from the previous round's candidate results and interacts and fuses them with the reinforced semantic memory state to generate the final query representation for retrieval. Based on the similarity calculation and ranking between the final query representation and the candidate image feature representations, the current round's retrieval results are output. Finally, to ensure the session state can continuously evolve and support the next round of interaction, the system writes the historical semantic key-value pairs generated in this round of dialogue into the historical semantic repository and updates the candidate result feedback information from the previous round for reuse in the next round of interaction. For example... Figure 1 As shown, the specific steps of this invention are as follows: (1) Perform feature pre-extraction on the image to be retrieved to obtain the feature representation of the image to be retrieved.

[0020] Given a by A candidate image library consisting of 100 images is used to extract the feature vector set of each candidate image using a pre-trained BLIP model. , Indicates the first The feature vector of an image.

[0021] (2) Encode the current round text with features to obtain the feature representation of the current round text.

[0022] For the current number During round-robin queries, given by The query dialogue consists of 10 words, and a pre-trained BLIP model is used to extract the feature vector set of each word. As the initial feature of the text, Indicates the first Wheel of Life The feature vector of each query dialogue word.

[0023] (3) Initialize and maintain a set of learnable memory features of fixed length, and construct a semantic memory update module to enable the memory vector to interact and update with the text feature representation of the current round to obtain the semantic memory representation of the current round. The specific steps are as follows: (3-1) Initialize a set of lengths... Memory characteristics , here Set to 32 Indicates the first Each memory feature vector.

[0024] (3-2) Starting from the first round of retrieval, the memory vector initialized in step (3-1) and the first round query text features obtained in step (2) are combined. The inputs are interactively encoded in a transformer to obtain memory features with the semantics of the first-round query text. ,Right now: In the above formula This represents a cross-attention transformer module obtained by modifying the input of a standard transformer module. The inputs in the standard transformer module are used as query / key / value pairs, respectively. In the middle, initial memory characteristics As a query, the first-round query dialogue characteristics It can be used as both a key and a value.

[0025] In subsequent search rounds In this step, the first step is changed to obtain the first step from step (2). Round-robin dialogue features With the Memory characteristics of the turn Input them into the transformer for interactive encoding. As a query, Simultaneously used as both key and value, it yields the accompanying... Memory features of round-robin query dialogue semantics ,Right now: (4) To preserve long-term dialogue semantics for later recall, a historical dialogue semantic database is first constructed to store semantic information from all dialogue rounds for subsequent retrieval. Each unit in this database is organized in a dictionary data format of (key:value), where the key is used to evaluate the relevance of the corresponding dialogue to the current memory state by calculating similarity, and the value stores the corresponding dialogue representation for recall. Taking the search wheel as an example, the specific steps are as follows: (4-1) To generate a reliable key, the features of this round of dialogue are enhanced by employing an interactive modeling process. Semantic comparability with memory features, rather than direct comparison. Perform aggregation. Specifically, first... Memory features with initial query text semantics obtained in step (3-2) Input them into the transformer for interactive encoding. As a query, Both are used as key and value to obtain features. ,Right now: (4-2) The features obtained in step (4-1) are processed through an attention mechanism. Aggregates into a single feature vector. Uses a learnable weight vector. and Perform dot product and pass Layer, to obtain Weights Use the calculated weights on the features Perform a weighted sum to obtain the final key features. ,Right now: (4-3) The first Retrieval turn dialogue features The first element (i.e., the global representation vector) The key obtained in step (4-2) The corresponding value. This yields the final storage unit for this retrieval round. . to the During round-robin retrieval, the stored content of the historical dialogue semantic database is as follows: (5) As the dialogue rounds progress, due to long-term continuous updates, the semantics of historical dialogues from a distant past may gradually fade from current memory. To alleviate long-term forgetting, these historical representations that are semantically distant from current memory are retrieved from the historical dialogue semantic database and reintegrated into current memory to achieve semantic integrity. (Based on the first...) Taking the search wheel as an example, the specific steps are as follows: (5-1) Using the same attention pooling method as in step (4-2), the memory features of the current round are processed. Aggregate to obtain , which serves as a global representation of the current memory state.

[0026] (5-2) To quantify the degree of forgetting for each historical dialogue, calculate the global representation of the current memory state. Semantic Library for Dialogue with History The key portion of all stored data The similarity is determined by the number of historical conversations that have low similarity to current memories. These conversations are more likely to contain forgotten or overwritten information. Therefore, the n most similar stored data entries are selected as semantic supplements. The value of n is set to 2. Meanwhile, for the selected semantic supplementary data... Normalized recall weights are calculated based on the inverse relationship of similarity. ,Right now: (6) For the semantic supplementary data selected in step (5-2) The value part corresponding to each data item The weights obtained in step (5-2) Aggregation into the final forgotten information retrieval representation And through cross-attention and memory features of the current retrieval round Interactive encoding is performed to obtain the enhanced memory features for the current retrieval round. , As a query, It can be used as both a key and a value, that is: (7) To further enhance the semantic continuity between retrieval rounds, retrieval results from the previous round are introduced to help refine the query representation of the current round. This mechanism enables the model to utilize accumulated visual evidence as contextual guidance, thereby improving the consistency and coherence of multi-round retrievals. (The last sentence appears to be incomplete and requires further context.) Taking the search wheel as an example, the specific steps are as follows: (7-1) Take the first The top k image results in the final search ranking during the search round (k is set to 100), and features of all candidate images obtained in step (1) Select the corresponding image features to form the candidate image feature set for the previous retrieval round. .

[0027] (7-2) Through a fully connected layer (FC) Projected onto the currently enhanced memory features The same dimensions, yield And will and Input them into the transformer for interactive encoding. As a query, Simultaneously serving as both key and value, the final memory features are obtained. Right now: (7-3) Final memory features The query vector for the current retrieval round is further aggregated using the same attention pooling process as in step (4-2), and projected onto the same embedding dimension as the image features to obtain the final query vector for the current retrieval round. .

[0028] (8) Calculation Features of all candidate images obtained in step (1) The cosine similarity is calculated, and candidate images are sorted from highest to lowest similarity. The top 100 images are then used as visual feedback for the next retrieval round.

[0029] (9) The model is trained end-to-end, and contrastive learning loss is used to align the final query text representation with the visual features of its corresponding image. In each retrieval round... In this process, the model calculates a retrieval loss that makes the query representation of the current retrieval round closer to its matching image, while distancing it from the non-matching samples in the same mini-batch. in This represents the size of a small batch of data. or This represents the query text and images that are positive samples to each other in this batch. or This represents the query text and images that are negative samples to each other in this batch. This is the temperature coefficient.

[0030] Final training loss The average loss across all dialogue rounds is: Where R represents the total number of dialogue rounds. This round-averaging optimization method ensures that the model learns a consistent and stable retrieval capability over multiple rounds of interaction, without overfitting to any single round.

[0031] (10) A well-trained, efficient conversational retrieval network model was obtained through the above steps. The video and multi-turn dialogue text were input into the trained model, and steps (2) to (8) were executed in a loop to complete the conversational retrieval.

[0032] Implementation Examples This invention selects a widely used visual dialogue dataset (Visual Dialog) to train the proposed network model, thereby demonstrating the effectiveness of the invention in practical applications. Each image in the visual dialogue dataset is paired with an initial query and ten rounds of dialogue. The dataset contains 123,287 training images and 2,064 validation images.

[0033] Previous work has primarily used Hit@10 as a model evaluation metric, which measures the proportion of samples in which a target image appears in the top-10 search results at any given dialogue round. However, using Hit@10 alone cannot truly reflect the real-time retrieval accuracy of each round. Because it accumulates successes from all historical rounds, once a target image ranks in the top 10 in similarity at a given round, it is considered successfully retrieved in subsequent rounds by definition, even if its ranking degrades later. Therefore, some models may achieve high Hit@10 scores even if their round-by-round retrieval quality is unstable (e.g., frequent ranking changes or occasional large fluctuations). To more accurately evaluate round-by-round retrieval quality, this invention also uses Recall@10, which measures only the proportion of samples in which a target image is ranked in the top 10 at the current search round. Furthermore, to strike a balance between cumulative success and real-time accuracy, this invention reports the mean of Hit@10 and Recall@10 at each round, denoted as MHR@10. Finally, this invention also reports the average performance over 10 rounds as the overall evaluation result.

[0034] Table 1. Performance comparison of the current method and the latest multi-turn conversational retrieval methods First, the framework of this invention is compared with representative state-of-the-art multi-turn conversational retrieval methods on the VisDial dataset. In addition to evaluating the original dialogue text on VisDial, the method is also evaluated on dialogue text reconstructed using a large model derived from this dataset to obtain a more comprehensive performance assessment. Table 1 summarizes the MHR@10 scores of each model across ten retrieval rounds. Overall, the method of this invention achieves the highest accuracy across all rounds and dialogue types, demonstrating its powerful multi-turn query understanding and reasoning capabilities. In the original dialogue setting, the best-performing method, ChatIR, directly concatenates the complete dialogue history with the current query. While this provides some initial benefits, it quickly introduces redundant or conflicting contexts. Unlike concatenation-based designs, this invention progressively integrates newly introduced user query intent, thus achieving more stable and continuous performance improvements throughout the dialogue process. In the reconstructed dialogue setting, the multi-turn conversational retrieval method PlugIR, based on large model reconstructed data, shows limited growth in the MHR@10 metric, indicating a gradual decline in its retrieval accuracy in later rounds. This also reveals a key limitation of Hit@10: its cumulative nature often masks the accuracy degradation in subsequent rounds. In contrast, the method of this invention maintains a consistent upward trend across all rounds, demonstrating its advantage in continuously understanding query intent and maintaining retrieval accuracy throughout the dialogue evolution.

[0035] Table 2 Comparison of the effectiveness of each module of the model The contribution of each module in this invention was evaluated through ablation experiments. A model containing only the progressive dialogue semantic memory module was used as the baseline, and historical semantic recall and inter-round retrieval result feedback modules were gradually added. The experimental results are shown in Table 2. Above the baseline, adding either historical semantic recall or inter-round retrieval result feedback alone significantly improved performance, indicating that they are complementary in enhancing memory retention and cross-round semantic continuity. Furthermore, the model achieved the highest performance after simultaneously introducing both modules, verifying that these two modules complement each other and synergistically improve the overall multi-round retrieval effect.

[0036] To evaluate the computational efficiency of this invention in multi-turn conversational image retrieval, Figure 2 A joint analysis was conducted to examine the growth of the average number of encoded characters and the corresponding computational cost across dialogue rounds. The average encoded character length was defined as the average number of text characters encoded per round on the VisDial test set. Figure 2As shown, the multi-turn dialogue concatenation method continuously appends historical dialogues to the query text, increasing the number of query characters from 25 to 122, and raising the encoding computation to nearly five times the initial value in the 10th round. The large model reconstruction method further amplifies this trend; the prompts required for large model reconstruction, along with the text generated by the large model, lead to a sharp increase in the length of the encoded characters and result in trillions of computational costs, thus forming the heaviest computational burden. In contrast, this invention only encodes the current round query and accumulates dialogue semantics through a fixed number of memory vectors, keeping the encoding length and computational cost almost constant throughout the dialogue process. This demonstrates that this invention can effectively suppress the increase in computational cost with each round and achieve stable computational efficiency in multi-turn conversational image retrieval.

[0037] Figure 3 This is an example of the network model proposed in this invention being trained and used for retrieval on a visual dialogue dataset.

[0038] Figure 3 The example above (a) illustrates the conversational retrieval capabilities of the model of this invention and the contribution of the memory recall mechanism. In rounds 5 and 10 of the dialogue, the model of this invention is able to retrieve semantically accurate results, while variants without the recall mechanism tend to over-focus on the latest round of dialogue content. Specifically, when the round 5 dialogue is "Where are they standing? On the train platform?", the model without recall overemphasizes the visual semantics "train platform" and ignores historical semantics, resulting in intent drift. With memory recall enabled, the model of this invention is able to retrieve the supplementary historical information "two men", thereby recovering forgotten information, constructing a more complete query representation, and improving retrieval accuracy. A similar trend was observed in round 10, further demonstrating that memory recall can effectively enhance long-term semantic integrity in multi-round dialogues.

[0039] exist Figure 3In the example below (b), although the query rewriting model produces relatively reasonable retrieval results in round 5, its performance drops significantly in round 10. This degradation stems from its query rewriting mechanism based on a large language model, where the reconstruction process may gradually deviate from the user's true intent as the conversation progresses. For example, by round 10, the reconstructed query introduces irrelevant attributes such as "a white table and a white chair," leading the model to a mismatched visual concept. Once these illusory details enter the reconstructed text, the retrieval model is forced to match along incorrect clues, ultimately resulting in semantic drift and incorrect ranking results. Compared to these baseline methods, the model of this invention maintains coherent intent understanding across multiple rounds of dialogue. By selectively recalling forgotten semantics and suppressing drift, it outputs more stable and accurate retrieval results throughout the entire dialogue process. Corresponding to the aforementioned embodiments of efficient conversational retrieval methods based on memory-enhanced query intent understanding, this invention also provides embodiments of efficient conversational retrieval devices based on memory-enhanced query intent understanding.

[0040] Corresponding to the aforementioned embodiment of an efficient conversational retrieval method based on memory-enhanced query intent understanding, the present invention also provides an embodiment of an efficient conversational retrieval device based on memory-enhanced query intent understanding.

[0041] See Figure 4 The present invention provides an efficient conversational retrieval device based on memory-enhanced query intent understanding, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement an efficient conversational retrieval method based on memory-enhanced query intent understanding as described in the above embodiment.

[0042] The present invention provides an embodiment of a high-efficiency conversational retrieval device based on memory-enhanced query intent understanding. This embodiment can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data-processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 4 The diagram shown is a hardware structure diagram of any device with data processing capabilities, in which the efficient conversational retrieval device based on memory-enhanced query intent understanding provided by the present invention is located. (Except for...) Figure 4In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0043] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0044] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0045] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements an efficient conversational retrieval method based on memory-enhanced query intent understanding as described in the above embodiments.

[0046] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0047] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned efficient conversational retrieval method based on memory-enhanced query intent understanding.

[0048] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0049] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. This application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A highly efficient conversational retrieval method based on memory-enhanced query intent understanding, characterized in that, include: Construct a conversational retrieval network model, where each round of dialogue includes: Feature extraction is performed on the image to be retrieved, and feature encoding is performed on the text of the current retrieval round; Maintain a set of learnable memory features of fixed length, and use the memory features to interact and update the text features to obtain the memory features of the current dialogue round; The features of this round of dialogue are interactively encoded and aggregated with the updated memory features to obtain key features. The global representation vector is used as the value corresponding to the key, and the key-value pairs are stored in the historical dialogue semantic library. Retrieve historical representations that are semantically distant from the current memory from the historical dialogue semantic database, reintegrate them into the current memory, enhance the memory features, introduce the results of the previous round of retrieval, and refine the query representation of the current round. The search results for the current round are obtained by ranking the similarity based on the features of the image to be searched and the final query representation of the current round. The model is trained by inputting video and multi-turn dialogue text into the trained model to perform conversational retrieval.

2. The efficient conversational retrieval method based on memory-enhanced query intent understanding according to claim 1, characterized in that, The process of extracting features from the image to be retrieved and encoding features for the text in the current retrieval round includes: For a candidate image library consisting of several images, a pre-trained BLIP model is used to extract the feature vector set of each candidate image; For the current number During round-robin queries, given a query dialogue consisting of several words, a pre-trained BLIP model is used to extract a set of feature vectors for each word.

3. The efficient conversational retrieval method based on memory-enhanced query intent understanding according to claim 1, characterized in that, Maintaining a fixed-length set of learnable memory features includes: A set of memory features is initialized in the first round of dialogue; The memory features and the first-round query text features are input into the transformer. The memory features are used as the query, and the query dialogue features are used as both the key and value. They are interactively encoded to obtain memory features with attached query text semantics. In subsequent rounds, the memory features from the previous round and the query text features from the current round are input into the transformer for interactive encoding.

4. The efficient conversational retrieval method based on memory-enhanced query intent understanding according to claim 1, characterized in that, The key-value pairs of the historical dialogue semantic database include: The semantic features of the attached query text are used as the query, and the text features of the words are used as both the key and value. These are then input into the transformer for interactive encoding to obtain the encoded key features. The weight vector is multiplied by the encoded key feature and passed through a Softmax layer to obtain the weight corresponding to each encoded key feature vector. The weighted aggregation is then used to obtain the final key feature of the current round. The global representation vector is the first and second digits of the word feature of the retrieved round dialogue, which serves as the value corresponding to the key feature of the current round.

5. The efficient conversational retrieval method based on memory-enhanced query intent understanding according to claim 1, characterized in that, The process of retrieving historical representations from the historical dialogue semantic database that are semantically distant from the current memory includes: By using an attention pooling method with the same key features, the memory features of the current round are aggregated as a global representation of the current memory state; Calculate the similarity between the global representation of the current memory state and the key portion of all stored data in the historical dialogue semantic database; select the n stored data with the lowest similarity as semantic supplements; Meanwhile, for the selected semantic supplementary data Normalized recall weights are calculated based on the inverse relationship of similarity. ,Right now: ； in, This is a global representation of the current memory state. This refers to the key portion of the data stored in the historical dialogue semantic library.

6. The efficient conversational retrieval method based on memory-enhanced query intent understanding according to claim 5, characterized in that, The enhancement of memory features includes: For the selected semantic supplementary data , divide the value corresponding to each data item By recall weight The aggregated data forms the final forgotten information retrieval representation. The memory features of the current retrieval round serve as the query, and the forgotten information retrieval representation serves as both the key and value. This is achieved through cross-attention combined with the memory features of the current retrieval round. Interactive encoding is performed to obtain the enhanced memory features for the current retrieval round.

7. The efficient conversational retrieval method based on memory-enhanced query intent understanding according to claim 1, characterized in that, The refinement of the current query representation by incorporating the search results from the previous round includes: Take the first The top k image results in the final search ranking during the search round and features of all images to be retrieved Select the corresponding image features to form the candidate image feature set for the previous retrieval round. ; Through a fully connected layer Projected onto the currently enhanced memory features The same dimensions, yield ; As a query, Simultaneously, these are used as both key and value inputs into the transformer for interactive encoding, resulting in the final memorized features. ; The final memory features The data is further aggregated into a global query vector for the current retrieval round using the same attention pooling method as the key features, and then projected onto the same embedding dimension as the image features to obtain the final query vector for the current retrieval round.

8. The efficient conversational retrieval method based on memory-enhanced query intent understanding according to claim 1, characterized in that, The training of the model includes: The model is trained end-to-end, and contrastive learning loss is used to align the final query text representation with the visual features of its corresponding image; in each retrieval round In this process, the model calculates a retrieval loss that makes the query representation of the current retrieval round closer to its matching image, while distancing it from the non-matching samples in the same mini-batch. ； in This represents the size of a small batch of data. or This represents the query text and images that are positive samples to each other in this batch. or This represents the query text and images that are negative samples to each other in this batch. Temperature coefficient; Final training loss This represents the average loss across all dialogue rounds.

9. A high-efficiency conversational retrieval device based on memory-enhanced query intent understanding, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements an efficient conversational retrieval method based on memory-enhanced query intent understanding as described in any one of claims 1-8.

10. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements an efficient conversational retrieval method based on memory-enhanced query intent understanding as described in any one of claims 1-8.

Images