Data retrieval method and electronic device

Abstract

The application provides a data retrieval method and an electronic device. The method comprises: extracting at least one target object from multi-modal data input by a user; converting the target object into a target object vector; determining, by using a model, a predicted probability value of at least one retrieval strategy corresponding to the target object vector, the predicted probability value representing an accuracy rate of retrieval by using the retrieval strategy according to the target object vector; based on each predicted probability value, sequentially using the retrieval strategy to retrieve according to the target object vector until a difference between an utility score of a current retrieval result and an utility score of a previous retrieval result is less than a dynamic threshold value, obtaining at least two retrieval results corresponding to the target object vector; the utility score represents a correlation degree between the retrieval result and the multi-modal data and other retrieval results, the dynamic threshold value is determined according to the utility scores of all retrieval results, the retrieval result is used to determine a target retrieval result, and the target retrieval result is used to generate prompt information capable of guiding the model to generate an inference result.

Description

Technical Field

[0001] This application relates to the field of data processing, and more particularly to a data retrieval method and electronic device. Background Technology

[0002] Multimodal retrieval enhancement generation technology is an important evolutionary direction in the interdisciplinary field of natural language processing and computer vision. Its core paradigm lies in enhancing the generation process of large language models by retrieving external multimodal knowledge sources, aiming to solve the inherent illusion problem, knowledge lag, and limitations of pure text understanding. This technology significantly improves the factual accuracy and information completeness of output content in scenarios such as complex question answering, data analysis report generation, and decision support by transforming unstructured multimodal information, such as images, charts, and tables, into searchable and inferable knowledge fragments. A standard multimodal retrieval enhancement generation process typically includes key modules such as multimodal knowledge parsing, cross-modal semantic representation, vectorized index construction, similarity retrieval, and information fusion generation. For the same search keywords, the accuracy of search results obtained using different search strategies varies significantly. Therefore, currently, multimodal retrieval enhancement generation requires searching for search keywords using all search strategies, wasting a lot of system resources and resulting in low search efficiency and accuracy. Summary of the Invention

[0003] This application provides a data retrieval method and an electronic device.

[0004] One embodiment of this application provides a data retrieval method, the method comprising:

[0005] Extract at least one target object from multimodal data input by the user; Convert the target object into a target object vector; The model is used to determine the predicted probability value of at least one retrieval strategy corresponding to the target object vector, and the predicted probability value represents the accuracy of retrieval using the retrieval strategy based on the target object vector; Based on the predicted probability values, the retrieval strategy is used sequentially to retrieve data according to the target object vector until the difference between the utility score of the current retrieval result and the previous retrieval result is less than a dynamic threshold, thereby obtaining at least two retrieval results corresponding to the target object vector. The utility score represents the relevance of the retrieval result to the multimodal data and other retrieval results. The dynamic threshold is determined based on the utility scores of all retrieval results. The retrieval results are used to determine the target retrieval result, and the target retrieval result is used to generate prompt information that can guide the model to generate inference results.

[0006] The step of converting the target object into a target object vector includes: Obtain at least one historical input from the user; The target object is matched with the at least one historical input to obtain at least one piece of associated information about the target object, wherein the associated information includes at least the modality type of the target object; The target object and at least one associated information of the target object are vectorized to obtain the target object vector.

[0007] The step of sequentially performing the retrieval based on the target object vector using the retrieval strategy according to each of the predicted probability values ​​includes: The multiple retrieval strategies are traversed according to their predicted probability values. The current search strategy is used to search based on the target object vector to obtain the current search results; The utility score of the current search result is determined based on the current search result, the multimodal data, the search duration, the corresponding predicted probability, and other search results. If the current search result meets the first target condition, the traversal continues until the difference between the utility score of the current search result and the previous search result is less than the dynamic threshold, and the traversal is completed to obtain at least two search results for the target object vector. The first target condition is that the current search strategy is the first search strategy of the target object vector or the difference between the utility score of the current search result and the previous search result is greater than or equal to the dynamic threshold.

[0008] The step of determining the utility score of the current search result based on the current search result, the multimodal data, and other search results includes: A relevance score for the current search result is determined based on the current search result and the multimodal data, wherein the relevance score characterizes the degree of relevance between the current search result and the multimodal data; The novelty score of the current search result is determined based on the current search result and other search results, and the novelty score represents the relevance of the current search result to other search results; The time cost score of the current search result is determined based on the search duration and the corresponding predicted probability value. The utility score of the current search result is determined based on the relevance score, the novelty score, and the time cost score.

[0009] The step of determining the relevance score of the current search result based on the current search result and the multimodal data includes: The multimodal data is converted into a multimodal data vector; The similarity between the multimodal data vector and the current search result vector is determined, and the similarity is defined as the relevance score of the current search result, wherein the current search result includes at least the current search result vector.

[0010] The step of determining the novelty score of the current search result based on the current search result and other search results includes: If the current retrieval strategy is the first retrieval strategy of the target object vector, then the preset novelty score is determined as the novelty score of the current retrieval result; If the current retrieval strategy is not the first retrieval strategy of the target object vector, then the similarity between the current retrieval result and each other retrieval result is determined; the novelty score of the current retrieval result is determined based on the similarity between the current retrieval result and each other retrieval result.

[0011] After obtaining the current search result, the method further includes: Determine the difference in utility scores between any two adjacent search results among all search results adjacent to the target object; Determine the mean and standard deviation of the differences; The dynamic threshold is determined based on the mean and the standard deviation.

[0012] The method further includes: All search results are sorted according to the utility scores to obtain the sorted results; The first preset number of search results in the sorting results are determined as the target search results.

[0013] After determining the target retrieval result, the method further includes: Determine the semantic relevance between the target retrieval results and the multimodal data; The target modality type of the target object to which the target retrieval result belongs is determined based on the multimodal data; The preset number of target search results are reordered based on the semantic relevance, the source confidence and modality type of the target search results, and the target modality type. The search results also include the source confidence and the modality type.

[0014] Another aspect of this application provides an electronic device, including: a processor; The processor extracts at least one target object from the multimodal data input by the user; converts the target object into a target object vector; uses a model to determine the predicted probability value of at least one retrieval strategy corresponding to the target object vector, the predicted probability value representing the accuracy of retrieval using the retrieval strategy based on the target object vector; and based on each predicted probability value, sequentially uses the retrieval strategy to retrieve the target object vector until the difference between the utility score of the current retrieval result and the previous retrieval result is less than a dynamic threshold, obtaining at least two retrieval results corresponding to the target object vector; the utility score representing the relevance of the retrieval result to the multimodal data and other retrieval results, the dynamic threshold being determined based on the utility scores of all retrieval results, the retrieval results being used to determine the target retrieval result, and the target retrieval result being used to generate prompt information that can guide the model to generate inference results.

[0015] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0016] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of this application are illustrated in the drawings by way of example and not limitation, in which: In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0017] Figure 1 A flowchart of a data retrieval method according to an embodiment of this application is shown; Figure 2 A flowchart of a data retrieval method according to another embodiment of this application is shown; Figure 3 A flowchart of a data retrieval method according to another embodiment of this application is shown; Figure 4 A flowchart of a data retrieval method according to another embodiment of this application is shown; Figure 5 A flowchart of a data retrieval method according to another embodiment of this application is shown; Figure 6 A flowchart of a data retrieval method according to another embodiment of this application is shown; Figure 7 A flowchart of a data retrieval method according to another embodiment of this application is shown; Figure 8A flowchart of a data retrieval method according to another embodiment of this application is shown; Figure 9 A flowchart of a data retrieval method according to another embodiment of this application is shown; Figure 10 A schematic diagram of the composition structure of an electronic device according to an embodiment of this application is shown; Figure 11 A schematic diagram of the structure of a data retrieval device according to an embodiment of this application is shown. Detailed Implementation

[0018] To make the objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] To prioritize the use of high-accuracy retrieval strategies during multimodal retrieval, thereby improving retrieval efficiency and accuracy, one embodiment of this application provides a data retrieval method, such as... Figure 1 As shown, the method includes: Step 101: Extract at least one target object from the multimodal data input by the user.

[0020] Multimodal data includes at least two of the following modalities: text, image, chart, audio, and document. The target object is a semantic unit extracted from the multimodal data that requires independent retrieval, and it is matched one-to-one with each modality type. For example, for a given multimodal dataset, the target object could be "company financial policy related content" extracted from its text data, "local features of the first image" extracted from its image data, and "financial data in the third row of the report" extracted from its tabular data.

[0021] In this embodiment, the extraction of target objects from multimodal data can be achieved through semantic role labeling, referential resolution techniques, entity recognition algorithms, and other methods.

[0022] Step 102: Convert the target object into a target object vector.

[0023] Step 103: Use the model to determine the predicted probability value of at least one retrieval strategy corresponding to the target object vector, wherein the predicted probability value represents the accuracy of retrieval using the retrieval strategy based on the target object vector.

[0024] Search strategies include searching text libraries, searching macro-level image libraries, searching detailed image libraries, calling OCR recognition tools, calling table parsers, searching audio feature libraries, and searching chart data dictionaries.

[0025] For each target object vector, the model is used to predict the accuracy of retrieval based on that target object vector using each retrieval strategy, thus obtaining the predicted probability value between each target object vector and each retrieval strategy.

[0026] Step 104: Based on each predicted probability value, the retrieval strategy is used sequentially to retrieve data according to the target object vector until the difference between the utility score of the current retrieval result and the previous retrieval result is less than a dynamic threshold, thereby obtaining at least two retrieval results corresponding to the target object vector; the utility score characterizes the relevance of the retrieval result to the multimodal data and other retrieval results; the dynamic threshold is determined based on the utility scores of all retrieval results; the retrieval results are used to determine the target retrieval result; and the target retrieval result is used to generate prompt information that can guide the model to generate inference results.

[0027] Based on the predicted probability values ​​corresponding to each retrieval strategy, the corresponding retrieval strategies are applied sequentially according to a preset order. Retrieval operations are performed based on the target object vector. After the retrieval is completed, the utility score of the current retrieval result is determined based on the current retrieval result, other retrieval results, and multimodal data. The difference between the utility score of the current retrieval result and the utility score of the previous retrieval result is also determined. If this difference is less than a dynamic threshold, the retrieval of the current target object vector is stopped. Ultimately, at least two retrieval results for the target object vector are obtained. These retrieval results are used to subsequently determine the target retrieval result, and the target retrieval result is used to generate prompts to guide the model in generating inference results.

[0028] The dynamic threshold is determined based on the utility scores of all search results. After each search is completed, the dynamic threshold is re-determined based on the utility scores of all existing search results for the current target object vector, and then used for the next search of the current target object vector.

[0029] In the above scheme, by extracting target objects matching the modality type from the multimodal data input by the user, the retrieval direction for multimodal data retrieval becomes more accurate. Converting target objects into target object vectors and then using a model to determine the predicted probability values ​​of each retrieval strategy corresponding to the target object vectors allows for the selection and priority use of highly accurate retrieval strategies, eliminating the need to try all retrieval strategies one by one and significantly reducing the waste of system resources. Retrieval operations are then executed sequentially based on the predicted probability values, and the decision to stop the retrieval is made based on the utility score of the retrieval results and a dynamic threshold. The dynamic threshold is updated in real time with the utility scores of existing retrieval results, ensuring that a sufficient number of highly relevant retrieval results are obtained while avoiding invalid retrieval extensions, further improving retrieval efficiency. Simultaneously, prioritizing highly accurate retrieval strategies also effectively improves the accuracy of retrieval results, making the prompts used to generate guidance model inference results more reliable, thus better serving the application of multimodal retrieval enhancement generation in relevant scenarios.

[0030] This application also provides a data retrieval method in one example, such as Figure 2 As shown, the step of converting the target object into a target object vector includes: Step 201: Obtain at least one historical input from the user.

[0031] Historical input refers to the input data previously entered by the user.

[0032] Step 202: Match the target object with the at least one historical input to obtain at least one piece of associated information of the target object, wherein the associated information includes at least the modality type of the target object.

[0033] Association information refers to relevant information that accurately characterizes the features of the target object, obtained by matching the target object with historical input. Association information includes the modality type of the target object, the referential associations of the target object in historical input (such as the correspondence between the target object and a certain entity and / or a certain piece of content mentioned in historical input), the required operations corresponding to the target object (such as evaluation, extraction, comparison, analysis, etc.), the expected retrieval granularity of the target object (such as macro description, fine-grained details, precise numerical values, technical terms, etc.), and the association attributes between the target object and related entities or content in historical input (such as time association, topic association, data source association, etc.).

[0034] Step 203: Perform vector transformation on the target object and at least one associated information of the target object to obtain the target object vector.

[0035] For example, based on the current multimodal data input by the user, the extracted target object is "the financial data in the third row of the report". The user's historical input includes "Please analyze the compliance of the company's 2023 financial statements" and the 2023 company financial statement file. Matching the target object "the financial data in the third row of the report" with these historical inputs yields association information including a table modality, a required operation of data extraction, an expected granularity of precise numerical values, and the file corresponding to "the 2023 company financial statements" from the historical inputs. Jointly encoding the target object "the financial data in the third row of the report" with the modality type, required operation, expected granularity, and association information generates a target object vector that includes both the semantics of the target object itself and its association information.

[0036] In the above scheme, previous user input data is used as historical input to obtain past user interaction information for assisted retrieval. The target object is then matched with the historical input to obtain association information across multiple dimensions, including modality type, referential relationships, required operations, and expected granularity. This association information comprehensively and accurately represents the characteristics of the target object. The target object and this association information are then combined to undergo vector transformation, resulting in a target object vector that includes not only the semantics of the target object itself but also association information representing its core features. This allows the target object vector to more accurately match the user's true search intent, significantly improving the accuracy of the search results.

[0037] This application also provides a data retrieval method in one example, such as Figure 3 As shown, the step of sequentially using the retrieval strategy based on the target object vector according to each of the predicted probability values ​​includes: Step 301: Traverse the multiple retrieval strategies according to their predicted probability values.

[0038] For example, if the target object vector is the vector corresponding to "financial data in the third row of the report", the model determines the retrieval strategies and predicted probability values ​​for the target object vector, including 0.5 for retrieving the text library, 0.47 for calling the table parser, 0.01 for retrieving the macro image library, 0.01 for calling OCR, 0.01 for retrieving the detailed image library, and 0.01 for retrieving the audio feature library. These retrieval strategies are traversed and sorted in descending order of predicted probability value, resulting in the following traversal order: retrieving the text library, calling the table parser, retrieving the macro image library, calling OCR, retrieving the detailed image library, and retrieving the audio feature library.

[0039] Step 302: Use the current search strategy to perform a search based on the target object vector to obtain the current search results.

[0040] Following the example above, and according to the previously determined traversal order, the search strategy with the highest probability is determined to be the text library search. Based on the target object vector corresponding to "financial data in the third row of the report", the search is performed in the knowledge base to obtain the current search results for the target object vector.

[0041] Step 303: Determine the utility score of the current search result based on the current search result, the multimodal data, the search duration, the corresponding predicted probability, and other search results.

[0042] Continuing with the example above, the retrieval time for searching based on the target object vector using the search text library is 0.8 seconds, and the predicted probability of this retrieval strategy for the target object vector is 0.5. Since the current retrieval is the first retrieval, the utility score of the current retrieval result is determined to be 0.56 based on the current retrieval result, multimodal data, retrieval time, and corresponding predicted probability.

[0043] Step 304: If the current search result satisfies the first target condition, the traversal continues until the difference between the utility score of the current search result and the previous search result is less than the dynamic threshold, and the traversal is completed to obtain at least two search results of the target object vector. The first target condition is that the current search strategy is the first search strategy of the target object vector or the difference between the utility score of the current search result and the previous search result is greater than or equal to the dynamic threshold.

[0044] Continuing with the example above, the current retrieval strategy, which searches the text library, is the first retrieval strategy for the target object vector, satisfying the first objective condition. Therefore, the next round of retrieval continues in traversal order. The next retrieval strategy is determined to be calling the table parser. A retrieval is performed based on the target object vector, yielding a retrieval result. Based on this retrieval result, multimodal data, retrieval duration, corresponding predicted probability, and other retrieval results, the utility score of this retrieval result is determined to be 0.72. The difference between this and the utility score of the previous retrieval result is 0.16. At this point, the dynamic threshold is determined to be 0.56 based on existing retrieval results. The difference is less than the dynamic threshold, therefore the first objective condition is not met, and traversal stops. Finally, two sets of retrieval results are obtained: one for the searched text library and one for the call to the table parser, corresponding to the target object vector.

[0045] In the above scheme, by traversing according to the predicted probability values ​​of the search strategies, the search can prioritize the use of search strategies with higher accuracy, making the search more targeted, reducing unnecessary search operations, and saving system resources. During the search process, the utility score is determined by combining the current search results, multimodal data, search duration, corresponding predicted probabilities, and other search results. This allows for a comprehensive evaluation of the actual value of each search result, avoiding the one-sidedness of a single-dimensional evaluation. Simultaneously, determining whether to continue traversing based on the first objective condition ensures that the result corresponding to the first search strategy can smoothly enter the subsequent process. Furthermore, by comparing the difference between the utility score of the current search result and the utility score of the previous search result with a dynamic threshold, the search is stopped, ensuring that at least two valuable search results are ultimately obtained, avoiding meaningless search extensions, and further improving search efficiency.

[0046] This application also provides a data retrieval method in one example, such as Figure 4 As shown, determining the utility score of the current search result based on the current search result, the multimodal data, and other search results includes: Step 401: Determine the relevance score of the current search result based on the current search result and the multimodal data. The relevance score represents the degree of relevance between the current search result and the multimodal data.

[0047] The relevance score of the current search result can be determined by measuring the Euclidean distance, Manhattan distance, Pearson correlation coefficient, or cosine similarity between the vector corresponding to the current search result and the vector corresponding to the multimodal data.

[0048] Step 402: Determine the novelty score of the current search result based on the current search result and other search results. The novelty score represents the relevance of the current search result to other search results.

[0049] First, determine the Euclidean distance, Manhattan distance, Pearson correlation coefficient, or cosine similarity between the vector corresponding to the current search result and the vector corresponding to other search results. Then, take the maximum value of the Euclidean distance, Manhattan distance, Pearson correlation coefficient, or cosine similarity between the current search result and each of the other search results as the novelty score of the current search result.

[0050] Step 403: Determine the time cost score of the current search result based on the search duration and the corresponding predicted probability value.

[0051] Specifically, the time cost score of the current search result can be determined using the following formula based on the search duration and the corresponding predicted probability value. :

[0052] in, To preset weights, The search duration corresponding to the current search result. This is the predicted probability value corresponding to the search strategy used for the current search result. It is a preset very small positive number to prevent division by zero errors.

[0053] Step 404: Determine the utility score of the current search result based on the relevance score, the novelty score, and the time cost score.

[0054] Specifically, the utility score of the current search result can be determined using the following formula based on the relevance score, novelty score, and time cost score. :

[0055] in, The relevance score of the current search result. The novelty score of the current search results. and All are preset weights. and .

[0056] In the above scheme, by separately determining relevance score, novelty score, and time cost score, and combining these scores to finally determine the utility score, the actual value of each search result can be comprehensively evaluated. The relevance score measures the degree of relevance between the current search result and multimodal data using cosine similarity, ensuring that the search results meet user needs. The novelty score measures the degree of relevance between the current search result and other search results, effectively avoiding redundancy of duplicate information and ensuring the information value of newly added search results. The time cost score is determined by combining search duration and the predicted probability value of the corresponding search strategy, considering both the efficiency of the search process and the reliability of the search strategy itself. Finally, by pre-setting weights, the scores of these three dimensions are integrated into a utility score. Ranking the search results using the utility score accurately filters out search results that are strongly relevant to multimodal data, have novel value, and are efficient and controllable, thereby improving the overall efficiency and accuracy of the search.

[0057] This application also provides a data retrieval method in one example, such as Figure 5 As shown, determining the relevance score of the current search result based on the current search result and the multimodal data includes: Step 501: Convert the multimodal data into a multimodal data vector.

[0058] Step 502: Determine the similarity between the multimodal data vector and the current search result vector, and determine the similarity as the relevance score of the current search result, wherein the current search result includes at least the current search result vector.

[0059] Specifically, the relevance score of the current search result can be determined using the following formula based on the multimodal data vector and the current search result vector. :

[0060] in, For cosine similarity, For multimodal data vectors, This is the vector of current search results.

[0061] In the above scheme, by using the cosine similarity between the multimodal data vector and the current search result as the relevance score, the semantic association between the current search result and the multimodal data can be accurately represented, thereby improving the accuracy of the search results.

[0062] This application also provides a data retrieval method in one example, such as Figure 6 As shown, determining the novelty score of the current search result based on the current search result and other search results includes: Step 601: If the current retrieval strategy is the first retrieval strategy of the target object vector, then the preset novelty score is determined as the novelty score of the current retrieval result.

[0063] In this embodiment, the preset novelty score is set to 1. In other embodiments, the preset novelty score can be set according to specific needs.

[0064] Step 602: If the current retrieval strategy is not the first retrieval strategy of the target object vector, then determine the similarity between the current retrieval result and each other retrieval result; determine the novelty score of the current retrieval result based on the similarity between the current retrieval result and each other retrieval result.

[0065] Specifically, the novelty score of the current search result can be determined using the following formula based on the current search results and the search results. :

[0066] in, The vector corresponding to the current search result and the first... Cosine similarity of vectors corresponding to other search results This is the maximum cosine similarity among the vectors corresponding to the current search result and the vectors corresponding to all other search results.

[0067] In the above scheme, if the current search result is the first search strategy, the preset novelty score is directly used as the novelty score of the current search result. If the current search result is not the first search strategy, by determining the similarity between the current search result and each other search result, and taking the maximum similarity as the novelty score, the novelty of the current search result relative to other search results can be accurately measured, effectively avoiding redundancy of duplicate information. This makes the determination of the novelty score more targeted and accurate, thereby improving the accuracy of the search results and giving the final search results higher information value.

[0068] This application also provides a data retrieval method in one example, such as Figure 7 As shown, after obtaining the current search result, the method further includes: Step 701: Determine the difference in utility scores between any two adjacent search results among all search results adjacent to the target object.

[0069] For example, for a specific target object vector, four search results are retrieved using different search strategies. In the search order, these are search result A, search result B, search result C, and search result D. The utility score of search result A is 0.70, the utility score of search result B is 0.68, the utility score of search result C is 0.68, and the utility score of search result D is 0.64. The utility score difference between search result A and search result B is determined to be 0.02, the utility score difference between search result B and search result C is 0, and the utility score difference between search result C and search result D is 0.04.

[0070] Step 702: Determine the mean and standard deviation of the difference.

[0071] Continuing with the example above, the mean of these differences is determined to be 0.02, and the standard deviation of these differences is determined to be approximately 0.016.

[0072] Step 703: Determine the dynamic threshold based on the mean and the standard deviation.

[0073] Continuing with the example above, subtracting the standard deviation from the mean gives 0.004, thus determining the dynamic threshold to be 0.004.

[0074] In the above scheme, the utility score difference between adjacent search results of the target object vector is determined, and then the mean and standard deviation are determined based on these differences. Finally, the dynamic threshold is obtained by subtracting the standard deviation from the mean. The dynamic threshold determined based on the actual search results can accurately represent the changing trend of the information value of the search results, avoiding the problem that fixed thresholds are difficult to adapt to different search scenarios. Using the dynamic threshold to determine whether to continue searching can effectively avoid stopping the search too early and missing key information, and can also prevent excessive searching from wasting system resources, thereby significantly improving the efficiency of the search.

[0075] This application also provides a data retrieval method in one example, such as Figure 8 As shown, the method further includes: Step 801: Sort all search results according to the utility score to obtain the sorting result.

[0076] For example, for a specific target object vector, four search results are retrieved using different search strategies. In the order of retrieval, these results are search result A, search result B, search result C, and search result D. Search result A has a utility score of 0.68, search result B has a utility score of 0.70, search result C has a utility score of 0.72, and search result D has a utility score of 0.64. Sorting these four results in descending order of utility score yields the following ranking: search result C, search result B, search result A, and search result D.

[0077] Step 802: The first preset number of search results in the sorting results are determined as the target search results.

[0078] Continuing with the example above, if the preset quantity is set to 2, then the target search results will be search result C and search result B.

[0079] In the above scheme, ranking all search results based on utility scores accurately distinguishes the information value of different search results, placing search results with higher information value at the top. Then, the top predetermined number of search results in the ranking are selected as target search results, accurately filtering out high-information-value results and effectively eliminating redundant low-information-value results. This prevents irrelevant information from interfering with subsequent processes and ensures the quality of the target search results.

[0080] This application also provides a data retrieval method in one example, such as Figure 9 As shown, after determining the target retrieval result, the method further includes: Step 901: Determine the semantic relevance between the target retrieval result and the multimodal data.

[0081] The semantic relevance between the target retrieval results and multimodal data can be determined by methods such as Euclidean distance, Manhattan distance, Pearson correlation coefficient, or cosine similarity.

[0082] Step 902: Determine the target modality type of the target object to which the target retrieval result belongs based on the multimodal data.

[0083] Based on the modality type of the extracted target object data, the target modality type of the target object is determined, thereby determining the target modality type to which the target retrieval results belong.

[0084] Step 903: Reorder the preset number of target search results according to the semantic relevance, the source confidence and modality type of the target search results, and the target modality type. The search results also include the source confidence and the modality type.

[0085] A preset number of target search results, along with the semantic relevance of each result, the source confidence level and modality type of each result, and the target modality type, are input into the ranking model for re-sorting. The re-sorted results are more accurate.

[0086] In the above scheme, by determining the semantic relevance between the target retrieval results and multimodal data, the target modality type is determined based on the data modality type of the extracted target object. Then, by combining the semantic relevance, the source confidence and modality type of the target retrieval results, and the target modality type, a ranking model is used to re-rank a preset number of target retrieval results, which further improves the accuracy of the ranking results and effectively filters out retrieval results with low value.

[0087] This application provides an electronic device, Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0088] like Figure 10 As shown, the electronic device 1000 includes: a processor 1001; The processor 1001 extracts at least one target object from the multimodal data input by the user; converts the target object into a target object vector; uses a model to determine the predicted probability value of at least one retrieval strategy corresponding to the target object vector, the predicted probability value representing the accuracy of retrieval using the retrieval strategy based on the target object vector; and based on each predicted probability value, sequentially uses the retrieval strategy to retrieve the target object vector until the difference between the utility score of the current retrieval result and the previous retrieval result is less than a dynamic threshold, obtaining at least two retrieval results corresponding to the target object vector; the utility score representing the relevance of the retrieval result to the multimodal data and other retrieval results, the dynamic threshold being determined based on the utility scores of all retrieval results, the retrieval results being used to determine the target retrieval result, and the target retrieval result being used to generate prompt information that can guide the model to generate inference results.

[0089] The processor 1001 obtains at least one historical input from the user; matches the target object with the at least one historical input to obtain at least one associated information of the target object, the associated information including at least the modality type of the target object; and performs vector transformation on the target object and the at least one associated information of the target object to obtain the target object vector.

[0090] The processor 1001 iterates through the multiple retrieval strategies according to the predicted probability values ​​of the multiple retrieval strategies; it uses the current retrieval strategy to perform a retrieval based on the target object vector to obtain the current retrieval result; it determines the utility score of the current retrieval result based on the current retrieval result, the multimodal data, the retrieval duration, the corresponding predicted probability, and other retrieval results; and if the current retrieval result meets a first target condition, it continues to iterate until the difference between the utility score of the current retrieval result and the previous retrieval result is less than a dynamic threshold, thus completing the traversal and obtaining at least two retrieval results for the target object vector, wherein the first target condition is that the current retrieval strategy is the first retrieval strategy for the target object vector or the difference between the utility score of the current retrieval result and the previous retrieval result is greater than or equal to the dynamic threshold.

[0091] The processor 1001 determines a relevance score for the current search result based on the current search result and the multimodal data, the relevance score representing the degree of relevance between the current search result and the multimodal data; determines a novelty score for the current search result based on the current search result and other search results, the novelty score representing the degree of relevance between the current search result and other search results; determines a time cost score for the current search result based on the search duration and the corresponding predicted probability value; and determines a utility score for the current search result based on the relevance score, the novelty score, and the time cost score.

[0092] The processor 1001 converts the multimodal data into a multimodal data vector; and determines the similarity between the multimodal data vector and the current search result vector, and determines the similarity as the relevance score of the current search result, wherein the current search result includes at least the current search result vector.

[0093] If the current retrieval strategy is the first retrieval strategy of the target object vector, the processor 1001 determines the novelty score of the current retrieval result as a preset novelty score; and if the current retrieval strategy is not the first retrieval strategy of the target object vector, it determines the similarity between the current retrieval result and each other retrieval result; and determines the novelty score of the current retrieval result based on the similarity between the current retrieval result and each other retrieval result.

[0094] The processor 1001 determines the difference in utility scores between any two adjacent search results in all search results adjacent to the target object; determines the mean and standard deviation of the difference; and determines the dynamic threshold based on the mean and the standard deviation.

[0095] The processor 1001 sorts all search results according to the utility score to obtain a sorting result; and determines the first preset number of search results in the sorting result as the target search results.

[0096] The processor 1001 determines the semantic relevance between the target retrieval result and the multimodal data; determines the target modality type of the target object to which the target retrieval result belongs based on the multimodal data; and reorders the preset number of target retrieval results based on the semantic relevance, the source confidence and modality type of the target retrieval result, and the target modality type, wherein the retrieval result also includes the source confidence and the modality type.

[0097] To implement the above data retrieval method, such as Figure 11 As shown, an example of this application provides a data retrieval apparatus, including: Processing module 1101 is used to extract at least one target object from multimodal data input by the user; Calculation module 1102 is used to convert the target object into a target object vector; The calculation module 1102 is further configured to use a model to determine the predicted probability value of at least one retrieval strategy corresponding to the target object vector, wherein the predicted probability value represents the accuracy of retrieval using the retrieval strategy based on the target object vector; The processing module 1101 is further configured to, based on each of the predicted probability values, sequentially perform retrieval according to the target object vector using the retrieval strategy until the difference between the utility score of the current retrieval result and the previous retrieval result is less than a dynamic threshold, thereby obtaining at least two retrieval results corresponding to the target object vector; the utility score characterizes the relevance of the retrieval result to the multimodal data and other retrieval results, the dynamic threshold is determined based on the utility scores of all retrieval results, the retrieval results are used to determine the target retrieval result, and the target retrieval result is used to generate prompt information that can guide the model to generate inference results.

[0098] The processing module 1101 is further configured to obtain at least one historical input from the user; The processing module 1101 is further configured to match the target object with the at least one historical input to obtain at least one association information of the target object, wherein the association information includes at least the modality type of the target object; The calculation module 1102 is further configured to perform vector conversion on the target object and at least one associated information of the target object to obtain the target object vector.

[0099] The calculation module 1102 is further configured to traverse the multiple retrieval strategies according to the predicted probability values ​​of the multiple retrieval strategies. The processing module 1101 is further configured to perform a search based on the target object vector using the current search strategy to obtain the current search result; The calculation module 1102 is also used to determine the utility score of the current search result based on the current search result, the multimodal data, the search duration, the corresponding prediction probability and other search results; The processing module 1101 is further configured to continue traversing if the current search result satisfies the first target condition, until the difference between the utility score of the current search result and the previous search result is less than a dynamic threshold, thereby completing the traversal and obtaining at least two search results for the target object vector. The first target condition is that the current search strategy is the first search strategy of the target object vector or the difference between the utility score of the current search result and the previous search result is greater than or equal to the dynamic threshold.

[0100] The calculation module 1102 is further configured to determine a relevance score of the current search result based on the current search result and the multimodal data, wherein the relevance score characterizes the degree of relevance between the current search result and the multimodal data; The calculation module 1102 is further configured to determine the novelty score of the current search result based on the current search result and other search results, wherein the novelty score characterizes the relevance of the current search result to other search results; The calculation module 1102 is further configured to determine the time cost score of the current search result based on the search duration and the corresponding predicted probability value; The calculation module 1102 is further configured to determine the utility score of the current search result based on the relevance score, the novelty score, and the time cost score.

[0101] The calculation module 1102 is further configured to convert the multimodal data into a multimodal data vector. The calculation module 1102 is further configured to determine the similarity between the multimodal data vector and the current search result vector, and to determine the similarity as the relevance score of the current search result, wherein the current search result includes at least the current search result vector.

[0102] The calculation module 1102 is further configured to determine the preset novelty score as the novelty score of the current search result if the current search strategy is the first search strategy of the target object vector. The calculation module 1102 is further configured to determine the similarity between the current search result and each other search result if the current search strategy is not the first search strategy of the target object vector; and to determine the novelty score of the current search result based on the similarity between the current search result and each other search result.

[0103] The calculation module 1102 is further configured to determine the difference in utility scores between any two adjacent search results among all search results adjacent to the target object. The calculation module 1102 is also used to determine the mean and standard deviation of the difference; The calculation module 1102 is further configured to determine the dynamic threshold based on the mean and the standard deviation.

[0104] The calculation module 1102 is further configured to sort all search results according to the utility score to obtain a sorting result; The processing module 1101 is further configured to determine the first preset number of search results in the sorting results as the target search results.

[0105] The calculation module 1102 is further configured to determine the semantic relevance between the target retrieval result and the multimodal data; The processing module 1101 is further configured to determine the target modality type of the target object to which the target retrieval result belongs based on the multimodal data; The calculation module 1102 is further configured to reorder the preset number of target retrieval results based on the semantic relevance, the source confidence and modality type of the target retrieval results, and the target modality type. The retrieval results also include the source confidence and the modality type.

[0106] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0107] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0108] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0109] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0110] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0111] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0112] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0113] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.

[0114] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A data retrieval method, the method comprising: Extract at least one target object from multimodal data input by the user; Convert the target object into a target object vector; The model is used to determine the predicted probability value of at least one retrieval strategy corresponding to the target object vector, and the predicted probability value represents the accuracy of retrieval using the retrieval strategy based on the target object vector; Based on the predicted probability values, the retrieval strategy is used sequentially to retrieve data according to the target object vector until the difference between the utility score of the current retrieval result and the previous retrieval result is less than a dynamic threshold, thereby obtaining at least two retrieval results corresponding to the target object vector. The utility score represents the relevance of the retrieval result to the multimodal data and other retrieval results. The dynamic threshold is determined based on the utility scores of all retrieval results. The retrieval results are used to determine the target retrieval result, and the target retrieval result is used to generate prompt information that can guide the model to generate inference results.

2. The method according to claim 1, wherein converting the target object into a target object vector comprises: Obtain at least one historical input from the user; The target object is matched with the at least one historical input to obtain at least one piece of associated information about the target object, wherein the associated information includes at least the modality type of the target object; The target object and at least one associated information of the target object are vectorized to obtain the target object vector.

3. The method according to claim 1, wherein the step of sequentially using the retrieval strategy based on each of the predicted probability values ​​to perform retrieval according to the target object vector comprises: The multiple retrieval strategies are traversed according to their predicted probability values. The current search strategy is used to search based on the target object vector to obtain the current search results; The utility score of the current search result is determined based on the current search result, the multimodal data, the search duration, the corresponding predicted probability, and other search results. If the current search result meets the first target condition, the traversal continues until the difference between the utility score of the current search result and the previous search result is less than the dynamic threshold, and the traversal is completed to obtain at least two search results for the target object vector. The first target condition is that the current search strategy is the first search strategy of the target object vector or the difference between the utility score of the current search result and the previous search result is greater than or equal to the dynamic threshold.

4. The method according to claim 3, wherein determining the utility score of the current search result based on the current search result, the multimodal data, and other search results comprises: A relevance score for the current search result is determined based on the current search result and the multimodal data, wherein the relevance score characterizes the degree of relevance between the current search result and the multimodal data; The novelty score of the current search result is determined based on the current search result and other search results, and the novelty score represents the relevance of the current search result to other search results; The time cost score of the current search result is determined based on the search duration and the corresponding predicted probability value. The utility score of the current search result is determined based on the relevance score, the novelty score, and the time cost score.

5. The method according to claim 4, wherein determining the relevance score of the current search result based on the current search result and the multimodal data comprises: The multimodal data is converted into a multimodal data vector; The similarity between the multimodal data vector and the current search result vector is determined, and the similarity is defined as the relevance score of the current search result, wherein the current search result includes at least the current search result vector.

6. The method according to claim 4, wherein determining the novelty score of the current search result based on the current search result and other search results comprises: If the current retrieval strategy is the first retrieval strategy of the target object vector, then the preset novelty score is determined as the novelty score of the current retrieval result; If the current retrieval strategy is not the first retrieval strategy of the target object vector, then the similarity between the current retrieval result and each other retrieval result is determined; The novelty score of the current search result is determined based on the similarity between the current search result and each other search result.

7. The method according to claim 3, wherein after obtaining the current search result, the method further comprises: Determine the difference in utility scores between any two adjacent search results among all search results adjacent to the target object; Determine the mean and standard deviation of the differences; The dynamic threshold is determined based on the mean and the standard deviation.

8. The method according to claim 1, further comprising: All search results are sorted according to the utility scores to obtain the sorted results; The first preset number of search results in the sorting results are determined as the target search results.

9. The method according to claim 8, wherein after determining the target retrieval result, the method further comprises: Determine the semantic relevance between the target retrieval results and the multimodal data; The target modality type of the target object to which the target retrieval result belongs is determined based on the multimodal data; The preset number of target search results are reordered based on the semantic relevance, the source confidence and modality type of the target search results, and the target modality type. The search results also include the source confidence and the modality type.

10. An electronic device, comprising: processor; The processor extracts at least one target object from the multimodal data input by the user; The target object is converted into a target object vector; the model is used to determine the predicted probability value of at least one retrieval strategy corresponding to the target object vector, and the predicted probability value represents the accuracy of retrieval using the retrieval strategy based on the target object vector; Based on the predicted probability values, the retrieval strategy is used sequentially to retrieve data according to the target object vector until the difference between the utility score of the current retrieval result and the previous retrieval result is less than a dynamic threshold, thereby obtaining at least two retrieval results corresponding to the target object vector. The utility score represents the relevance of the retrieval result to the multimodal data and other retrieval results. The dynamic threshold is determined based on the utility scores of all retrieval results. The retrieval results are used to determine the target retrieval result, and the target retrieval result is used to generate prompt information that can guide the model to generate inference results.

Images