Data processing method, model training method, electronic device, and storage medium

Abstract

The invention discloses a data processing method, a model training method, electronic equipment and a storage medium, and relates to the technical field of large models and computers. The method comprises the following steps: acquiring multi-modal query data; performing knowledge retrieval and information generation on the multi-modal query data by adopting a target retrieval generation model to obtain a target answer; wherein the target retrieval generation model is used for obtaining a target retrieval result meeting a preset screening condition from candidate retrieval results associated with the multi-modal query data, and generating response content contained in a target answer based on the target retrieval result, the preset screening condition is used for screening the candidate retrieval results according to the association degree sorting result of the candidate retrieval results. According to the method and the device, the technical problems of low retrieval generation efficiency and poor accuracy when the multi-modal query data is retrieved in the prior art are solved.

Description

(19) State Intellectual Property Office (12) Invention Patent Application (10) Application Publication Number (43) Application Publication Date (21) Application Number 202411850113.0 (22) Application Date 2024.12.13 (71) Applicant Alibaba (China) Co., Ltd. Address 310052, Room 508, 5th Floor, Building 4, No. 699, Wangshang Road, Changhe Street, Binjiang District, Hangzhou City, Zhejiang Province (72) Inventors Long Dingkun, Zhang Yanzhao, Wang Shihang, Ding Ruixue, Xie Pengjun (74) Patent Agency Beijing Bohao Bairui Intellectual Property Agency Co., Ltd. 11134 Patent Attorney Zhao Yunbin (51) Int.Cl. G06F 16 / 3329 (2025.01) G06F 16 / 334 (2025.01) G06F 18 / 213 (2023.01) G06F 18 / 214(2023.01) G06F 18 / 10(2023.01) G06F 18 / 22(2023.01) G06N 3 / 08(2023.01) (54) Invention Title: Data Processing Method, Model Training Method, Electronic Device and Storage Medium (57) Abstract: This application discloses a data processing method, a model training method, an electronic device and a storage medium, relating to large model technology and computer technology. The method includes: acquiring multimodal query data; using a target retrieval generation model to perform knowledge retrieval and information generation on the multimodal query data to obtain a target answer; wherein the target retrieval generation model is used to obtain target retrieval results that meet preset filtering conditions from candidate retrieval results associated with the multimodal query data, and to generate response content contained in the target answer based on the target retrieval results; the preset filtering conditions are used to filter candidate retrieval results according to the degree of association of the candidate retrieval results. This application solves the technical problems of low retrieval generation efficiency and poor accuracy in the related art when retrieving multimodal query data. Claims 3 pages, Description 19 pages, Drawings 7 pages, CN 122220450 A 2026.06.16 CN 1 22 22 04 50 A 1. A data processing method, characterized in that it includes: acquiring multimodal query data; using a target retrieval generation model to perform knowledge retrieval and information generation on the multimodal query data to obtain a target answer; wherein, the target retrieval generation model is used to obtain target retrieval results that meet preset filtering conditions from candidate retrieval results associated with the multimodal query data, and to generate response content contained in the target answer based on the target retrieval results, and the preset filtering conditions are used to filter the candidate retrieval results according to the ranking result of the degree of association of the candidate retrieval results. 2. The data processing method according to claim 1, characterized in that, the target retrieval generation model includes: multimodal query data.The modal recall sub-model, multimodal ranking sub-model, and multimodal generation sub-model utilize the target retrieval and generation model to perform knowledge retrieval and information generation on the multimodal query data to obtain the target answer. This includes: using the multimodal recall sub-model to perform recall processing on the multimodal query data to obtain the candidate retrieval results; using the multimodal ranking sub-model to perform ranking processing on the candidate retrieval results to obtain the ranking results; and using the multimodal generation sub-model to perform response content generation processing on the ranking results to obtain the target answer. 3. The data processing method according to claim 2, characterized in that the multimodal recall sub-model comprises: a feature encoding module, a feature mapping module, and a feature matching module. The multimodal recall sub-model is used to perform recall processing on the multimodal query data to obtain the candidate retrieval results, comprising: performing feature encoding processing on the multimodal query data using the feature encoding module to obtain a multimodal feature vector; performing feature mapping processing on the multimodal feature vector using the feature mapping module to obtain a multimodal mapping result, wherein the multimodal mapping result is used to represent the multimodal representation obtained by mapping feature vectors of different modalities to a shared multimodal space; and using the feature matching module to recall the candidate retrieval results that match the multimodal mapping result from the target multimodal dataset. 4. The data processing method according to claim 3, characterized in that, recalling the candidate retrieval results that match the multimodal mapping result from the target multimodal dataset using the feature matching module includes: in the feature matching module, recalling the candidate retrieval results that match the multimodal mapping result from the target multimodal dataset based on a preset index structure. 5. The data processing method according to claim 4, characterized in that, the data processing method further includes: obtaining an initial multimodal dataset; performing data cleaning and preprocessing on the initial multimodal dataset to obtain an intermediate multimodal dataset; performing data alignment and data annotation on the intermediate multimodal dataset to obtain the target multimodal dataset; performing feature encoding on the target multimodal dataset to construct the preset index structure. 6. The data processing method according to claim 4, characterized in that, recalling the candidate retrieval results matching the multimodal mapping result from the target multimodal dataset based on the preset index structure includes: recalling the initial retrieval result from the target multimodal dataset based on the preset index structure; calculating the similarity between the multimodal mapping result and the initial retrieval result to obtain a calculation result; and selecting the candidate retrieval result matching the multimodal mapping result from the initial retrieval result based on the calculation result.7. The data processing method according to claim 4, characterized in that the data processing method further includes: concatenating a portion of the candidate search results in the sorting results based on a preset prompt text format to obtain the target prompt text. 8. The data processing method according to claim 7, characterized in that using the multimodal generation sub-model to perform response content generation processing on the sorting results to obtain the target answer includes: inputting the target prompt text into the multimodal generation sub-model, so that the multimodal generation sub-model performs response content generation processing based on the multimodal concatenation data recorded in the target prompt text to obtain the target answer. 9. A data processing method, characterized in that it includes: acquiring multimodal product query data, wherein the multimodal product query data is used to provide descriptive information of products to be selected; using a target retrieval generation model to perform product knowledge retrieval and product information generation on the multimodal product query data to obtain a target product answer; wherein the target retrieval generation model is used to obtain target product retrieval results that meet preset filtering conditions from candidate product retrieval results associated with the multimodal product query data, and to generate response content contained in the target product answer based on the target product retrieval results, wherein the preset filtering conditions are used to filter the candidate product retrieval results according to the degree of association of the candidate product retrieval results. 10. A data processing method, characterized in that it includes: obtaining a data processing request through a first application programming interface (API), wherein the request data carried in the data processing request includes: multimodal query data; returning a data processing response through a second API, wherein the response data carried in the data processing response includes: a target answer, wherein the target answer is obtained by performing knowledge retrieval and information generation on the multimodal query data using a target retrieval generation model, the target retrieval generation model being used to obtain target retrieval results that meet preset filtering conditions from candidate retrieval results associated with the multimodal query data, and to generate response content contained in the target answer based on the target retrieval results, wherein the preset filtering conditions are used to filter the candidate retrieval results according to the ranking result of the degree of association of the candidate retrieval results. 11. A data processing method, characterized in that it includes: acquiring a currently input data processing dialogue request, wherein the request data carried in the data processing dialogue request includes: multimodal query data; responding to the data processing dialogue request, returning a data processing dialogue response, wherein the information carried in the data processing dialogue response includes: a target answer, wherein the target answer is obtained by performing knowledge retrieval and information generation on the multimodal query data using a target retrieval generation model, and the target retrieval generation model is used to extract relevant information from the multimodal query data.The method involves obtaining target search results that meet preset filtering conditions from candidate search results, and generating response content contained in the target answer based on the target search results. The preset filtering conditions are used to filter the candidate search results according to the relevance ranking of the candidate search results. The target answer is then displayed in a graphical user interface. 12. A model training method, characterized in that it includes: (Claims 2 / 3, page 3, CN 122220450 A) Obtaining multimodal training data; Adjusting a pre-trained model using a target training strategy and the multimodal training data to obtain a target retrieval generation model, wherein a cross-modal attention mechanism is introduced in the target training strategy; Deploying the target retrieval generation model to an execution device, wherein the target retrieval generation model is invoked by the execution device to execute the data processing method according to any one of claims 1 to 11. 13. A data processing system, characterized in that it comprises: a client for sending multimodal query data; a server connected to the client for performing knowledge retrieval and information generation on the multimodal query data using a target retrieval generation model to obtain a target answer, wherein the target retrieval generation model is used to obtain target retrieval results that meet preset filtering conditions from candidate retrieval results associated with the multimodal query data, and to generate response content contained in the target answer based on the target retrieval results, wherein the preset filtering conditions are used to filter the candidate retrieval results according to the degree of association of the candidate retrieval results; the client is further used to output the target answer. 14. An electronic device, characterized in that it comprises: a memory storing an executable program; a processor for running the program, wherein the program executes the data processing method according to any one of claims 1 to 11 or the model training method according to claim 12 when running. 15. A computer-readable storage medium, characterized in that the computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device where the computer-readable storage medium is located to execute the data processing method according to any one of claims 1 to 11 or the model training method according to claim 12. 16. A computer program product, characterized in that it includes a computer program, which, when executed by a processor, implements the data processing method according to any one of claims 1 to 11 or the model training method according to claim 12. Claims 3 / 3 Page 4 CN 122220450 A Data Processing Method, Model Training Method, Electronic Device and Storage Medium Technical Field

[0001] This application relates to large model technology and computer technology, specifically, to a data processing method, a model training method, an electronic device and a storage medium. Background Art

[0002] With the rapid development of multimodal pre-trained language model technology and applications, user data is showing a trend of diversification and multimodality. Traditional large-model retrieval-augmented generation (RAG) mainly relies on single-modal data, such as text retrieval relying on text descriptions. However, single-modal data often fails to fully express user needs, resulting in limited accuracy and relevance of retrieval results. In order to adapt to multimodal understanding scenarios, related technologies can convert all multimodal content into text content, and then rely on text retrieval and large-scale text language models to achieve multimodal retrieval-augmented generation. However, this approach has shortcomings in information fusion, which affects the comprehensiveness and accuracy of retrieval results. Related technologies can also use image-text retrieval schemes to decompose multimodal content input and process it separately. However, this approach not only increases the complexity of the system, but also limits the model's ability to deeply integrate multimodal information, resulting in low retrieval efficiency and difficulty in dealing with complex multimodal scenarios. Therefore, multimodal retrieval systems in related technologies still face problems such as low retrieval generation efficiency and insufficient accuracy when processing large-scale and diverse data.

[0003] No effective solution has yet been proposed to address the above-mentioned problems. Summary of the Invention

[0004] Embodiments of this application provide a data processing method, a model training method, an electronic device, and a storage medium to at least solve the technical problems of low retrieval generation efficiency and poor accuracy in the related art when retrieving multimodal query data.

[0005] According to one aspect of an embodiment of this application, a data processing method is provided, comprising: acquiring multimodal query data; using a target retrieval generation model to perform knowledge retrieval and information generation on the multimodal query data to obtain a target answer; wherein the target retrieval generation model is used to obtain target retrieval results that meet preset filtering conditions from candidate retrieval results associated with the multimodal query data, and to generate response content contained in the target answer based on the target retrieval results, and the preset filtering conditions are used to filter the candidate retrieval results according to the degree of association of the candidate retrieval results.

[0006] According to another aspect of the embodiments of this application, a data processing method is also provided, including: acquiring multimodal product query data, wherein the multimodal product query data is used to provide descriptive information of products to be purchased; using a target retrieval generation model to perform product knowledge retrieval and product information generation on the multimodal product query data to obtain a target product answer; wherein the target retrieval generation model is used to obtain target product retrieval results that meet preset filtering conditions from candidate product retrieval results associated with the multimodal product query data, and to generate response content contained in the target product answer based on the target product retrieval results, and the preset filtering conditions are used to filter the candidate product retrieval results according to the ranking results of the relevance of the candidate product retrieval results.

[0007] According to another aspect of the embodiments of this application, a data processing method is also provided, comprising: obtaining a data processing request through a first application programming interface, wherein the request data carried in the data processing request includes: multimodal query data; returning a data processing response through a second application programming interface, wherein the response data carried in the data processing response includes: a target answer, wherein the target answer is obtained by performing knowledge retrieval and information generation on the multimodal query data using a target retrieval generation model, the target retrieval generation model is used to obtain target retrieval results that meet preset filtering conditions from candidate retrieval results associated with the multimodal query data, and to generate response content contained in the target answer based on the target retrieval results, wherein the preset filtering conditions are used to filter the candidate retrieval results according to the ranking results of the relevance of the candidate retrieval results.

[0008] According to another aspect of the embodiments of this application, a data processing method is also provided, comprising: acquiring a currently input data processing dialogue request, wherein the request data carried in the data processing dialogue request includes: multimodal query data; responding to the data processing dialogue request and returning a data processing dialogue response, wherein the information carried in the data processing dialogue response includes: a target answer, wherein the target answer is obtained by performing knowledge retrieval and information generation on the multimodal query data using a target retrieval generation model, the target retrieval generation model is used to obtain target retrieval results that meet preset filtering conditions from candidate retrieval results associated with the multimodal query data, and to generate response content contained in the target answer based on the target retrieval results, the preset filtering conditions are used to filter the candidate retrieval results according to the degree of association of the candidate retrieval results; and displaying the target answer in a graphical user interface.

[0009] According to another aspect of the embodiments of this application, a model training method is also provided, comprising: acquiring multimodal training data; adjusting a pre-trained model using a target training strategy and the multimodal training data to obtain a target retrieval generation model, wherein a cross-modal attention mechanism is introduced in the target training strategy; and deploying the target retrieval generation model to an execution device, wherein the target retrieval generation model is invoked by the execution device to execute the methods in various embodiments of this application.

[0010] According to another aspect of the embodiments of this application, a data processing system is also provided, comprising: a client for sending multimodal query data; and a server connected to the client for using a target retrieval generation model to perform knowledge retrieval and information generation on the multimodal query data to obtain a target answer, wherein the target retrieval generation model is used to obtain target retrieval results that meet preset filtering conditions from candidate retrieval results associated with the multimodal query data, and to generate response content contained in the target answer based on the target retrieval results, wherein the preset filtering conditions are used to sort the candidate retrieval results according to their degree of association.The results are used to filter candidate search results; the client is also used to output the target answer.

[0011] According to another aspect of the embodiments of this application, a computing device is also provided, including: a memory storing an executable program; a processor for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0012] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory storing an executable program; a processor connected to the memory via a bus for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0013] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein the executable program controls the device where the computer-readable storage medium is located to execute the methods in various embodiments of this application when it runs.

[0014] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program, which implements the methods in various embodiments of this application when executed by a processor.

[0015] According to another aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the methods in various embodiments of this application.

[0016] According to another aspect of the embodiments of this application, a computer program is also provided, which, when executed by a processor, implements the methods in various embodiments of this application.

[0017] In the embodiments of this application, by acquiring multimodal query data, and then using a target retrieval generation model to perform knowledge retrieval and information generation on the multimodal query data, a target answer is obtained. The target retrieval generation model is used to obtain target retrieval results that meet preset filtering conditions from the candidate retrieval results associated with the multimodal query data (page 2 / 19 of the specification, 6 CN 122220450 A), and to generate response content contained in the target answer based on the target retrieval results. The preset filtering conditions are used to filter the candidate retrieval results according to the degree of association of the candidate retrieval results. Thus, using a target retrieval generation model to process multimodal query data can significantly improve the efficiency, accuracy, and user experience of information retrieval and generation, while enhancing the knowledge coverage and adaptability of the model. The target retrieval generation model can effectively filter target retrieval results that meet preset filtering conditions from a large number of candidate retrieval results. The filtering process is based on comprehensive analysis of multimodal query data, which is more comprehensive and accurate than relying on query understanding based on only a single modality. This achieves the goal of efficiently and accurately processing multimodal query data, thereby improving the technical effect of improving the retrieval generation efficiency and accuracy of multimodal query data, and thus solving the problem of multimodal query data processing in related technologies.The technical problems of low retrieval efficiency and poor accuracy exist when retrieving data in a state query.

[0018] It is easy to note that the above general description and the following detailed description are only for illustrative purposes and explanation, and do not constitute a limitation on the present application. Brief Description of the Drawings

[0019] The accompanying drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the accompanying drawings:

[0020] FIG1 is a schematic diagram of an application scenario of a data processing method according to an embodiment of the present application;

[0021] FIG2 is a flowchart of a data processing method according to an embodiment of the present application;

[0022] FIG3 is a schematic diagram of a multimodal recall sub-model according to an embodiment of the present application;

[0023] FIG4 is a schematic diagram of a target retrieval generation model according to an embodiment of the present application;

[0024] FIG5 is a flowchart of yet another data processing method according to an embodiment of the present application;

[0025] FIG6 is a flowchart of yet another data processing method according to an embodiment of the present application;

[0026] FIG7 is a flowchart of yet another data processing method according to an embodiment of the present application;

[0027] FIG8 is a flowchart of a model training method according to an embodiment of the present application;

[0028] FIG9 is a structural block diagram of a data processing device according to an embodiment of the present application;

[0029] FIG10 is a structural block diagram of yet another data processing device according to an embodiment of the present application;

[0030] FIG11 is a structural block diagram of yet another data processing device according to an embodiment of the present application;

[0031] Figure 12 is a structural block diagram of another data processing device according to an embodiment of this application;

[0032] Figure 13 is a structural block diagram of a model training device according to an embodiment of this application;

[0033] Figure 14 is a structural block diagram of a computing device according to an embodiment of this application;

[0034] Figure 15 is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Description

[0035] In order to enable those skilled in the art to better understand the solutions of this application, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0036] It should be noted that the terms "first," "second," etc. in the specification, claims, and above-mentioned drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use of the specification 3 / 19 pages 7 CN 122220450 AThe data may be interchanged where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than those illustrated or described herein. Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0037] The technical solutions provided in this application are mainly implemented using large model technology. Here, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, billions, trillions, or even tens of trillions of model parameters. Large models, also known as foundation models, are pre-trained using large-scale unlabeled corpora to produce pre-trained models with hundreds of millions of parameters. These models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0038] It should be noted that in practical applications, large models can be fine-tuned using a small number of samples to make them applicable to different tasks. For example, large models can be widely used in Natural Language Processing (NLP), computer vision, speech processing, and other fields. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and image generation, as well as natural language processing tasks such as text-based sentiment classification, text summarization, and machine translation. Therefore, the main application scenarios of large models include, but are not limited to, digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design. In this application embodiment, the data processing through a target retrieval generation model in a multimodal query scenario is used as an example for explanation.

[0039] First, some nouns or terms appearing in the description of the embodiments of this application are subject to the following interpretation:

[0040] Multimodal: generally refers to the combination of multiple different types of data or information patterns. Multimodal can include text, images, audio, video, etc. In the field of computing, multimodal processing means using these different types of data to improve the performance of tasks or achieve more complex functions.

[0041] Multimodal pre-trained large model: generally refers to a large-scale machine learning model that can process multiple data modalities simultaneously.Models can utilize complementary information from different types of data (such as text, images, audio, etc.) to improve the overall performance of the model. They are often used for complex tasks, such as text-to-image generation and image understanding.

[0042] Multimodal retrieval: Multimodal retrieval is an information retrieval technology that can use data from multiple modalities for searching. For example, users may provide text descriptions and images for retrieval. The system will return relevant results based on the comprehensive information of these modalities, thereby improving the accuracy of retrieval and user experience.

[0043] Large model retrieval enhancement (RAG): This is a method of combining large language models with external information retrieval systems to improve the knowledge coverage and answer accuracy of the model. This enhancement method compensates for the knowledge blind spots or lack of specific domain knowledge of large models after the training data deadline by accessing and utilizing relevant information in external databases, document libraries, or search engines in real time when generating answers.

[0044] Multimodal large model retrieval enhancement: Multimodal large model retrieval enhancement is based on large model retrieval enhancement and further extends to information retrieval and integration of multiple data modalities. This method not only utilizes text information, but also combines other types of media data to provide richer and more comprehensive answers or generated content.

[0045] In order to adapt to the scenario of multimodal understanding, the multimodal content can be converted into text content, and then multimodal retrieval enhancement generation can be achieved by relying on text retrieval and text language large model. In the related technology, the multimodal content input can also be decomposed and processed separately by using the image and text retrieval scheme. The retrieval generation method in the related technology has the following disadvantages: (1) Insufficient information fusion: The scheme in the related technology is lacking in the deep fusion of different modal data, and it is difficult to fully understand and utilize the correlation information between multimodalities. (2) Low computational efficiency: The complex model structure and multimodal data processing lead to high computational resource consumption, which limits the real-time performance and large-scale application of the retrieval system. (3) Lack of a unified pre-training mechanism: Most solutions rely on single-modal pre-trained models and lack a unified pre-training mechanism for multimodal data, resulting in poor multimodal feature extraction and fusion effects.

[0046] According to an embodiment of this application, a data processing method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than that shown here.

[0047] Considering the large number of model parameters of large models and the limited computing resources of mobile terminals, the above-mentioned method provided by the embodiments of this application can be applied to the application scenario shown in Figure 1, but is not limited to this. In the application scenario shown in Figure 1In this embodiment, the large model is deployed on server 10. Server 10 can connect to one or more client devices 20 via LAN, WAN, Internet, or other types of data networks. Client devices 20 may include, but are not limited to, smartphones, tablets, laptops, PDAs, personal computers, smart home devices, and in-vehicle devices. Client devices 20 can interact with users through a graphical user interface to call the large model, thereby implementing the method provided in this application embodiment.

[0048] In this application embodiment, the system composed of client devices and server can perform the following steps: the server obtains the currently input data processing dialogue request from the client device. The request data carried in the data processing dialogue request includes multimodal query data. In response to the data processing dialogue request, the server returns a data processing dialogue reply to the client device. The information carried in the data processing dialogue response includes: the target answer, which is obtained by performing knowledge retrieval and information generation on the multimodal query data using a target retrieval generation model. The target retrieval generation model is used to obtain target retrieval results that meet preset filtering conditions from the candidate retrieval results associated with the multimodal query data, and to generate response content contained in the target answer based on the target retrieval results. The preset filtering conditions are used to filter the candidate retrieval results according to the degree of relevance of the candidate retrieval results. After obtaining the data processing dialogue response, the client device can display the target answer in the graphical user interface.

[0049] It should be noted that with the rapid development of high-performance computing units, the above-mentioned method provided in the embodiments of this application can also be applied to model all-in-one machines in other application scenarios. In an optional embodiment, the model all-in-one machine has multiple built-in models. Users can select a model to adjust as needed to obtain their own model. Thus, the high-performance computing unit built into the model all-in-one machine can directly call the adjusted model to execute the above-mentioned method provided in the embodiments of this application. In another optional embodiment, the large model all-in-one machine has a pre-trained model built in, so the high-performance computing unit built into the model all-in-one machine can directly call the model to execute the above-mentioned method provided in the embodiments of this application.

[0050] Further, when a user needs to train their own model, they can also upload their own dataset through the client. The dataset is sent by the client to the server, so that the server can adjust the pre-trained model with the dataset to obtain the user's own model, and then deploy it to the production environment. In order to facilitate the user's model adjustment needs, the server can provide complete adjustment tools, development framework and process, and can support multiple adjustment strategies, so that the adjusted model can better adapt to different domain applications and achieve high customization.

[0051] In the above operating environment, this application provides a data processing method as shown in Figure 2. Figure 2 is a diagram according to this application.A flowchart of a data processing method according to an embodiment. As shown in Figure 2, the method may include the following steps:

[0052] Step S21, obtaining multimodal query data; Specification 5 / 19 Page 9 CN 122220450 A

[0053] Step S22, using a target retrieval generation model to perform knowledge retrieval and information generation on the multimodal query data to obtain the target answer; wherein, the target retrieval generation model is used to obtain target retrieval results that meet preset filtering conditions from the candidate retrieval results associated with the multimodal query data, and to generate response content contained in the target answer based on the target retrieval results, and the preset filtering conditions are used to filter the candidate retrieval results according to the degree of association of the candidate retrieval results.

[0054] The above-mentioned multimodal query data can be query data containing two or more different modal information, and different modalities include but are not limited to text, images, audio, video, etc. In practical applications, multimodal query data can be input containing multiple types of information provided by users when searching, asking or interacting, for example, a user uploads an image and attaches a descriptive text to ask for detailed information about a certain product in the image.

[0055] For example, the above-mentioned methods for obtaining multimodal query data include, but are not limited to, user input collection, social media collection, mobile application logs, and database integration. Specifically, multimodal query data can be collected directly from user-provided queries. For example, search engines or intelligent customer service systems can receive queries containing text and images, combined queries containing voice and text, etc. Images, videos, and text comments can be collected from social media platforms as sources of multimodal query data, which can be used to analyze user interests or trends. User query data can be extracted from mobile application logs, including voice input, photos taken, and text descriptions by users in the application. Records containing different modal information can be integrated from existing databases, such as product images and descriptive text in a product database. In addition, multimodal query data can be obtained from user interaction records of virtual assistants and smart speakers. When a smart device processes a user's voice command, if the user also provides images or videos, the device interaction records can constitute multimodal query data.

[0056] It should be noted that when obtaining multimodal query data, it is necessary to ensure the quality and diversity of the data, and to comply with data usage regulations to ensure compliant use of the data. In addition, preprocessing the collected user query data, such as noise reduction, format standardization, and quality checks, can further ensure the quality of subsequent retrieval generation.

[0057] After obtaining the multimodal query data, the multimodal query data is input into the target retrieval generation model. The target retrieval generation model performs knowledge retrieval and information generation on the multimodal query data to obtain the target answer. During the knowledge retrieval process, the target retrieval generation model can retrieve candidate retrieval results related to the multimodal query data from the multimodal knowledge base.As a result, the multimodal knowledge base can be a large collection of documents, a database, or Internet resources containing information of multiple modalities. In the information generation process, the candidate retrieval results are sorted based on the degree of correlation between the candidate retrieval results and the multimodal query data to obtain the correlation degree ranking result. The candidate retrieval results are then filtered according to the correlation degree ranking result, and the target retrieval result that meets the preset filtering conditions is selected from the ranked candidate retrieval results. For example, the top k candidate retrieval results in the correlation degree ranking result are selected as the target retrieval result. Using the information in the target retrieval result and combining the modal characteristics of the multimodal query data, a comprehensive, multimodal target answer is generated. The response content contained in the target answer may include, but is not limited to, text descriptions, image examples, audio explanations, or video demonstrations to meet the user's comprehensive information needs.

[0058] For example, the target retrieval generation model can be obtained by adjusting the pre-trained model using a target training strategy and multimodal training data. A cross-modal attention mechanism can be introduced into the target training strategy. The target retrieval generation model is deployed to the execution device, which calls the data processing method of any one of the embodiments of this application to execute the target retrieval generation model.

[0059] The aforementioned multimodal training data can be obtained from various channels, including but not limited to social media, online forums, Q&A communities, news websites, educational materials, and industry-specific databases (such as medical, legal, and technological databases). Multimodal training data typically contains information from multiple modalities, such as text, images, audio, or video, and there are correlations or relationships between these modalities. Since the acquired raw data may contain noise and irrelevant information, preprocessing is required, such as data cleaning, format conversion, size standardization, and text segmentation, so that the pre-trained model can learn effectively. For supervised learning, data can be labeled manually or automatically to indicate the correlations between different modalities, thereby helping the model learn the correct correspondence between modalities.

[0060] The preprocessed multimodal training data is input into the pre-trained model, allowing the model to be fine-tuned or further trained on the dataset. During training, target training strategies, such as cross-modal attention mechanisms, are used to guide the updating of model parameters to improve the ability to process multimodal information. A cross-modal attention mechanism is introduced into the model architecture, enabling the model to pay attention to and integrate information from different modalities when processing input, thereby enhancing mutual understanding and connection between modalities.

[0061] The data processing method in this application embodiment can be applied to scenarios such as intelligent customer service, e-commerce, news, medical care, education, social media, and smart homes, and this application embodiment does not limit it. The above application scenarios require the system to be able to process and understand complex multimodal information and provide accurate, timely, and relevant responses or content. By fusing text,Query data in multiple modalities, such as images and audio, can effectively improve the accuracy and efficiency of information retrieval and content generation. For example, in intelligent customer service and Q&A systems, when users inquire about product information or seek advice, they can simultaneously provide text descriptions and examples of images, audio, or video, requesting the system to provide accurate and comprehensive answers. In e-commerce search and recommendation systems, when users search for products on e-commerce platforms, they can upload product images and provide text descriptions to more precisely express their needs. Social media platforms, when analyzing user behavior or content trends, need to process composite information including text, images, and videos to more accurately understand user needs and interests, providing a basis for content recommendation and ad targeting.

[0062] Based on the above steps S21 to S22, by acquiring multimodal query data, and then using a target retrieval generation model to perform knowledge retrieval and information generation on the multimodal query data, the target answer is obtained. The target retrieval generation model is used to obtain target retrieval results that meet preset filtering conditions from the candidate retrieval results associated with the multimodal query data, and to generate response content contained in the target answer based on the target retrieval results. The preset filtering conditions are used to filter the candidate retrieval results according to the degree of association of the candidate retrieval results. Thus, using the target retrieval generation model to process multimodal query data can significantly improve the efficiency, accuracy and user experience of information retrieval and generation, while enhancing the knowledge coverage and adaptability of the model. The target retrieval generation model can effectively filter target retrieval results that meet preset filtering conditions from a large number of candidate retrieval results. The filtering process is based on the comprehensive analysis of multimodal query data, which is more comprehensive and accurate than relying on a single modality query. This achieves the goal of efficiently and accurately processing multimodal query data, thereby improving the technical effect of improving the retrieval generation efficiency and accuracy of multimodal query data. It also solves the technical problems of low retrieval generation efficiency and poor accuracy in related technologies when retrieving multimodal query data.

[0063] The data processing method in the embodiments of this application will be further described below.

[0064] In an optional embodiment, the target retrieval generation model includes: a multimodal recall sub-model, a multimodal ranking sub-model, and a multimodal generation sub-model. In step S22, the target retrieval generation model is used to perform knowledge retrieval and information generation on the multimodal query data to obtain the target answer, including:

[0065] Step S221, using the multimodal recall sub-model to perform recall processing on the multimodal query data to obtain candidate retrieval results;

[0066] Step S222, using the multimodal ranking sub-model to perform ranking processing on the candidate retrieval results to obtain ranking results;

[0067] Step S223, using the multimodal generation sub-model to perform response content generation processing on the ranking results to obtain the target answer.

[0068] In the process of recalling multimodal query data, a multimodal recall sub-model can be used to extract deep features from the multimodal query data, converting data of different modalities such as text, images, audio, or video into a unified feature representation. This can usually be done in the embedding layer of the multimodal recall sub-model. The multimodal recall sub-model has the ability to process multimodal information and can generate high-quality multimodal feature vectors. Further, based on the extracted multimodal features, vector recall is performed to retrieve candidate documents or instruction manual fragments that match the multimodal feature vectors from a large-scale multimodal knowledge base, thereby obtaining candidate retrieval results.

[0069] In the process of sorting the candidate retrieval results, a multimodal sorting sub-model can be used to evaluate the recalled candidate retrieval results and assign a relevance score to each candidate retrieval result. The relevance score is calculated by analyzing the correlation between each modal information in the candidate retrieval results and the multimodal query data, as well as the mutual influence between modalities. Furthermore, based on the relevance score, the candidate retrieval results are comprehensively ranked using a multimodal ranking sub-model to obtain a ranking result, which includes multiple candidate retrieval results. It should be noted that the ranking algorithms used in the ranking process include, but are not limited to, ranking based on vector similarity, ranking based on multimodal attention mechanisms, or personalized ranking algorithms that combine user behavior data and contextual information. This application embodiment does not limit these algorithms.

[0070] In the process of generating response content from the ranking results, the multimodal generation sub-model processes the ranking results to fuse multimodal information. Specifically, this may include integrating relevant information from modalities such as text, images, audio, or video, so that all available information can be fully considered when generating the target answer. Based on the fused multimodal information and the original multimodal query data, the multimodal generation sub-model can generate comprehensive response content in the target answer. The target answer can be a detailed and accurate answer containing multimodal information, such as an answer containing textual explanations and relevant image examples, or a guidance scheme combining textual descriptions and audio explanations. After generating the response content, the multimodal generation sub-model can also adjust the response content to ensure the logical coherence and information integrity of the output content, specifically including but not limited to proofreading the generated content, supplementing missing information, or adjusting the format of the generated content according to the context.

[0071] Based on the above optional embodiments, by using a multimodal recall sub-model to perform recall processing on multimodal query data to obtain candidate retrieval results, then using a multimodal ranking sub-model to perform ranking processing on the candidate retrieval results to obtain ranking results, and finally using a multimodal generation sub-model to perform response content generation processing on the ranking results to obtain the target answer, which is then processed by...This target retrieval generative model can effectively handle complex multimodal queries, accurately retrieve relevant data from the knowledge base, and generate a high-quality response containing multiple modal information, which can significantly improve user experience and information acquisition efficiency.

[0072] In an optional embodiment, the multimodal recall sub-model includes: a feature encoding module, a feature mapping module, and a feature matching module. In step S221, the multimodal recall sub-model is used to perform recall processing on the multimodal query data to obtain candidate retrieval results, including:

[0073] Step S2211: The feature encoding module is used to perform feature encoding processing on the multimodal query data to obtain multimodal feature vectors;

[0074] Step S2212: The feature mapping module is used to perform feature mapping processing on the multimodal feature vectors to obtain multimodal mapping results, wherein the multimodal mapping results are used to represent the multimodal representation obtained by mapping feature vectors of different modalities to a shared multimodal space;

[0075] Step S2213: The feature matching module is used to recall candidate retrieval results that match the multimodal mapping results from the target multimodal dataset.

[0076] During the feature encoding process, the multimodal query data is input to the feature encoding module, and the feature encoding module is used to perform deep feature extraction on the multimodal query data. For example, for text data, word segmentation and word embedding can be performed; for image data, image feature extraction can be performed to obtain feature maps; for audio and video data, corresponding time-frequency features or scene features can be extracted. Furthermore, the feature encoding module can integrate the extracted feature data into a unified multimodal feature vector. The multimodal feature vector is a high-dimensional feature vector that can contain core information and cross-modal associations in the multimodal query data.

[0077] During the feature mapping process, the feature mapping module maps the generated multimodal feature vector to a shared multimodal space. The multimodal space is a unified representation space. The multimodal feature vector contains feature vectors from different modalities, and these feature vectors can be effectively compared and matched through a cross-modal attention mechanism. The result of the feature mapping process is a multimodal mapping result, which can represent the multimodal representation obtained by mapping feature vectors from different modalities to the shared multimodal space. This allows information from different modalities to be converted into a unified form that can be used for similarity calculation.

[0078] Further, the feature matching module calculates a similarity score between the multimodal mapping result and the feature vector of each document or fragment in the target multimodal dataset within the shared multimodal space. Based on the similarity score, the feature matching module recalls candidate retrieval results that are relatively well-matched from the target multimodal dataset. The recall process can employ near-similarity...Similar to the nearest neighbor algorithm, it can quickly find the most similar items on large-scale datasets. The recalled candidate retrieval results are documents or fragments that match the feature vector of the query data in the multimodal space.

[0079] Figure 3 is a schematic diagram of a multimodal recall sub-model according to an embodiment of this application. As shown in Figure 3, the feature encoding module performs feature encoding processing on the multimodal query data to obtain a multimodal feature vector. The feature mapping module performs feature mapping processing on the multimodal feature vector to obtain a multimodal mapping result. The feature matching module recalls candidate retrieval results that match the multimodal mapping result from the target multimodal dataset.

[0080] Based on the above optional embodiments, by using a feature encoding module to perform feature encoding processing on the multimodal query data to obtain a multimodal feature vector, and then using a feature mapping module to perform feature mapping processing on the multimodal feature vector to obtain a multimodal mapping result, and finally using a feature matching module to recall candidate retrieval results that match the multimodal mapping result from the target multimodal dataset, it is possible to effectively process complex multimodal query data. Through deep feature encoding and mapping, multimodal query data is converted into a unified representation form, and highly relevant candidate results are quickly recalled from a large-scale multimodal dataset, thereby providing high-quality input data for subsequent sorting and generation stages.

[0081] In an optional embodiment, in step S2213, using a feature matching module to recall candidate retrieval results that match the multimodal mapping result from the target multimodal dataset includes: in the feature matching module, based on a preset index structure, recalling candidate retrieval results that match the multimodal mapping result from the target multimodal dataset.

[0082] The above-mentioned preset index structure includes, but is not limited to, inverted index, vector index, or other data structures. The preset index structure can improve retrieval efficiency and accuracy, and can be flexibly selected according to the characteristics of the dataset and task requirements.

[0083] Specifically, in processing text data, inverted index is a common index structure. Inverted index is built based on a vocabulary or feature dictionary, recording which documents each word or feature appears in and its position in the document. Inverted index can quickly locate documents containing specific words or features, and is more effective for text-intensive multimodal datasets. For feature vector-based retrieval, vector index is a more efficient choice. Vector index stores the feature vector of each document or segment in the dataset and quickly finds the vector most similar to the query vector through an approximate nearest neighbor algorithm. Vector index can handle large-scale multimodal datasets and maintain good retrieval performance even in high-dimensional feature spaces.

[0084] Based on the above optional embodiments, by recalling candidate retrieval results that match the multimodal mapping results from the target multimodal dataset based on the preset index structure in the feature matching module, the feature matching module can retrieve candidate retrieval results that match the multimodal mapping results based on the preset index.The structure quickly retrieves candidate search results that match the multimodal query data, providing high-quality input for subsequent multimodal sorting and generation steps, thereby ensuring the performance and user experience of the entire RAG process.

[0085] In an optional embodiment, the data processing method in this application embodiment further includes:

[0086] obtaining an initial multimodal dataset; performing data cleaning and data preprocessing on the initial multimodal dataset to obtain an intermediate multimodal dataset; performing data alignment and data annotation processing on the intermediate multimodal dataset to obtain a target multimodal dataset; performing feature encoding on the target multimodal dataset to construct a preset index structure.

[0087] The above-mentioned initial multimodal dataset may include data containing multimodal information such as text, images, audio or video collected from the Internet, social media, professional databases, literature, image libraries, audio or video resources, etc., to ensure that the initial multimodal dataset covers a wide range of topics and fields, thereby improving the generalization ability of the model.

[0088] Further, data cleaning can remove duplicate, irrelevant, or low-quality data. For example, it can remove text containing spam, blurry images, blank or meaningless audio segments. The above data preprocessing includes, but is not limited to, format unification and data augmentation. Format unification is to convert data into a uniform format for easier model processing. For example, it can scale all images to the same resolution and perform word segmentation and standardization on text. Data augmentation techniques, such as image flipping, text rewriting, and audio reverberation, can increase the diversity and richness of the dataset.

[0089] After obtaining the intermediate multimodal dataset through data cleaning and data preprocessing, the intermediate multimodal dataset is subjected to data alignment and data annotation to obtain the target multimodal dataset. Data alignment ensures that the multimodal information in the target multimodal dataset is related and consistent. For example, for image-text pairing, it ensures that the text description is related to the image content; for video and audio, it ensures that the timestamps of both are aligned. Data annotation involves adding labels or keywords to each multimodal entry in the intermediate multimodal dataset to indicate the relationship between queries and responses, as well as the correlation between modalities, thereby helping the model learn the correct multimodal correspondences and retrieval logic.

[0090] Feature extraction is performed on each data item in the target multimodal dataset to generate feature vectors containing multimodal information. Feature vectors from different modalities are fused into a shared multimodal space to ensure consistency and comparability between modalities. Based on the feature-encoded data, a pre-defined index structure is constructed, such as an inverted index or a vector index, to support efficient retrieval operations. The design of the index structure can be determined according to the size of the dataset, the dimension of the feature vectors, and the needs of the retrieval task.

[0091] Based on the above optional embodiments, by obtaining an initial multimodal dataset, then performing data cleaning and preprocessing on the initial multimodal dataset to obtain an intermediate multimodal dataset, and then performing data alignment and annotation on the intermediate multimodal dataset to obtain a target multimodal dataset, and finally performing feature encoding on the target multimodal dataset to construct a preset index structure, thereby quickly recalling candidate retrieval results that match the multimodal query data, providing high-quality input for subsequent multimodal sorting and generation steps, and further improving retrieval generation efficiency.

[0092] In an optional embodiment, recalling candidate retrieval results that match the multimodal mapping results from the target multimodal dataset based on the preset index structure includes:

[0093] Recalling the initial retrieval results from the target multimodal dataset based on the preset index structure; performing similarity calculation on the multimodal mapping results and the initial retrieval results to obtain the calculation results; and selecting candidate retrieval results that match the multimodal mapping results from the initial retrieval results based on the calculation results.

[0094] Specifically, a pre-defined index structure is used to query the target multimodal dataset. The pre-defined index structure can quickly locate and recall a series of potentially related documents or fragments as initial search results in the target multimodal dataset. For each recalled initial search result, a similarity score is calculated between it and the multimodal mapping result, for example, by calculating the cosine similarity or Euclidean distance between vectors. When calculating similarity, a cross-modal attention mechanism can also be used to gain a deeper understanding of the correlation between different modalities, thereby providing more accurate matching results.

[0095] Based on the similarity calculation results, the initial search results are sorted from high to low according to the relevance score, and the top K initial search results are selected from the sorted list as candidate search results. The value of K is adjusted based on actual needs and the system performance specification (page 10 / 19, CN 122220450 A) to ensure the accuracy and diversity of the candidate search results.

[0096] Based on the above optional embodiments, the initial retrieval results are retrieved from the target multimodal dataset based on a preset index structure, and then the similarity between the multimodal mapping results and the initial retrieval results is calculated to obtain the calculation results. Finally, based on the calculation results, candidate retrieval results that match the multimodal mapping results are selected from the initial retrieval results, providing high-quality input for subsequent sorting and generation steps, thereby ensuring the fast response and high relevance of the retrieval process.

[0097] In an optional embodiment, the data processing method in this application embodiment further includes:

[0098] Concatenating some candidate retrieval results in the sorting results based on a preset prompt text format to obtain target prompt text.

[0099] The above preset prompt text format is a template that can define how to integrate the sorting results into a unified...In the text structure, the preset prompt text format may include specific text tags, instructions, or placeholders to indicate the content type and location that the generation model focuses on.

[0100] Specifically, by concatenating the first few relevant contents in the retrieved ranking results with prompt words in the preset prompt text format, a comprehensive input can be formed, i.e., the target prompt text is obtained. For example, the first few candidate search results in the ranking results are reorganized according to the requirements of the preset prompt text format to ensure that each modal information is properly inserted into the target prompt text. The target prompt text contains multimodal concatenation data. During the concatenation process, the format of the information can also be fine-tuned to ensure its compatibility with the preset prompt text format, while maintaining the integrity and accuracy of the information.

[0101] Based on the above optional embodiments, by concatenating some candidate search results in the ranking results based on the preset prompt text format, the target prompt text is obtained. This ensures that the multimodal generation sub-model can make full use of the multimodal information in the search results when generating response content, providing richer and more accurate answers, and further improving the accuracy of the search generation results.

[0102] In an optional embodiment, in step S223, the multimodal generation sub-model is used to process the ranking results to generate response content and obtain the target answer, including:

[0103] Inputting the target prompt text into the multimodal generation sub-model, so that the multimodal generation sub-model performs response content generation processing based on the multimodal splicing data recorded in the target prompt text to obtain the target answer.

[0104] Specifically, the target prompt text is input into the multimodal generation sub-model. The multimodal generation sub-model can combine information from multiple modalities such as text and images to understand the context and generate the final target answer. The generated result not only depends on the model's generation capability but also integrates the retrieved highly relevant information to ensure the accuracy and richness of the output content.

[0105] Figure 4 is a schematic diagram of a target retrieval generation model according to an embodiment of the present application. As shown in Figure 4, the target retrieval generation model includes: a multimodal recall sub-model, a multimodal ranking sub-model, and a multimodal generation sub-model. The target retrieval generation model is used to perform knowledge retrieval and information generation on multimodal query data to obtain the target answer. In the process, the multimodal recall sub-model is used to perform recall processing on the multimodal query data to obtain candidate retrieval results. The multimodal ranking sub-model is used to perform ranking processing on the candidate retrieval results to obtain ranking results. The multimodal generation sub-model is used to perform response content generation processing on the ranking results to obtain the target answer.

[0106] This embodiment of the present application targets multimodal retrieval enhancement scenarios. Based on the target generation retrieval model, a unified multimodal retrieval generation architecture can be constructed, capable of simultaneously processing multiple data modalities such as text and images. It achieves this through a cross-modal attention mechanism.This approach achieves deep information fusion. This unified architecture avoids the fragmented processing between different modalities in traditional multimodal models, enhancing the overall understanding and representation capabilities of the model. Specifically, based on the target generation retrieval model, a recall + ranking link is constructed in the multimodal retrieval chain (page 11 / 19, CN 122220450 A) to achieve efficient fusion of multimodal features. Cross-modal attention and a shared representation space are utilized to achieve deep fusion and collaborative expression of features from different modalities. This fusion mechanism not only enhances the expressive power of features but also significantly improves the matching accuracy between data from different modalities. Utilizing retrieval enhancement strategies based on large model features, including deep query understanding, intelligent index construction, and efficient similarity calculation, the strong representation capabilities of the target retrieval generation model are combined with efficient retrieval algorithms, significantly improving the accuracy and response speed of the retrieval system.

[0107] Figure 5 is a flowchart of another data processing method according to an embodiment of this application. As shown in Figure 5, the method may include the following steps:

[0108] Step S51: Obtain multimodal product query data, wherein the multimodal product query data is used to provide description information of the products to be purchased;

[0109] Step S52: Use a target retrieval generation model to perform product knowledge retrieval and product information generation on the multimodal product query data to obtain the target product answer; wherein the target retrieval generation model is used to obtain the target product retrieval results that meet the preset filtering conditions from the candidate product retrieval results associated with the multimodal product query data, and to generate the response content contained in the target product answer based on the target product retrieval results. The preset filtering conditions are used to filter the candidate product retrieval results according to the ranking results of the correlation degree of the candidate product retrieval results.

[0110] Based on the above steps S51 to S52, by acquiring multimodal product query data, and then using a target retrieval generation model to perform product knowledge retrieval and product information generation on the multimodal product query data, the target product answer is obtained. The target retrieval generation model is used to obtain target product retrieval results that meet preset filtering conditions from the candidate product retrieval results associated with the multimodal product query data, and to generate response content contained in the target product answer based on the target product retrieval results. The preset filtering conditions are used to filter the candidate product retrieval results according to the ranking results of the relevance of the candidate product retrieval results. Thus, using the target retrieval generation model to process multimodal product query data can significantly improve the efficiency, accuracy, and user experience of information retrieval and generation, while enhancing the knowledge coverage and adaptability of the model. The target retrieval generation model can effectively filter target product retrieval results that meet preset filtering conditions from a large number of candidate product retrieval results. The filtering process is based on the comprehensive analysis of multimodal product query data, compared to relying only on query understanding of a single modality.More comprehensive and accurate, thus achieving the goal of efficiently and accurately processing multimodal product query data, thereby improving the retrieval generation efficiency and accuracy of multimodal product query data, and solving the technical problems of low retrieval generation efficiency and poor accuracy in related technologies when retrieving multimodal query data.

[0111] Figure 6 is a flowchart of another data processing method according to an embodiment of this application. As shown in Figure 6, the method may include the following steps:

[0112] Step S61: Obtain a data processing request through a first application programming interface (API), wherein the request data carried in the data processing request includes: multimodal query data;

[0113] Step S62: Return a data processing response through a second API, wherein the response data carried in the data processing response includes: a target answer, which is obtained by performing knowledge retrieval and information generation on the multimodal query data using a target retrieval generation model. The target retrieval generation model is used to obtain target retrieval results that meet preset filtering conditions from the candidate retrieval results associated with the multimodal query data, and to generate response content contained in the target answer based on the target retrieval results. The preset filtering conditions are used to filter the candidate retrieval results according to the degree of association of the candidate retrieval results.

[0114] The first API and the second API can be the same API or different APIs. In an optional embodiment, the interface parameters in the first and second application programming interfaces (APIs) may include, but are not limited to: a global interface identifier, an interface signature key, an interface timestamp, an interface request identifier, and a system call credential identifier. The first API can use GET or POST as the interface request method to obtain file processing requests. The second API can use JSON format to return file processing responses.

[0115] Based on the above steps S61 to S62, a data processing request is obtained through the first application programming interface. The request data carried in the data processing request includes: multimodal query data. Then, a data processing response is returned through the second application programming interface. The response data carried in the data processing response includes: the target answer. The target answer is obtained by performing knowledge retrieval and information generation on the multimodal query data using a target retrieval generation model. The target retrieval generation model is used to obtain target retrieval results that meet preset filtering conditions from the candidate retrieval results associated with the multimodal query data, and to generate response content contained in the target answer based on the target retrieval results. The preset filtering conditions are used to filter the candidate retrieval results according to the degree of association of the candidate retrieval results. Thus, the target retrieval generation model is used to process the multimodal query data.According to reports, this method can significantly improve the efficiency, accuracy, and user experience of information retrieval and generation, while enhancing the model's knowledge coverage and adaptability. The target retrieval generation model can effectively filter target retrieval results that meet preset filtering conditions from a large number of candidate retrieval results. The filtering process is based on comprehensive analysis of multimodal query data, which is more comprehensive and accurate than relying solely on a single modality for query understanding. This achieves the goal of efficiently and accurately processing multimodal query data, thereby improving the technical efficiency and accuracy of multimodal query data retrieval and generation. It also solves the technical problems of low retrieval generation efficiency and poor accuracy in related technologies when retrieving multimodal query data.

[0116] Figure 7 is a flowchart of another data processing method according to an embodiment of this application. As shown in Figure 7, the method may include the following steps:

[0117] Step S71: Obtain the currently input data processing dialogue request, wherein the request data carried in the data processing dialogue request includes: multimodal query data;

[0118] Step S72: In response to the data processing dialogue request, return a data processing dialogue reply, wherein the information carried in the data processing dialogue reply includes: target answer, which is obtained by performing knowledge retrieval and information generation on the multimodal query data using a target retrieval generation model. The target retrieval generation model is used to obtain target retrieval results that meet preset filtering conditions from the candidate retrieval results associated with the multimodal query data, and to generate response content contained in the target answer based on the target retrieval results. The preset filtering conditions are used to filter the candidate retrieval results according to the degree of association of the candidate retrieval results;

[0119] Step S73: Display the target answer in the graphical user interface.

[0120] Based on the above steps S71 to S73, by acquiring the currently input data processing dialogue request, the request data carried in the data processing dialogue request includes: multimodal query data. Then, in response to the data processing dialogue request, a data processing dialogue reply is returned. The information carried in the data processing dialogue reply includes: the target answer. The target answer is obtained by using a target retrieval generation model to perform knowledge retrieval and information generation on the multimodal query data. The target retrieval generation model is used to obtain target retrieval results that meet preset filtering conditions from the candidate retrieval results associated with the multimodal query data, and to generate response content contained in the target answer based on the target retrieval results. The preset filtering conditions are used to filter the candidate retrieval results according to the degree of association of the candidate retrieval results. Finally, the target answer is displayed in the graphical user interface. Thus, using the target retrieval generation model to process multimodal query data can significantly improve the efficiency, accuracy, and user experience of information retrieval and generation, while enhancing the knowledge coverage and adaptability of the model. The target retrieval generation model can effectively filter out target retrieval results that meet preset filtering conditions from a large number of candidate retrieval results. The filtering process is based on multimodal query data.According to the comprehensive analysis, compared with relying on a single modality query understanding, it is more comprehensive and accurate, thereby achieving the goal of efficiently and accurately processing multimodal query data, thus realizing the technical effect of improving the retrieval generation efficiency and accuracy of multimodal query data, and thus solving the technical problems of low retrieval generation efficiency and poor accuracy in the related technology when retrieving multimodal query data.

[0121] Figure 8 is a flowchart of a model training method according to an embodiment of this application. As shown in Figure 8, the method may include the following steps:

[0122] Step S81, obtaining multimodal training data;

[0123] Step S82, adjusting the pre-trained model using the target training strategy and multimodal training data to obtain a target retrieval generation model, wherein a cross-modal attention mechanism is introduced in the target training strategy;

[0124] Step S83, deploying the target retrieval generation model to the execution device, wherein the target retrieval generation model is called by the execution device to execute any one of the data processing methods in the embodiments of this application.

[0125] Based on the above steps S81 to S83, by acquiring multimodal training data, and then adjusting the pre-trained model using the target training strategy and multimodal training data, a target retrieval generation model is obtained. A cross-modal attention mechanism is introduced into the target training strategy. Finally, the target retrieval generation model is deployed to the execution device. Thus, the introduced cross-modal attention mechanism enables the model to more effectively capture and integrate the correlation information between different modal data. Therefore, when understanding and generating content, it can make full use of the complementary characteristics of multimodal training data, further significantly enhance the model's ability to process multimodal queries, improve the efficiency and accuracy of retrieval generation, optimize the use of computing resources, improve the user experience, and make the information retrieval and generation tasks on the execution device more efficient and intelligent. The target retrieval generation model can effectively filter target retrieval results that meet preset filtering conditions from a large number of candidate retrieval results. The filtering process is based on the comprehensive analysis of multimodal query data, which is more comprehensive and accurate than relying on a single modality query understanding. This achieves the goal of efficiently and accurately processing multimodal query data, thereby improving the retrieval generation efficiency and accuracy of multimodal query data. This solves the technical problems of low retrieval generation efficiency and poor accuracy in related technologies when retrieving multimodal query data.

[0126] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations.The relevant laws, regulations and standards of the relevant countries and regions are provided, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0127] It should be noted that, for the foregoing method embodiments, in order to simplify the description, they are all described as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0128] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. Based on this understanding, the technical solution of this application, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0129] According to an embodiment of this application, a data processing apparatus for implementing the above-described data processing method is also provided. FIG9 is a structural block diagram of a data processing apparatus according to an embodiment of this application. As shown in FIG9, the apparatus includes: Specification 14 / 19 pages 18 CN 122220450 A

[0130] an acquisition module 901, used to acquire multimodal query data;

[0131] a processing module 902, used to perform knowledge retrieval and information generation on the multimodal query data using a target retrieval generation model to obtain a target answer; wherein, the target retrieval generation model is used to acquire target retrieval results that meet preset filtering conditions from candidate retrieval results associated with the multimodal query data, and to generate response content contained in the target answer based on the target retrieval results, and the preset filtering conditions are used to filter the candidate retrieval results according to the ranking results of the correlation degree of the candidate retrieval results.

[0132] Optionally, the processing module 902 is further configured to: perform recall processing on the multimodal query data using a multimodal recall sub-model to obtain candidate retrieval results; perform sorting processing on the candidate retrieval results using a multimodal ranking sub-model to obtain ranking results; and perform response content generation processing on the ranking results using a multimodal generation sub-model to obtain the target answer.

[0133] Optionally, the processing module 902 is further configured to: perform feature encoding processing on the multimodal query data using a feature encoding module to obtain multimodal feature vectors; and perform feature mapping processing on the multimodal feature vectors using a feature mapping module to obtain multimodal mapping results, wherein the multimodal mapping results are used to represent mapping feature vectors of different modalities to a shared multimodal feature vector.The multimodal representation obtained from the modality space; the feature matching module is used to recall candidate retrieval results that match the multimodal mapping results from the target multimodal dataset.

[0134] Optionally, the processing module 902 is further used to: in the feature matching module, recall candidate retrieval results that match the multimodal mapping results from the target multimodal dataset based on a preset index structure.

[0135] Optionally, the acquisition module 901 is further used to acquire the initial multimodal dataset; the data processing device further includes: a preprocessing module 903, used to perform data cleaning and data preprocessing on the initial multimodal dataset to obtain an intermediate multimodal dataset; a labeling module 904, used to perform data alignment and data labeling on the intermediate multimodal dataset to obtain the target multimodal dataset; and an encoding module 905, used to perform feature encoding on the target multimodal dataset to construct a preset index structure.

[0136] Optionally, the processing module 902 is further configured to: recall the initial retrieval results from the target multimodal dataset based on a preset index structure; calculate the similarity between the multimodal mapping results and the initial retrieval results to obtain the calculation results; and select candidate retrieval results that match the multimodal mapping results from the initial retrieval results based on the calculation results.

[0137] Optionally, the processing module 902 is further configured to: perform concatenation processing on some candidate retrieval results in the sorting results based on a preset prompt text format to obtain the target prompt text.

[0138] Optionally, the processing module 902 is further configured to: input the target prompt text into the multimodal generation sub-model so that the multimodal generation sub-model performs response content generation processing based on the multimodal concatenation data recorded in the target prompt text to obtain the target answer.

[0139] It should be noted here that the above-mentioned acquisition module 901 and processing module 902 correspond to steps S21 to S22 in the above embodiments. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in memory and processed by one or more processors. The above-mentioned modules may also be part of the device and run in the server 10 provided in the above embodiments.

[0140] FIG10 is a structural block diagram of another data processing device according to an embodiment of the present application. As shown in FIG10, the device includes:

[0141] an acquisition module 1001, used to acquire multimodal product query data, wherein the multimodal product query data is used to provide description information of the products to be purchased;

[0142] a processing module 1002, used to perform product knowledge retrieval and product information generation on the multimodal product query data using a target retrieval generation model to obtain the target product answer; wherein the target retrieval generation model is used to retrieve product knowledge and generate product information from the multimodal product query data.The system retrieves target product search results that meet preset filtering conditions from the candidate product search results associated with the data, and generates response content contained in the target product answer based on the target product search results. The preset filtering conditions are used to filter the candidate product search results according to the degree of association of the candidate product search results.

[0143] It should be noted here that the above-mentioned acquisition module 1001 and processing module 1002 correspond to steps S51 to S52 in the above embodiments. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also be part of the device and run in the server 10 provided in the above embodiments.

[0144] FIG11 is a structural block diagram of another data processing device according to an embodiment of the present application. As shown in FIG11, the device includes:

[0145] an acquisition module 1101, used to acquire a data processing request through a first application programming interface, wherein the request data carried in the data processing request includes: multimodal query data;

[0146] a return module 1102, used to return a data processing response through a second application programming interface, wherein the response data carried in the data processing response includes: a target answer, the target answer is obtained by performing knowledge retrieval and information generation on the multimodal query data using a target retrieval generation model, the target retrieval generation model is used to acquire target retrieval results that meet preset filtering conditions from candidate retrieval results associated with the multimodal query data, and to generate response content contained in the target answer based on the target retrieval results, the preset filtering conditions are used to filter the candidate retrieval results according to the ranking results of the correlation degree of the candidate retrieval results.

[0147] It should be noted here that the above-mentioned acquisition module 1101 and return module 1102 correspond to steps S61 to S62 in the above embodiments. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in memory and processed by one or more processors. The above-mentioned modules may also be part of the device and run in the server 10 provided in the above embodiments.

[0148] FIG12 is a structural block diagram of another data processing device according to an embodiment of the present application. As shown in FIG12, the device includes:

[0149] an acquisition module 1201, used to acquire the currently input data processing dialogue request, wherein the request data carried in the data processing dialogue request includes: multimodal query data;

[0150] a return module 1202, used to respond to the data processing dialogue request and return a data processing dialogue reply, wherein the data processing dialogue request includes: multimodal query data;The information carried in the processed dialogue response includes: the target answer, which is obtained by using a target retrieval generation model to perform knowledge retrieval and information generation on the multimodal query data. The target retrieval generation model is used to obtain target retrieval results that meet preset filtering conditions from the candidate retrieval results associated with the multimodal query data, and to generate response content contained in the target answer based on the target retrieval results. The preset filtering conditions are used to filter the candidate retrieval results according to the degree of relevance of the candidate retrieval results;

[0151] The display module 1203 is used to display the target answer in the graphical user interface.

[0152] It should be noted here that the above-mentioned acquisition module 1201, return module 1202 and display module 1203 correspond to steps S71 to S73 in the above embodiments. The three modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also be part of the device and can run in the server 10 provided in the above embodiments. Specification page 16 / 19 20 CN 122220450 A

[0153] Figure 13 is a structural block diagram of a model training device according to an embodiment of the present application. As shown in Figure 13, the device includes:

[0154] an acquisition module 1302, used to acquire multimodal training data;

[0155] an adjustment module 1302, used to adjust the pre-trained model using the target training strategy and the multimodal training data to obtain a target retrieval generation model, wherein a cross-modal attention mechanism is introduced in the target training strategy;

[0156] a deployment module 1303, used to deploy the target retrieval generation model to an execution device, wherein the target retrieval generation model is called by the execution device to execute the data processing method in the embodiment of the present application.

[0157] It should be noted here that the above acquisition module 1301, adjustment module 1302 and deployment module 1303 correspond to steps S81 to S83 in the above embodiments. The three modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in memory and processed by one or more processors. The above-mentioned modules may also be part of a device and run in the server 10 provided in the above embodiments.

[0158] It should be noted that the preferred embodiments involved in the above embodiments of this application are the same as the solutions, application scenarios, and implementation processes provided in the above embodiments, but are not limited to the solutions provided in the above embodiments.

[0159] The embodiments of this application can provide a data processing system, including: a client for sending multimodal queries.Data; server, connected to the client, used to perform knowledge retrieval and information generation on multimodal query data using a target retrieval generation model to obtain target answers, wherein the target retrieval generation model is used to obtain target retrieval results that meet preset filtering conditions from candidate retrieval results associated with multimodal query data, and to generate response content contained in the target answer based on the target retrieval results, and the preset filtering conditions are used to filter candidate retrieval results according to the degree of relevance of the candidate retrieval results; the client is also used to output the target answer.

[0160] Embodiments of this application can provide a computing device. Figure 14 is a structural block diagram of a computing device according to an embodiment of this application. As shown in Figure 14, the computing device may include: one or more (only one is shown in the figure) processors 142, memory 144, storage controller, and peripheral interfaces.

[0161] The above-mentioned computing device can be understood as an integrated intelligent terminal, including but not limited to servers, desktop computers, PCs (Personal Computers), all-in-one models, etc., and the computing device may be pre-installed with the models described in the above embodiments of this application.

[0162] Specifically, the computing device can pre-install various types of models, including but not limited to models in fields such as natural language processing, visual processing, speech processing, code processing, and multimodal task processing, thereby providing diverse model selection. In different product forms, the computing device can support one or more model usage methods, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference, and application. In some product forms, the computing device also supports model management, including but not limited to multi-type model management (supporting the management of various types of models such as discriminative and generative models), model version control (supporting the control of different model versions), and model evaluation (evaluating the performance and effectiveness of models based on model evaluation tools). In other product forms, the computing device can also create applications based on models, providing API calling capabilities, allowing models to be called into created applications through API interfaces, and providing application management tools to manage and monitor applications.

[0163] Further, the computing device may also include data management (supporting the creation and management of model tuning datasets), a training center (providing rich training resources to help users learn and master AI technology), and basic control capabilities (providing enterprise-level basic control capabilities to ensure the security and efficient operation of the system). Through the above functions, a comprehensive and integrated AI development, training, deployment, and application device is provided. Specification 17 / 19 pages 21 CN 122220450 A

[0164] Wherein, the memory can be used to store software programs and modules, such as the methods and apparatus in the embodiments of this application.The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of the above networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0165] The processor can call the executable program stored in the memory through a transmission device to execute the method described in any one of the above embodiments.

[0166] Embodiments of this application can provide an electronic device. FIG15 is a structural block diagram of an electronic device according to an embodiment of this application. As shown in FIG15, the electronic device may include: an input / output device 152; a memory 154 and a processor 156, wherein the processor 156 is connected to the input / output device 152 and the memory 154 via a bus 158.

[0167] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and devices in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, to implement the methods in the above embodiments. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of the above networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0168] The processor can call the executable program stored in the memory through a transmission device to execute the method described in any one of the above embodiments.

[0169] Those skilled in the art will understand that the structure shown in the figure is only illustrative, and the computing device may also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, and a mobile Internet device (MID), PAD, and other terminal devices. The figure does not limit the structure of the above computing device. For example, computing device 100 may also include more or fewer components (such as network interfaces, display devices, etc.) than shown in the figure, or have a different configuration than shown in the figure.

[0170] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device, and the program can be stored in a computer-readable storage medium.In this embodiment, the storage medium may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0171] Embodiments of this application also provide a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium may be used to store program code executed by the method provided in the above embodiments.

[0172] Optionally, in this embodiment, the storage medium may be located in a computing device.

[0173] Optionally, in this embodiment, the computer-readable storage medium is configured to store an executable program, which, when the executable program is running, controls the device where the computer-readable storage medium is located to execute the method described in any one of the above embodiments.

[0174] Embodiments of this application also provide a computer program product. Optionally, in this embodiment, the computer program product may include a computer program, which, when executed by a processor, implements the method provided in the above embodiments.

[0175] Embodiments of this application also provide a computer program product. Optionally, the above-mentioned computer program product may include a non-volatile computer-readable storage medium, as described on pages 18 / 19 of CN 122220450 A. The non-volatile computer-readable storage medium can be used to store a computer program, which, when executed by a processor, implements the method provided in the above embodiments.

[0176] The embodiments of this application also provide a computer program. Optionally, in this embodiment, the above-mentioned computer program, when executed by a processor, implements the method provided in the above embodiments.

[0177] In the above embodiments of this application, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, refer to the relevant descriptions of other embodiments.

[0178] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical functional division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be an indirect coupling or communication connection through some interfaces, units or modules, and can be electrical or other forms.

[0179] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0180] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0181] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in the various embodiments of this application. The aforementioned storage medium includes: USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk, and other media that can store program code.

[0182] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application. Instruction Manual 19 / 19 Page 23 CN 122220450 A Figure 1 Figure 2 Instruction Manual Appendix 1 / 7 Page 24 CN 122220450 A Figure 3 Instruction Manual Appendix 2 / 7 Page 25 CN 122220450 A Figure 4 Figure 5 Instruction Manual Appendix 3 / 7 Page 26 CN 122220450 A Figure 6 Figure 7 Instruction Manual Appendix 4 / 7 Page 27 CN 122220450 A Figure 8 Figure 9 Figure 10 Figure 11 Instruction Manual Appendix 5 / 7 Page 28 CN 122220450 A Figure 12 Figure 13 Figure 14 Instruction Manual Appendix 6 / 7 Page 29 CN 122220450 A Figure 15 Instruction Manual Appendix 7 / 7 Page 30 CN 122220450 A Abstract The application discloses a data processing method, a model training method, an electronic device and a storage medium, and relates to thetechnical field of large models and computer technology. The method comprises the following steps: acquiring multi-modal query data; performing knowledge retrieval and information generation on the multi-modal query data by using a target retrieval generation model to obtain a target answer; the target retrieval generation model is used to acquire a target retrieval result meeting a preset screening condition from candidate retrieval results associated with the multi-modal query data, and generate response content contained in the target answer based on the target retrieval result; and the preset screening condition is used to screen the candidate retrieval results according to the associated degree of the candidate retrieval results. The application solves the technical problems of low retrieval generation efficiency and poor accuracy in the related art when the multi-modal query data is retrieved.

Claims

1. A data processing method, characterized in that, include: Retrieve multimodal query data; A target retrieval and generation model is used to perform knowledge retrieval and information generation on the multimodal query data to obtain the target answer; The target retrieval generation model is used to obtain target retrieval results that meet preset filtering conditions from the candidate retrieval results associated with the multimodal query data, and to generate response content contained in the target answer based on the target retrieval results. The preset filtering conditions are used to filter the candidate retrieval results according to the ranking results of the relevance of the candidate retrieval results.

2. The data processing method according to claim 1, characterized in that, The target retrieval and generation model includes a multimodal recall sub-model, a multimodal ranking sub-model, and a multimodal generation sub-model. Using this model, knowledge retrieval and information generation are performed on the multimodal query data to obtain the target answer, including: The multimodal recall sub-model is used to recall the multimodal query data to obtain the candidate retrieval results; The candidate retrieval results are sorted using the multimodal ranking sub-model to obtain the ranking result; The multimodal generation sub-model is used to process the ranking results to generate response content, thereby obtaining the target answer.

3. The data processing method according to claim 2, characterized in that, The multimodal recall sub-model includes a feature encoding module, a feature mapping module, and a feature matching module. The multimodal recall sub-model is used to recall the multimodal query data to obtain the candidate retrieval results, including: The feature encoding module is used to perform feature encoding processing on the multimodal query data to obtain a multimodal feature vector; The feature mapping module is used to perform feature mapping processing on the multimodal feature vectors to obtain a multimodal mapping result, wherein the multimodal mapping result is used to represent the multimodal representation obtained by mapping feature vectors of different modalities to a shared multimodal space; The feature matching module is used to recall the candidate retrieval results that match the multimodal mapping results from the target multimodal dataset.

4. The data processing method according to claim 3, characterized in that, The feature matching module is used to recall candidate retrieval results from the target multimodal dataset that match the multimodal mapping results, including: In the feature matching module, candidate retrieval results that match the multimodal mapping results are retrieved from the target multimodal dataset based on a preset index structure.

5. The data processing method according to claim 4, characterized in that, The data processing method further includes: Obtain the initial multimodal dataset; The initial multimodal dataset is cleaned and preprocessed to obtain an intermediate multimodal dataset; The intermediate multimodal dataset is subjected to data alignment and data annotation to obtain the target multimodal dataset; The target multimodal dataset is feature-encoded to construct the preset index structure.

6. The data processing method according to claim 4, characterized in that, Retrieving candidate search results that match the multimodal mapping results from the target multimodal dataset based on the preset index structure includes: Based on the preset index structure, the initial search results are retrieved from the target multimodal dataset; The similarity between the multimodal mapping result and the initial retrieval result is calculated to obtain the calculation result; Based on the calculation results, candidate search results that match the multimodal mapping results are selected from the initial search results.

7. The data processing method according to claim 4, characterized in that, The data processing method further includes: Based on a preset prompt text format, some candidate search results in the sorting results are concatenated to obtain the target prompt text.

8. The data processing method according to claim 7, characterized in that, The multimodal generation sub-model is used to process the ranking results into response content, resulting in the target answer, which includes: The target prompt text is input into the multimodal generation sub-model, so that the multimodal generation sub-model performs response content generation processing based on the multimodal splicing data recorded in the target prompt text to obtain the target answer.

9. A data processing method, characterized in that, include: Acquire multimodal product query data, wherein the multimodal product query data is used to provide descriptive information of the products to be selected; A target retrieval and generation model is used to perform product knowledge retrieval and product information generation on the multimodal product query data to obtain the target product answer; The target retrieval generation model is used to obtain target product retrieval results that meet preset filtering conditions from the candidate product retrieval results associated with the multimodal product query data, and to generate response content contained in the target product answer based on the target product retrieval results. The preset filtering conditions are used to filter the candidate product retrieval results according to the ranking result of the relevance of the candidate product retrieval results.

10. A data processing method, characterized in that, include: A data processing request is obtained through a first application programming interface, wherein the request data carried in the data processing request includes: multimodal query data; A data processing response is returned through a second application programming interface, wherein the response data carried in the data processing response includes: a target answer, which is obtained by performing knowledge retrieval and information generation on the multimodal query data using a target retrieval generation model. The target retrieval generation model is used to obtain target retrieval results that meet preset filtering conditions from the candidate retrieval results associated with the multimodal query data, and to generate response content contained in the target answer based on the target retrieval results. The preset filtering conditions are used to filter the candidate retrieval results according to the ranking results of the relevance of the candidate retrieval results.

11. A data processing method, characterized in that, include: Obtain the currently input data processing dialogue request, wherein the request data carried in the data processing dialogue request includes: multimodal query data; In response to the data processing dialogue request, a data processing dialogue reply is returned, wherein the information carried in the data processing dialogue reply includes: a target answer, which is obtained by performing knowledge retrieval and information generation on the multimodal query data using a target retrieval generation model. The target retrieval generation model is used to obtain target retrieval results that meet preset filtering conditions from the candidate retrieval results associated with the multimodal query data, and to generate response content contained in the target answer based on the target retrieval results. The preset filtering conditions are used to filter the candidate retrieval results according to the ranking results of the relevance of the candidate retrieval results. The target answer is displayed within the graphical user interface.

12. A model training method, characterized in that, include: Acquire multimodal training data; The pre-trained model is adjusted using the target training strategy and the multimodal training data to obtain the target retrieval generation model, wherein a cross-modal attention mechanism is introduced into the target training strategy; The target retrieval generation model is deployed to an execution device, wherein the target retrieval generation model is invoked by the execution device to execute the data processing method according to any one of claims 1 to 11.

13. A data processing system, characterized in that, include: The client is used to send multimodal query data; The server, connected to the client, is used to perform knowledge retrieval and information generation on the multimodal query data using a target retrieval generation model to obtain a target answer. The target retrieval generation model is used to obtain target retrieval results that meet preset filtering conditions from the candidate retrieval results associated with the multimodal query data, and to generate response content contained in the target answer based on the target retrieval results. The preset filtering conditions are used to filter the candidate retrieval results according to the ranking results of the relevance of the candidate retrieval results. The client is also used to output the target answer.

14. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the data processing method according to any one of claims 1 to 11 or the model training method according to claim 12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the computer-readable storage medium is located to perform the data processing method of any one of claims 1 to 11 or the model training method of claim 12.

16. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the data processing method of any one of claims 1 to 11 or the model training method of claim 12.

Images