Data processing method, data processing device, electronic equipment and storage medium

Abstract

The invention provides a data processing method, a data processing device, electronic equipment and a storage medium, the method can be applied to electronic equipment such as a mobile phone and a tablet personal computer, and the method comprises the following steps: processing a target problem through a target large model to obtain initial reply content comprising at least one target label; and then replacing at least one target label in the initial reply content with the corresponding multimedia data to obtain and output target reply content corresponding to the target question. The label can be generated when the initial reply content is generated, direct insertion of complex multimedia data is avoided, the processing process of the model is simplified, the generated target reply content contains at least one piece of multimedia data through label replacement, the method can adapt to the scene of outputting single multimedia data, and the user experience is improved. The method can adapt to scenes outputting multiple pieces of multimedia data, meets multimedia requirements in simple scenes and complex scenes, and realizes efficient and accurate multi-modal data mixed question answering in an RAG framework.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a data processing method, a data processing apparatus, an electronic device, and a storage medium. Background Technology

[0002] With the rapid development of artificial intelligence and natural language processing (NLP) technologies, intelligent question answering systems have been widely used in various fields. Intelligent question answering systems retrieve relevant fragments from knowledge bases constructed in various ways based on retrieval augmented generation (RAG) in response to user questions, and then use large models to perform reasoning and summarization.

[0003] For example, most existing intelligent question-answering systems provide answers in plain text format, which makes it difficult to effectively combine multimedia data. At the same time, some intelligent question-answering systems only support recalling the top-related reference images, which is difficult to meet the needs of multi-step operations or complex information display, thus limiting the richness of the answers. As a result, in questions and answers that require detailed operation steps, after receiving the answer from the intelligent question-answering system, users still need to search for the relevant operation steps in the original document that records the detailed operation steps, which reduces the user experience and information acquisition efficiency.

[0004] Therefore, for intelligent question-answering systems, how to achieve efficient and accurate question-answering with mixed text and images within the RAG framework has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides a data processing method, a data processing apparatus, an electronic device, and a storage medium, which enables efficient and accurate question-and-answer functionality with mixed text and graphics within the RAG framework.

[0006] Firstly, a data processing method is provided, which is applied to an electronic device, comprising: obtaining initial response content corresponding to a target question through a target large model, wherein the initial response content includes at least one target tag, the target tag being used to replace the corresponding multimedia data; replacing at least one target tag in the initial response content with the corresponding multimedia data to obtain the target response content corresponding to the target question; and outputting the target response content.

[0007] In the above technical solution, the target question is processed through a target large model to obtain initial response content including at least one target tag. Then, at least one target tag in the initial response content is replaced with corresponding multimedia data to obtain and output the target response content corresponding to the target question. This approach generates tags during the initial response content generation, avoiding the direct insertion of complex multimedia data and simplifying the model's processing. Furthermore, by replacing tags, the generated target response content includes at least one multimedia data point, adapting to scenarios involving outputting single or multiple multimedia data points. It can simultaneously meet the multimedia needs of both simple and complex scenarios, and is particularly suitable for step-by-step question-answering scenarios. It can achieve efficient and accurate multimodal data mixed-format question answering within the RAG framework.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, replacing at least one target tag in the initial response content with the corresponding multimedia data to obtain the target response content corresponding to the target question includes: querying the target mapping relationship related to the target tag, the target mapping relationship being used to characterize the correspondence between the target tag and the multimedia data; and based on the target mapping relationship, replacing at least one target tag in the initial response content with the corresponding multimedia data to obtain the target response content corresponding to the target question.

[0009] In the above technical solution, at least one target tag in the initial response content is replaced with the corresponding multimedia data by querying the target mapping relationship, so as to obtain the target response content corresponding to the target question. The target mapping relationship can ensure the accurate correspondence between the target tag and the multimedia data, further improving the reliability of the target response content.

[0010] Combining the first aspect and the above implementation method, the initial response content corresponding to the target question is obtained through the target large model, including: in response to obtaining the target question, obtaining at least one search enhancement content corresponding to the target question, the search enhancement content including at least one target tag; inputting the target question and at least one search enhancement content into the target large model, processing the target question and at least one search enhancement content through the target large model, and outputting the initial response content.

[0011] In the above technical solution, the acquired search enhancement content and target question are processed by the target large model to output the initial response content. The generation range of the initial response content can be constrained by the search enhancement content, which improves the accuracy of the initial response content and reduces the possibility of hallucination.

[0012] Combining the first aspect and the above implementation method, the target question and at least one search enhancement content are processed through the target big model to output the initial response content, including: obtaining a prompt text template, which is used to prompt that the target tags in the search enhancement content are retained when generating the response content; filling the prompt text template with the target question and search enhancement content and processing it to obtain the target prompt text; and processing the target prompt text through the target big model to output the initial response content.

[0013] In the above technical solution, the target prompt text is obtained by filling the target question and search enhancement content into the acquired prompt text template. The target prompt text is then processed by the target big model to output the initial response content. The output format of the target big model is dynamically controlled by the prompt text template, so that the target big model can recognize and maintain predefined tags, thereby generating the initial response content containing the target tags, which further ensures the accurate replacement of multimedia data and tags.

[0014] Combining the first aspect and the above implementation method, in response to obtaining the target question, obtaining at least one search enhancement content corresponding to the target question includes: in response to obtaining the target question, determining the query vector corresponding to the target question; matching the query vector with multiple candidate vectors in the enhancement content database to obtain at least one target vector, wherein the target vector is a candidate vector that successfully matches the query vector; and determining the search enhancement content corresponding to at least one target vector as at least one search enhancement content corresponding to the target question.

[0015] In the above technical solution, the target vector is obtained by matching the query vector corresponding to the target question with multiple candidate vectors in the enhanced content database. The enhanced content corresponding to the target vector is used as at least one enhanced content corresponding to the target question. This can accurately locate the enhanced content related to the target question and further improve the accuracy of the initial response content.

[0016] Combining the first aspect and the above implementation methods, the target response content is output, including: determining the target display format of the target response content; adjusting the target response content based on the target display format to output the adjusted target response content.

[0017] In the above technical solution, by determining the target display format, the output method of the target response content is dynamically adjusted, which improves the flexibility of the target response content output and enables the output of target response content according to user preferences, thereby improving the user experience.

[0018] In conjunction with the first aspect and the above implementation method, the method further includes: acquiring at least one multimodal data, and extracting text data and multimedia data from the multimodal data, wherein the text data is used to explain the multimedia data; determining the tags corresponding to the multimedia data, and establishing a mapping relationship between the multimedia data and the tags corresponding to the multimedia data; and storing the mapping relationship between the multimedia data and the tags corresponding to the multimedia data.

[0019] In the above technical solution, by extracting text data and multimedia data from the acquired multimodal data, determining and establishing a mapping relationship between the multimedia data and its corresponding tags, and storing the mapping relationship, the multimedia data in the multimodal data can be replaced with predefined tags. This simplifies the input of the target large model, reduces the difficulty of the target large model in processing input data, and improves the accuracy and consistency of the target large model's answers. At the same time, the target mapping relationship can be stored in the tag mapping management module, which supports dynamically adding, updating, and deleting the mapping relationship between tags and multimedia data, ensuring the flexibility and maintainability of the mapping relationship.

[0020] Combining the first aspect and the above implementation method, the tags corresponding to the multimedia data are determined, including: extracting the semantic information corresponding to the multimedia data from the text data; adding the semantic information corresponding to the multimedia data to the tags corresponding to the multimedia data, or adding the semantic information corresponding to the multimedia data to the context of the tags corresponding to the multimedia data.

[0021] In the above technical solution, by extracting semantic information corresponding to multimedia data from text data and adding the semantic information into the label or the context of the label, the target large model can more accurately understand the meaning of the input label, thereby improving the accuracy of the response content output by the target large model.

[0022] In conjunction with the first aspect and the above implementation method, the method further includes: replacing multimedia data based on the tags corresponding to the multimedia data to obtain preprocessed multimedia data; dividing the preprocessed multimedia data and text data into blocks to obtain multiple candidate data blocks, wherein the candidate data blocks are a portion of the data in the multimodal data; and storing the candidate data blocks in the enhanced content database.

[0023] In the above technical solution, the preprocessed multimedia data and text data obtained by replacing the multimedia data are processed into blocks to store the candidate data blocks in the enhanced content database. This allows for the distributed storage of data blocks containing tags, facilitating the retrieval of enhanced content from the enhanced content database. Furthermore, the enhanced content database can be continuously expanded according to actual conditions, further improving the accuracy of the target large model in processing data.

[0024] Combining the first aspect and the above implementation method, the preprocessed multimedia data and text data are segmented to obtain multiple candidate data blocks, including: according to the order of the preprocessed multimedia data and text data in the multimodal data, and based on the semantic similarity between the preprocessed multimedia data and text data, the preprocessed multimedia data and text data are segmented to obtain multiple candidate data blocks.

[0025] In the above technical solution, the preprocessed multimedia data and text data are divided into blocks based on their order and semantic similarity in the multimodal data, resulting in multiple candidate data blocks. This allows similar content in the multimodal data to be divided into blocks in order, facilitating subsequent retrieval and matching processes.

[0026] Combining the first aspect and the above implementation method, storing candidate data blocks in the enhanced content database includes: encoding candidate data blocks into multiple candidate vectors, and storing the candidate vectors in the enhanced content database.

[0027] In the above technical solution, by storing multiple candidate vectors encoded by candidate data blocks into the enhanced content database, vector matching can be performed directly when using the enhanced content database to match and retrieve enhanced content, reducing the time for outputting the target response content and improving the processing efficiency of the data processing method.

[0028] In a second aspect, a data processing apparatus is provided, which is applied to an electronic device and includes: The acquisition module is used to acquire the initial response content corresponding to the target question through the target large model. The initial response content includes at least one target tag, which is used to replace the corresponding multimedia data. The replacement module is used to replace at least one target tag in the initial response content with the corresponding multimedia data to obtain the target response content corresponding to the target question. The output module is used to output the target response content.

[0029] In conjunction with the second aspect and the above implementation methods, in some possible implementation methods, the replacement module is also used to query the target mapping relationship related to the target tag, the target mapping relationship is used to characterize the correspondence between the target tag and the multimedia data; based on the target mapping relationship, at least one target tag in the initial response content is replaced with the corresponding multimedia data to obtain the target response content corresponding to the target question.

[0030] In conjunction with the second aspect and the above implementation methods, in some possible implementation methods, the acquisition module is further configured to, in response to acquiring the target question, acquire at least one search enhancement content corresponding to the target question, the search enhancement content including at least one target tag; input the target question and at least one search enhancement content into the target large model, process the target question and at least one search enhancement content through the target large model, and output the initial response content.

[0031] In conjunction with the second aspect and the above implementation methods, in some possible implementation methods, the acquisition module is also used to acquire a prompt text template, which is used to prompt the retention of target tags in the search enhancement content when generating the response content; the target question and search enhancement content are filled into the prompt text template for processing to obtain the target prompt text; the target prompt text is processed through the target big model to output the initial response content.

[0032] In conjunction with the second aspect and the above implementation methods, in some possible implementation methods, the acquisition module is further configured to, in response to acquiring the target question, determine the query vector corresponding to the target question; match the query vector with multiple candidate vectors in the enhanced content database to obtain at least one target vector, wherein the target vector is a candidate vector that successfully matches the query vector; and determine the retrieval enhanced content corresponding to at least one target vector as at least one retrieval enhanced content corresponding to the target question.

[0033] In conjunction with the second aspect and the above implementation methods, in some possible implementation methods, the output module is also used to determine the target display format of the target response content; adjust the target response content based on the target display format, and output the adjusted target response content.

[0034] In conjunction with the second aspect and the above implementation methods, in some possible implementation methods, the acquisition module is further configured to acquire at least one multimodal data and extract text data and multimedia data from the multimodal data, wherein the text data is used to explain the multimedia data; the device further includes a determination module configured to determine the tags corresponding to the multimedia data and establish a mapping relationship between the multimedia data and the tags corresponding to the multimedia data; the device further includes a storage module configured to store the mapping relationship between the multimedia data and the tags corresponding to the multimedia data.

[0035] In conjunction with the second aspect and the above implementation methods, in some possible implementation methods, the determining module is also used to extract semantic information corresponding to multimedia data from text data; add the semantic information corresponding to multimedia data to the tag corresponding to multimedia data, or add the semantic information corresponding to multimedia data to the context of the tag corresponding to multimedia data.

[0036] In conjunction with the second aspect and the above implementation methods, in some possible implementation methods, the replacement module is further used to perform replacement processing on the multimedia data based on the tags corresponding to the multimedia data to obtain preprocessed multimedia data; the device also includes a block segmentation module, which is used to perform block processing on the preprocessed multimedia data and text data to obtain multiple candidate data blocks, wherein the candidate data blocks are a portion of the data in the multimodal data; the storage module is further used to store the candidate data blocks in the enhanced content database.

[0037] In combination with the second aspect and the above implementation methods, in some possible implementation methods, the segmentation module is also used to segment the preprocessed multimedia data and text data according to the order of the preprocessed multimedia data and text data in the multimodal data, based on the semantic similarity between the preprocessed multimedia data and text data, to obtain multiple candidate data blocks.

[0038] In conjunction with the second aspect and the above implementation methods, in some possible implementation methods, the storage module is also used to store candidate data blocks into the enhanced content database, including: encoding the candidate data blocks into multiple candidate vectors and storing the candidate vectors into the enhanced content database.

[0039] Thirdly, an electronic device is provided, including a memory and a processor, wherein the memory is used to store executable program code; and the processor is used to call and run the executable program code from the memory, causing the electronic device to perform the data processing method in the first aspect or any possible implementation thereof.

[0040] Fourthly, a computer-readable storage medium is provided that stores computer program code, which, when executed on a computer, causes the computer to perform the data processing method described in the first aspect or any possible implementation thereof.

[0041] Fifthly, a computer program product is provided, comprising: computer program code, which, when run on a computer, causes the computer to perform the data processing method described in the first aspect or any possible implementation thereof.

[0042] The method provided in this application, through the above-described approach, processes the target question using a target large model to obtain initial response content including at least one target tag. Then, it replaces at least one target tag in the initial response content with corresponding multimedia data to obtain and output the target response content corresponding to the target question. This method generates tags during the initial response content generation, avoiding the direct insertion of complex multimedia data and simplifying the model's processing. Furthermore, by replacing tags, the generated target response content includes at least one multimedia data point, adapting to scenarios involving outputting single or multiple multimedia data points. It simultaneously meets the multimedia needs of both simple and complex scenarios, and is particularly suitable for step-by-step question-and-answer scenarios. It enables efficient and accurate multimodal data mixed-format question-and-answer within the RAG framework. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating a process for processing multimodal data, provided in an embodiment of this application. Figure 2 This is a flowchart illustrating the workflow of a text-based question-and-answer system provided in an embodiment of this application. Figure 3 This is a schematic diagram of an interface for outputting target response content provided in an embodiment of this application; Figure 4 This is an illustrative flowchart of a data processing method provided in an embodiment of this application. Figure 1 ; Figure 5 This is an illustrative flowchart of a data processing method provided in an embodiment of this application. Figure 2 ; Figure 6 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0044] The technical solutions in this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0045] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0046] Before introducing the solutions of the embodiments of this application, the technical terms that may be involved in the embodiments of this application will be explained first.

[0047] Natural Language Processing (NLP) is an interdisciplinary field combining computer science, artificial intelligence (AI), and linguistics. Its aim is to enable computers to understand, interpret, and generate human language. The core goal of NLP is to use algorithms and models to allow machines to process natural language, including text and speech, just like humans, and to achieve effective human-machine communication. Specifically, NLP mainly includes Natural Language Understanding (NLU) and Natural Language Generation (NLG). NLU aims to convert human language into a form that computers can process; NLG aims to convert computer-generated information into natural language for output or interaction.

[0048] Large Language Models (LLMs) are artificial intelligence models designed to understand and generate human language. They are trained on massive amounts of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, and more. LLMs are characterized by their massive scale, containing billions of parameters that help them learn complex patterns in language data. These models are typically based on deep learning architectures, such as transformers, which contributes to their impressive performance on various NLP tasks.

[0049] RAG: An NLP technique that combines the capabilities of information retrieval and generative models to generate more accurate and informative answers or text. This approach is particularly effective in handling complex question-answering systems and knowledge-enhanced dialogue systems.

[0050] A Uniform Resource Locator (URL) is used to identify the server and specific webpage location of an internet host. It is the name used by internet users to identify a host. This name is unique, hierarchical, and its literal meaning can represent the host's account, function, nature, region, or organization.

[0051] Before introducing the solutions of the embodiments of this application, we will first introduce the application scenarios of the embodiments of this application.

[0052] With the rapid development of artificial intelligence and NLP technologies, intelligent question answering systems have been widely used in various fields. Intelligent question answering systems retrieve relevant fragments from knowledge bases built in various ways based on RAG (Research Aggregate), and then use large models to perform reasoning and summarization in response to user questions.

[0053] For example, most existing intelligent question-answering systems provide answers in plain text format, which makes it difficult to effectively combine multimedia data. At the same time, some intelligent question-answering systems only support recalling the top-related reference images, which is difficult to meet the needs of multi-step operations or complex information display, thus limiting the richness of the answers. As a result, in questions and answers that require detailed operation steps, after receiving the answer from the intelligent question-answering system, users still need to search for the relevant operation steps in the original document that records the detailed operation steps, which reduces the user experience and information acquisition efficiency.

[0054] Therefore, for intelligent question-answering systems, how to achieve efficient and accurate question-answering with mixed text and images within the RAG framework has become an urgent problem to be solved.

[0055] The following is combined Figures 1 to 3 The data processing method provided in the embodiments of this application is described by way of example.

[0056] Figure 1 This is a flowchart of a process for processing multimodal data provided in an embodiment of this application.

[0057] For example, after acquiring the document (corresponding to multimodal data), document content parsing 11 is triggered to parse the document and extract text data and multimedia data from the multimodal data; tag generation 12 is triggered to generate tags corresponding to the multimedia data according to the type of multimedia data (e.g., image data, hyperlink data, video data, etc.); mapping relationship storage 13 is triggered to store the mapping relationship between the tags corresponding to the multimedia data and the multimedia data; tag replacement 14 is triggered to replace the multimedia data with the tags corresponding to the multimedia data to obtain preprocessed multimedia data; document content segmentation 15 is triggered to segment the preprocessed multimedia data and text data to obtain multiple candidate data blocks; vector extraction 16 is triggered to encode the multiple candidate data blocks to obtain multiple candidate vectors; and storage to vector database 17 (corresponding to the enhanced content database) is triggered to store the multiple candidate vectors to the vector database.

[0058] Figure 2 This is a flowchart of a text-based question-and-answer system provided in an embodiment of this application.

[0059] For example, user 21 triggers user query 22 to obtain a target question. After determining the query vector corresponding to the target question, document content retrieval 23 matches at least one target vector with multiple candidate vectors in the vector database 24. This target vector is a candidate vector that successfully matches the query vector and is encoded by candidate data blocks. The target vector is used as retrieval enhancement content. The target question and retrieval enhancement content are filled into the obtained prompt text template to obtain the target prompt text. The target prompt text is processed by the target big model 25 to output initial response content containing target tags. Further, by outputting the answer 27, the tag mapping management module 26 queries the target mapping relationship related to the target tags. At least one target tag in the initial response content is replaced with the corresponding multimedia data to obtain the target response content corresponding to the target question, and the target response content is output.

[0060] It should be noted that before conducting text and image Q&A, the obtained document 28 is processed by document preprocessing module 29 for document parsing and tag generation to obtain the mapping relationship between the tags corresponding to the multimedia data and the multimedia data. The mapping relationship between the tags corresponding to the multimedia data and the multimedia data is stored in tag mapping management module 26, so that the tag replacement mechanism can be triggered when conducting text and image Q&A.

[0061] Figure 3 This is a schematic diagram of an interface for outputting target response content provided in an embodiment of this application.

[0062] For example, in response to obtaining a target question—which inquires about specific operations for contract management in a smart mobile office platform software—the target response content 30 is output after processing by the data processing method provided in this application. This target response content 30 is in Rich Text Format (RTF) format, including text data and image data. The image data includes a first image 31 and a second image 32, and the text data is used to explain the first image 31 and the second image 32. Using the data processing method of this application, the generated image-text fusion response can be displayed on the user interface.

[0063] The above embodiments combined Figures 1 to 3 This paper introduces the execution logic and specific results of the data processing method from the perspectives of interaction and user interface. The following will combine... Figure 4 This section introduces the underlying implementation process upon which this method depends.

[0064] Figure 4 This is an illustrative flowchart of a data processing method provided in an embodiment of this application. Figure 1It should be understood that this method 400 can be applied to electronic devices such as mobile phones, tablets, and in-vehicle devices; the embodiments of this application do not limit the type of electronic device. For example, such as... Figure 4 As shown, the data processing method 400 includes the following steps.

[0065] 401. Using the target big model, obtain the initial response content corresponding to the target question. The initial response content includes at least one target tag, which is used to replace the corresponding multimedia data.

[0066] The target large-scale model refers to a machine learning model with a large number of parameters and a complex computational structure, capable of processing massive amounts of data and completing various complex tasks. In some embodiments, the target large-scale model is an LLM (Limited Learning Model). The target large-scale model can be integrated into an application for intelligent question answering. The target question refers to the data input into the target large-scale model. In some embodiments, the target question is retrieved in response to a user's query. The initial response content is the answer generated by the target large-scale model after understanding the target question. The initial response content includes at least one target tag. The target tag is information used to classify and label multimedia data. In some embodiments, the data type of the target tag is text; that is, this application can use text data to replace multimedia data in various forms. Multimedia data refers to an information expression method that combines multiple forms of information carriers and stores and transmits information through digital technology. In some embodiments, multimedia data may include, but is not limited to, at least one of the following: text data, image data, audio data, video data, URL data, etc.

[0067] 402. Replace at least one target tag in the initial response content with the corresponding multimedia data to obtain the target response content corresponding to the target question.

[0068] The target response content refers to the data obtained by replacing the target tags in the initial response content with multimedia data. In some embodiments, after obtaining the initial response content, at least one target tag in the initial response content is filtered, and then the at least one target tag is replaced with the corresponding multimedia data to obtain the target response content corresponding to the target question.

[0069] 403, output the target response content.

[0070] Since the target response content includes multimedia data corresponding to at least one target tag, the multimedia data that needs to be embedded in the target response content can be derived through a combination of text and images. Therefore, when outputting the target response content, multimodal data, including but not limited to text data, image data, URL data, and video data, can be output.

[0071] The method provided in this application processes a target question using a target large model to obtain initial response content including at least one target tag. Then, it replaces at least one target tag in the initial response content with corresponding multimedia data to obtain and output the target response content corresponding to the target question. This method generates tags during the initial response content generation, avoiding the direct insertion of complex multimedia data and simplifying the model's processing. Furthermore, by replacing tags, the generated target response content includes at least one multimedia data point, adapting to scenarios involving outputting single or multiple multimedia data points. It can simultaneously meet the multimedia needs of both simple and complex scenarios, and is particularly suitable for step-by-step question-and-answer scenarios. It can achieve efficient and accurate multimodal data mixed-format question-and-answer within the RAG framework.

[0072] It should be noted that steps 401-403 above are a simplified description of the data processing method provided in the embodiments of this application. The data processing method provided in the embodiments of this application will be described in more detail below with reference to some examples. See [link to relevant documentation]. Figure 5 , Figure 5 This is an illustrative flowchart of a data processing method provided in an embodiment of this application. Figure 2 .

[0073] It should be understood that this method 500 can be applied to electronic devices such as mobile phones, tablets, and in-vehicle devices; the embodiments of this application do not limit the type of electronic device. For example, such as... Figure 5 As shown, the data processing method 500 includes the following steps.

[0074] 501. Obtain at least one multimodal data and extract text data and multimedia data from the multimodal data, wherein the text data is used to interpret and explain the multimedia data.

[0075] Multimodal data is a logical unit in data processing and transmission, used to represent independent data blocks or information fragments. In some embodiments, the number of multimodal data is at least one. The form of multimodal data may include, but is not limited to, at least one of the following: documents, videos, URLs, audio, etc. In some embodiments, the method of acquiring multimodal data may include, but is not limited to, at least one of the following: acquiring multimodal data from documents integrated within an application, acquiring multimodal data from an open-source online platform, etc. Text data is data that interprets and explains multimedia data within multimodal data. Multimedia data refers to an information expression method that combines multiple forms of information carriers and stores and transmits information through digital technology. In some embodiments, multimedia data may include, but is not limited to, at least one of the following: text data, image data, audio data, video data, URL data, etc.

[0076] It's important to note that when extracting data from multimodal data, if only multimedia data is extracted, the initial response content generated based on the target model will only include the target tags corresponding to the multimedia data. Furthermore, when generating the target response content, this content will also consist solely of multimedia data, making it difficult for users to understand the target response content based solely on the multimedia data. Therefore, when extracting data from multimodal data, it's necessary to extract both text and multimedia data. The text data is used to explain and clarify the multimedia data. Consequently, when generating the target response content, it will include both multimedia and text data, facilitating user comprehension.

[0077] In some embodiments, the format of multimodal data may include, but is not limited to, one of the following: Hyper Text Markup Language (HTML), Markdown, Portable Document Format (PDF), etc. HTML is a markup language that includes a series of tags that unify the format of documents on the web, connecting disparate internet resources into a logical whole. Markdown is a lightweight markup language that can be used to add formatting elements to plain text documents. PDF is used to share and view documents in a way that is independent of applications, hardware, and operating systems; PDF documents can contain links and buttons, form fields, audio, video, and transaction logic.

[0078] In some embodiments, before extracting text data and multimedia data from multimodal data, it is necessary to process multimodal data in different formats into multimodal data in a target format, which may be Markdown or HTML.

[0079] In some embodiments, before extracting text and multimedia data from multimodal data, the multimodal data is preprocessed. This preprocessing may include, but is not limited to, text cleaning, text normalization, semantic enhancement, and audio-to-text transcription. Text cleaning is a key step in NLP, aiming to preprocess text data to improve the accuracy and efficiency of subsequent analysis and modeling. Text cleaning can be any suitable operation, such as unifying text encoding, removing useless characters, merging consecutive spaces / newlines, etc. Text normalization is an important step in NLP, aiming to convert the raw text into a uniform format to improve the accuracy and efficiency of subsequent processing. Text normalization can be any suitable operation, such as case unification, abbreviation expansion, etc. Semantic enhancement is a technique that enhances the semantic expressiveness of text through a series of technical means, making it more accurate and richer. Semantic enhancement can be any suitable operation, such as spell correction, synonym replacement, named entity recognition, etc. By preprocessing multimodal data, the efficiency and accuracy of extracting text and multimedia data from multimodal data can be improved.

[0080] 502. Determine the tags corresponding to the multimedia data and establish a mapping relationship between the multimedia data and the tags corresponding to the multimedia data.

[0081] The tags corresponding to multimedia data are used to classify and label the multimedia data. In some embodiments, the tags corresponding to multimedia data are unique, and different multimedia data have different tags. In some embodiments, multimedia data is divided into types to determine the tags corresponding to different types of multimedia data. The mapping relationship between multimedia data and its corresponding tags is used to characterize the correspondence between tags and multimedia data. In some embodiments, each piece of multimedia data has one mapping relationship, and the mapping relationship is unique; different multimedia data have different mapping relationships with their corresponding tags.

[0082] For example, multimedia data types include image data, URL data, and video data. A unique label is generated for each detected image data point in the multimodal data, for example,<IMG_001> ,<IMG_002> Etc. Generate unique labels for each detected ULR data in the multimodal data, for example,<LINK_001> ,<LINK_002> Etc. Generate unique labels for each detected video data point in the multimodal data, for example,<WMV_001> ,<WMV_002> wait.

[0083] In one possible implementation, semantic information corresponding to the multimedia data is extracted from the text data. This semantic information is then added to the tags corresponding to the multimedia data, or the semantic information is added to the context of the tags corresponding to the multimedia data.

[0084] Semantic information refers to the meaning and logical connotation expressed by data or symbols; it is meaningful information that can eliminate uncertainty. In some embodiments, the semantic information corresponding to multimedia data is located in the form of text within the multimodal data. After extracting text data and multimedia data from the multimodal data, the semantic information corresponding to the multimedia data is extracted from the text data.

[0085] In some embodiments, when extracting semantic information corresponding to multimedia data from text data, the text description associated with the multimedia data is first identified from the text data, and then keywords, entities, relationships, and other content are extracted from the associated text description. Then, semantic information corresponding to the multimedia data is generated based on the keywords, entities, relationships, and other content.

[0086] It should be noted that when the target big model obtains initial response content including at least one target label based on the target question, since the expression form of the target label is different from that of text data, the target big model needs to understand the semantic information of the target label. When generating labels by obtaining the semantic information corresponding to the multimedia data, the semantic information corresponding to the multimedia data can be directly added to the label, or the semantic information corresponding to the multimedia data can be added to the context of the label corresponding to the multimedia data. This allows the target big model to understand the meaning of the label when processing the target question and obtain initial response content including at least one target label.

[0087] In this implementation, by extracting semantic information corresponding to multimedia data from text data and adding the semantic information to the tag or the tag's context, the target large model can more accurately understand the meaning of the input tag, thereby improving the accuracy of the response content output by the target large model.

[0088] 503 stores the mapping relationship between multimedia data and the corresponding tags of multimedia data.

[0089] Specifically, a tag mapping management module is established, and the mapping relationship between multimedia data and its corresponding tags is stored in this module. Multimedia data refers to an information expression method that combines multiple forms of information carriers and stores and transmits information through digital technology. In some embodiments, multimedia data may include, but is not limited to, at least one of the following: text data, image data, audio data, video data, URL data, etc.

[0090] In some embodiments, the tag mapping management module includes a mapping table, which stores the mapping relationship between multimedia data and the tags corresponding to the multimedia data, and manages it using a database (such as SQLite, PostgreSQL) or an in-memory data structure (such as a hash table).

[0091] 504. Based on the tags corresponding to the multimedia data, the multimedia data is replaced to obtain preprocessed multimedia data.

[0092] The tags corresponding to the multimedia data are information used to classify and label the multimedia data. The preprocessed multimedia data is multimedia data that has undergone tag replacement. In some embodiments, the tags corresponding to the multimedia data are unique, and different multimedia data have different tags. In some embodiments, the multimedia data is divided according to type to determine the tags corresponding to the multimedia data according to different types of multimedia data.

[0093] For example, replacing the image data in the multimedia data with the corresponding...<IMG_xxx> Tags that replace hyperlinks in multimedia data with corresponding tags.<LINK_xxx> Tags replace video data in multimedia data with corresponding tags.<WMV_xxx> Label.

[0094] 505. The preprocessed multimedia data and text data are divided into blocks to obtain multiple candidate data blocks, which are portions of the multimodal data.

[0095] Candidate data blocks are data obtained by segmenting preprocessed multimedia and text data, and represent a portion of the multimodal data. In some embodiments, there are multiple candidate data blocks.

[0096] In one possible implementation, the preprocessed multimedia data and text data are divided into blocks according to their order in the multimodal data and based on the semantic similarity between them, to obtain multiple candidate data blocks.

[0097] Semantic similarity refers to the degree of semantic closeness between two linguistic units (e.g., words, phrases, sentences, documents). Based on the semantic similarity between preprocessed multimedia data and text data, the preprocessed multimedia data and text data can be segmented into blocks, allowing highly relevant content to be grouped into the same candidate data block. For example, for multimodal data representing application operation steps, the preprocessed multimedia data and text data for the same operation step can be grouped into the same candidate data block based on the semantic similarity between them.

[0098] In some embodiments, the preprocessed multimedia data and text data are divided into blocks according to their order in the multimodal data and based on the semantic similarity between them, to obtain multiple candidate data blocks with a sequential order.

[0099] It should be noted that multimodal data may contain long, continuous segments of text, and preprocessed multimedia data contains at least one tag. Since the preprocessed multimedia data and text data are processed in blocks in a sequential manner, the resulting candidate data blocks may contain at least one tag or may contain only text data.

[0100] In this implementation, the preprocessed multimedia data and text data are segmented based on their order and semantic similarity in the multimodal data to obtain multiple candidate data blocks. This allows for the sequential segmentation of similar content in the multimodal data, facilitating subsequent retrieval and matching processes.

[0101] 506. Store the candidate data blocks in the enhanced content database.

[0102] The enhanced content database stores multiple search enhancement content items. Candidate data blocks are data obtained by segmenting preprocessed multimedia and text data. In some embodiments, multiple candidate data blocks are used as multiple search enhancement content items and stored in the enhanced content database.

[0103] In one possible implementation, candidate data blocks are encoded into multiple candidate vectors, and the candidate vectors are stored in an enhanced content database.

[0104] In this system, multiple search enhancements in the enhanced content database are stored as candidate vectors. In some embodiments, the enhanced content database is Opensearch. Candidate data blocks are encoded into multiple candidate vectors, and these candidate vectors are stored sequentially in the enhanced content database, with one candidate vector corresponding to one candidate data block.

[0105] In some embodiments, candidate data blocks are encoded into multiple candidate vectors through vector transformation. Vector transformation can be implemented using embedding, a technique that transforms high-dimensional sparse feature vectors into low-dimensional dense word vectors. Embedding can be used for dimensionality reduction, dimensionality increase, similarity calculation, etc. Through embedding, candidate data blocks can be encoded into multiple candidate vectors. In some embodiments, vector transformation can be implemented using residual networks, a type of network capable of classification and object recognition. Through residual networks, candidate data blocks can be encoded into multiple candidate vectors.

[0106] In this implementation, by storing multiple candidate vectors encoded by candidate data blocks into the enhanced content database, vector matching can be performed directly when using the enhanced content database to retrieve enhanced content, reducing the time required to output the target response content and improving the processing efficiency of the data processing method.

[0107] 507, in response to retrieving the target question, retrieve at least one search enhancement content corresponding to the target question, the search enhancement content including at least one target tag.

[0108] Here, the target question refers to the data input into the target large model. In some embodiments, the target question is retrieved in response to a user's query. Retrieval enhancement content is content obtained from external knowledge sources through retrieval of the target question. Since retrieval enhancement content is obtained by vector encoding of candidate data blocks including at least one label, it includes at least one target label. In some embodiments, a target question has at least one retrieval item.

[0109] In some embodiments, the target problem is the data input to the target application. The target problem can be generated by any suitable target object, such as a person, a robot, etc. The target problem can be of any suitable type, such as audio data, text data, gesture data, etc. This application does not limit the type of target problem.

[0110] In some embodiments, the target question is obtained through a target interface. The target interface may include, but is not limited to, at least one of the following: a chat box, a form, etc. The target interface can be set in the application, so that the target question can be obtained by listening for chat box or form submissions.

[0111] In some embodiments, the target problem is obtained through an Application Programming Interface (API). An API serves as a medium for communication and data exchange between software systems. Requests can be sent to external services via an API, and the returned data constitutes the target problem.

[0112] In some embodiments, the target problem is acquired through sensors in an electronic device. Sensors may include, but are not limited to, cameras, microphones, etc. Listening events for the sensors can be set, and the target problem can then be acquired in real time through these listening events.

[0113] In some embodiments, a query is performed on a first relational table using the target question to obtain at least one enhanced retrieval content corresponding to the target question.

[0114] The first relation table stores multiple target questions and at least one enhanced search result corresponding to each target question. By querying the first relation table using a target question, at least one enhanced search result corresponding to the target question can be obtained. This first relation table is labeled by a technician according to the actual situation, and this application embodiment does not limit it.

[0115] In one possible implementation, in response to obtaining the target question, a query vector corresponding to the target question is determined. The query vector is then matched against multiple candidate vectors in an enhanced content database to obtain at least one target vector, which is a candidate vector that successfully matches the query vector. The enhanced content corresponding to the at least one target vector is then determined as at least one enhanced content corresponding to the target question.

[0116] To provide a clearer explanation of the above implementation methods, the following section will describe the process of determining the query vector corresponding to the target question and determining the enhanced retrieval content in two parts.

[0117] Part 1: In response to obtaining the target question, determine the query vector corresponding to the target question.

[0118] Here, the target question refers to the data input into the target large model. In some embodiments, the target question is retrieved in response to a user's query. The query vector is the vector corresponding to the target question. In some embodiments, different target questions correspond to different query vectors, with one target question corresponding to one query vector.

[0119] In some embodiments, when determining the query vector corresponding to the target question, the target question can be directly encoded to obtain the query vector corresponding to the target question, or the keywords of the target question can be extracted first, and then the keywords of the target question can be encoded to obtain the query vector corresponding to the target question.

[0120] In some embodiments, after obtaining the target question, the amount of text data in the target question is first determined. If the amount of text data in the target question is greater than or equal to a preset value, keywords of the target question are extracted first, and then the keywords are encoded to obtain the query vector corresponding to the target question. If the amount of text data in the target question is less than the preset value, the target question is directly encoded to obtain the query vector corresponding to the target question. The preset value can be any suitable size, such as 20, 25, etc. Through this process, the length of the query vector corresponding to the target question can be reduced, thereby improving the processing efficiency of the target large model.

[0121] In some embodiments, the target question is encoded into a query vector through vector transformation. This vector transformation can be implemented using embedding, a technique that transforms high-dimensional sparse feature vectors into low-dimensional dense word vectors. Embedding can be used for dimensionality reduction, dimensionality increase, similarity calculation, etc. Through embedding, the target question can be encoded into a query vector. In some embodiments, vector transformation can be implemented using residual networks, a type of network capable of classification and object recognition. Through residual networks, the target question can be encoded into a query vector.

[0122] The second part involves matching the query vector with multiple candidate vectors in the enhanced content database to obtain at least one target vector.

[0123] The enhanced content database stores multiple search enhancements. These enhancements are stored as vectors, serving as candidate vectors. The query vector is matched against these candidate vectors, and at least one candidate vector in the enhanced content database that is related to the query vector is used as the target vector to obtain at least one target vector.

[0124] In some embodiments, at least one target vector is obtained by using a query vector and multiple candidate vectors in an enhanced content database through vector search or keyword matching. Vector search is a search technique used to find similar items or data points in a large collection. Keyword matching is a common text processing task, typically used for searching, replacing, and validating text.

[0125] In some embodiments, when processing a query vector and multiple candidate vectors in an enhanced content database through vector search, a target similarity between the query vector and the multiple candidate vectors in the enhanced content database can be determined, and then a target problem can be determined based on the target similarity.

[0126] In some embodiments, candidate vectors with a target similarity greater than or equal to a similarity threshold are used as target vectors to obtain at least one target vector. The similarity threshold can be any suitable value, such as 85%, 90%, etc. In some embodiments, the target similarities are sorted, and the candidate vectors with the top N target similarities are used as target vectors to obtain at least one target vector. N can be any suitable positive integer, such as 5, 3, etc.

[0127] In some embodiments, the target similarity is determined using a similarity calculation algorithm. This similarity calculation algorithm may include, but is not limited to, cosine similarity (CS) and Euclidean distance (ED). For example, using the cosine similarity algorithm, the cosine of the angle between the query vector and multiple candidate vectors in the enhanced content database is calculated, and this cosine value is used as the target similarity. As another example, using the Euclidean distance algorithm, the distance between the midpoints of the query vector and multiple candidate vectors in the enhanced content database is calculated, and this distance value is used as the target similarity.

[0128] Part Three: Determine at least one retrieval enhancement content corresponding to a target vector as at least one retrieval enhancement content corresponding to the target question.

[0129] In this context, multiple search enhancements in the enhanced content database are stored as candidate vectors. After determining the target vector, the search enhancement corresponding to the target vector is identified as at least one search enhancement relevant to the target question in the enhanced content database.

[0130] In this implementation, the target vector is obtained by matching the query vector corresponding to the target question with multiple candidate vectors in the enhanced content database. The enhanced content corresponding to the target vector is used as at least one enhanced content corresponding to the target question. This can accurately locate the enhanced content related to the target question and further improve the accuracy of the initial response content.

[0131] 508. Input the target question and at least one search enhancement into the target big model. The target big model processes the target question and at least one search enhancement and outputs the initial response content.

[0132] Here, a target large-scale model refers to a machine learning model with a large number of parameters and a complex computational structure, capable of processing massive amounts of data and completing various complex tasks. In some embodiments, the target large-scale model is an LLM (Limited Learning Model). The target large-scale model can be integrated into applications for intelligent question answering. The initial response content is the answer generated by the target large-scale model after understanding the target question. The initial response content includes at least one target label. The target label is information used to classify and tag multimedia data.

[0133] In some embodiments, the target question and at least one search enhancement are concatenated, and then the concatenated target question and at least one search enhancement are converted into vector form and input into the target large model. The target large model first encodes and then performs multiple rounds of iterative decoding to output the initial response content, so as to realize the processing of the target question and at least one search enhancement through the target large model and output the initial response content.

[0134] In some embodiments, the target question and at least one search enhancement content are concatenated using methods such as point-by-point addition or vector concatenation. Point-by-point addition adds the target question and at least one search enhancement content with the same feature dimensions to obtain the concatenated target question and at least one search enhancement content. Vector concatenation concatenates the target question and at least one search enhancement content in a specific order to obtain the concatenated target question and at least one search enhancement content.

[0135] For example, if the user inputs the target question as "How to manage contracts in DingTalk?", after obtaining at least one enhanced search result for this target question, the target question and at least one enhanced search result are processed through the target big model, and the initial response is output as follows: **Operation Steps**: 1. Switch to "Contract Management" and click "Add";<image_hris_4-19> 2. Employees who have signed the "Employment Relationship Transfer Agreement" should fill in the following information, selecting the new and correct "Contracting Company" and "Effective Date," etc. Contract Status: -If the employee's current agreement has already been signed and the contract is effective on October 1, 2022, then the new operation should be performed on or after October 1, and the contract status should be selected as "effective". - It is recommended to complete this operation on the same day or the next day if possible.<image_hris_4-20> 3. After completing the "Add" operation, manually invalidate the labor contract of the source contract subject; -**Before expiration**: Please ensure that the employee's employment contract information has been correctly updated, that the contract has reached its effective date, and that the status is "effective".<image_hris_4-21><image_hris_4-22> In one possible implementation, a prompt text template is obtained, which is used to prompt the retention of target tags from the search enhancement content when generating the response content. The target question and search enhancement content are populated into the prompt text template and processed to obtain the target prompt text. The target prompt text is then processed using a target big data model to output the initial response content.

[0136] To provide a clearer explanation of the above implementation methods, the process of obtaining the target prompt text and outputting the initial response content in the above implementation methods will be described in two parts below.

[0137] Part 1: Obtaining the prompt text template.

[0138] The prompt text template is a keyword or phrase used to guide the target large model in performing a specific task or dialogue. It prompts the model to retain the target tags from the retrieved augmented content when generating the response, helping the target large model understand the current settings and more accurately parse and process the input data. In some embodiments, the prompt text template is used to prompt the model to retain the target tags from the retrieved augmented content when generating the response. Methods for obtaining the prompt text template may include, but are not limited to, obtaining the prompt text template from the cloud or server, or obtaining the prompt text template via API.

[0139] The second part involves filling the target question and search enhancement content into the prompt text template for processing, resulting in the target prompt text.

[0140] The target suggestion text is data obtained by filling the suggestion text template with the target question and search enhancement content. In some embodiments, the suggestion text template has reserved spaces for filling in the target question and search enhancement content, which can be filled into the corresponding spaces to obtain the target suggestion text.

[0141] For example, the prompt text template could be: You are an expert using a smart mobile office platform software. Using the enhanced search content retrieved based on the target question as context, please answer the target question based on the context. Context: {xxxx}, Target Question: {xxxx}. When answering, please: 1. Include at least one target tag in the context and retain the target tag in the context. 2. An example of the target tag format is as follows:<IMG_xxx> ,<LINK_xxx> ,<WMV_xxx> .

[0142] Part Three: Process the target prompt text using the target large model and output the initial response content.

[0143] Here, a target large-scale model refers to a machine learning model with a large number of parameters and a complex computational structure, capable of processing massive amounts of data and completing various complex tasks. In some embodiments, the target large-scale model is an LLM (Limited Learning Model). The target large-scale model can be integrated into applications for intelligent question answering. The initial response content is the answer generated by the target large-scale model after understanding the target prompt text. The initial response content includes at least one target tag. Target tags are information used to classify and label multimedia data.

[0144] In this implementation, the target prompt text is obtained by filling the target question and search enhancement content into the acquired prompt text template. The target prompt text is then processed by the target big model to output the initial response content. The output format of the target big model is dynamically controlled by the prompt text template, enabling the target big model to recognize and maintain predefined tags, thereby generating the initial response content containing the target tags. This further ensures the accurate replacement of multimedia data and tags.

[0145] 509. Replace at least one target tag in the initial response content with the corresponding multimedia data to obtain the target response content corresponding to the target question.

[0146] The target response content refers to the data obtained by replacing the target tags in the initial response content with multimedia data. In some embodiments, after obtaining the initial response content, at least one target tag in the initial response content is filtered, and then the at least one target tag is replaced with the corresponding multimedia data to obtain the target response content corresponding to the target question.

[0147] In one possible implementation, a target mapping relationship related to the target tag is queried. This target mapping relationship characterizes the correspondence between the target tag and multimedia data. Based on the target mapping relationship, at least one target tag in the initial response content is replaced with the corresponding multimedia data to obtain the target response content corresponding to the target question.

[0148] Here, the target mapping relationship refers to the correspondence between target tags and multimedia data. In some embodiments, the target mapping relationship is unique, and each target mapping relationship includes a target tag and the multimedia data corresponding to that target tag. The multimedia data is the multimedia data corresponding to the target tag. In some embodiments, the multimedia data may include, but is not limited to, at least one of the following: text data, image data, audio data, video data, URL data, etc.

[0149] In some embodiments, the target mapping relationship is stored in a tag mapping management module, which includes a mapping table. The mapping relationship between multimedia data and the tags corresponding to the multimedia data is stored in the mapping table and managed using a database (such as SQLite, PostgreSQL) or an in-memory data structure (such as a hash table).

[0150] In this implementation, at least one target tag in the initial response content is replaced with the corresponding multimedia data by querying the target mapping relationship, so as to obtain the target response content corresponding to the target question. The target mapping relationship can ensure the accurate correspondence between the target tag and the multimedia data, further improving the reliability of the target response content.

[0151] 510, Output the target response content.

[0152] Since the target response content includes multimedia data corresponding to at least one target tag, the multimedia data that needs to be embedded in the target response content can be derived through a combination of text and images. Therefore, when outputting the target response content, multimodal data, including but not limited to text data, image data, URL data, and video data, can be output.

[0153] In one possible implementation, a target display format for the target response content is determined. The target response content is then adjusted based on the target display format to output the adjusted target response content.

[0154] The target display format refers to the format in which the target response content is displayed. The target display format can include, but is not limited to, one of the following: HTML, RTF, Graphical User Interface (GUI) format, audio format, video format, etc. RTF is a text and graphic document format that is easy to view on different devices and systems. GUI is a computer user interface that displays information graphically.

[0155] In some embodiments, the target display format of the target response content is related to user preferences, and the target display format of the target response content can be determined based on user preferences. First, user preferences are determined based on historically output target response content. Then, the target display format of the target response content is determined based on user preferences. For example, if most historically output target response content uses a GUI format, it can be determined that the user's preference is for a GUI format. Based on user preferences, the target display format is determined to be a GUI format, the target response content is adjusted to the GUI format, and the adjusted target response content is output.

[0156] In some embodiments, the target display format of the target response content is related to the type of the target question. After obtaining the target response content, the type of the target question is used as the target display format, and the target response content is adjusted based on the target display format to output the adjusted target response content. The target question can be any suitable type, such as audio data, text data, gesture data, etc. For example, if the target question is audio data, the determined target response content will be output in audio format.

[0157] In this implementation, by dynamically adjusting the output method of the target response content through a defined target display format, the flexibility of the target response content output is improved. The target response content can be output according to user preferences, thereby enhancing the user experience.

[0158] In summary, the method provided in this application processes the target question using a target large model to obtain initial response content including at least one target tag. Then, it replaces at least one target tag in the initial response content with corresponding multimedia data to obtain and output the target response content corresponding to the target question. This method generates tags during the initial response content generation, avoiding the direct insertion of complex multimedia data and simplifying the model's processing. Furthermore, by replacing tags, the generated target response content includes at least one multimedia data point, adapting to scenarios involving outputting single or multiple multimedia data points. It can simultaneously meet the multimedia needs of both simple and complex scenarios, and is particularly suitable for step-by-step question-and-answer scenarios. It can achieve efficient and accurate multimodal data mixed-format question-and-answer within the RAG framework.

[0159] It should be understood that the above examples are provided to help those skilled in the art understand the embodiments of this application, and are not intended to limit the embodiments of this application to the specific values ​​or scenarios illustrated. Those skilled in the art can obviously make various equivalent modifications or changes based on the above examples, and such modifications or changes also fall within the scope of the embodiments of this application.

[0160] The above text combined Figures 1 to 5 The data processing method provided in the embodiments of this application is described in detail below; the following will be combined with Figure 6 and Figure 7 The apparatus embodiments of this application are described in detail below. It should be understood that the apparatus in the embodiments of this application can perform the various methods described in the foregoing embodiments of this application, that is, the specific working processes of the various products described below can be referred to the corresponding processes in the foregoing method embodiments.

[0161] Figure 6 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Figure 6As shown, the data processing device 600 includes: an acquisition module 610, a replacement module 620, and an output module 630. Wherein: The acquisition module 610 is used to acquire the initial response content corresponding to the target question through the target large model. The initial response content includes at least one target tag, which is used to replace the corresponding multimedia data. Replacement module 620 is used to replace at least one target tag in the initial response content with the corresponding multimedia data to obtain the target response content corresponding to the target question; Output module 630 is used to output the target response content.

[0162] In one possible implementation, the replacement module 620 is further configured to query a target mapping relationship related to a target tag, the target mapping relationship being used to characterize the correspondence between the target tag and multimedia data; based on the target mapping relationship, at least one target tag in the initial response content is replaced with the corresponding multimedia data to obtain the target response content corresponding to the target question.

[0163] In one possible implementation, the acquisition module 610 is further configured to, in response to acquiring a target question, acquire at least one search enhancement content corresponding to the target question, the search enhancement content including at least one target tag; input the target question and at least one search enhancement content into a target big model, process the target question and at least one search enhancement content through the target big model, and output initial response content.

[0164] In one possible implementation, the acquisition module 610 is further configured to acquire a prompt text template, which is used to prompt the retention of target tags in the search enhancement content when generating response content; fill the prompt text template with the target question and search enhancement content for processing to obtain target prompt text; process the target prompt text through the target big model to output the initial response content.

[0165] In one possible implementation, the acquisition module 610 is further configured to, in response to acquiring the target question, determine the query vector corresponding to the target question; match the query vector with multiple candidate vectors in the enhanced content database to obtain at least one target vector, wherein the target vector is a candidate vector that successfully matches the query vector; and determine the retrieval enhanced content corresponding to the at least one target vector as at least one retrieval enhanced content corresponding to the target question.

[0166] In one possible implementation, the output module 630 is further configured to determine the target display format of the target response content; adjust the target response content based on the target display format, and output the adjusted target response content.

[0167] In one possible implementation, the acquisition module 610 is further configured to acquire at least one multimodal data and extract text data and multimedia data from the multimodal data, wherein the text data is used to explain the multimedia data; the device further includes a determination module for determining the tags corresponding to the multimedia data and establishing a mapping relationship between the multimedia data and the tags corresponding to the multimedia data; the device further includes a storage module for storing the mapping relationship between the multimedia data and the tags corresponding to the multimedia data.

[0168] In one possible implementation, the determining module is further configured to extract semantic information corresponding to multimedia data from text data; add the semantic information corresponding to multimedia data to the tag corresponding to multimedia data, or add the semantic information corresponding to multimedia data to the context of the tag corresponding to multimedia data.

[0169] In one possible implementation, the replacement module 620 is further configured to perform replacement processing on the multimedia data based on the tags corresponding to the multimedia data to obtain preprocessed multimedia data; the device also includes a block segmentation module, configured to perform block processing on the preprocessed multimedia data and text data to obtain multiple candidate data blocks, wherein the candidate data blocks are a portion of the data in the multimodal data; the storage module is further configured to store the candidate data blocks in the enhanced content database.

[0170] In one possible implementation, the segmentation module is further configured to segment the preprocessed multimedia data and text data according to the order of the preprocessed multimedia data and text data in the multimodal data, based on the semantic similarity between the preprocessed multimedia data and text data, to obtain multiple candidate data blocks.

[0171] In one possible implementation, the storage module is further configured to store candidate data blocks into an enhanced content database, including: encoding the candidate data blocks into a plurality of candidate vectors and storing the candidate vectors into the enhanced content database.

[0172] The division of modules in the above data processing device is only for illustrative purposes. In other embodiments, the data processing device can be divided into different modules as needed to complete all or part of the functions of the above data processing device.

[0173] The various modules in the data processing apparatus provided in this application embodiment can be implemented in the form of computer programs. These computer programs can run on server or client electronic devices. The program modules constituted by these computer programs can be stored in the memory of the server or client electronic devices. When the computer program is executed by a processor, it implements all or part of the steps of the methods described in this application embodiment.

[0174] It should be noted that the aforementioned data processing device 600 is embodied in the form of a functional unit. The term "module" here can be implemented in software and / or hardware, without specific limitations.

[0175] For example, a "module" can be a software program, a hardware circuit, or a combination of both that implements the above functions. The hardware circuit may include an application-specific integrated circuit (ASIC), electronic circuitry, a processor (e.g., a shared processor, a proprietary processor, or a group processor) and memory for executing one or more software or firmware programs, integrated logic circuitry, and / or other suitable components that support the described functions.

[0176] Therefore, the units of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0177] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0178] For example, such as Figure 7 As shown, the electronic device 700 includes a memory 701 and a processor 702. The memory 701 stores executable program code 7011, and the processor 702 is used to call and execute the executable program code 7011 to perform a data processing method.

[0179] For example, memory 701 can be used to store related programs of the data processing method provided in the embodiments of this application; processor 702 can call the related programs of the data processing method stored in memory 701 to execute the data processing method of the embodiments of this application; for example, by using a target large model, the initial response content corresponding to the target question is obtained, the initial response content includes at least one target tag, the target tag is used to replace the corresponding multimedia data; at least one target tag in the initial response content is replaced with the corresponding multimedia data to obtain the target response content corresponding to the target question; the target response content is output.

[0180] This embodiment can divide the device into functional modules based on the above method example. For example, each module can correspond to a separate function, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0181] When the functional modules are divided according to their respective functions, the device may also include a processing module and a communication module. It should be noted that all relevant content regarding the steps involved in the above method embodiments can be referenced from the functional descriptions of the corresponding functional modules, and will not be repeated here.

[0182] It should be understood that the apparatus provided in this embodiment is used to perform the above-described data processing method, and therefore can achieve the same effect as the above-described implementation method.

[0183] When using integrated units, the device may include a processing module and a storage module. The processing module may be a processor or a controller that can implement or execute various exemplary logic blocks, modules, and circuits shown in conjunction with the disclosure of this application. The processor may also be a combination of functions that implement computing capabilities, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc., and the storage module may be a memory.

[0184] In addition, the apparatus provided in the embodiments of this application may specifically be a chip, component or module. The chip may include a connected processor and a memory. The memory is used to store instructions. When the processor calls and executes the instructions, the chip can execute a data processing method provided in the above embodiments.

[0185] This application also provides a computer-readable storage medium storing computer program code, which, when run on a computer, causes the computer to execute the aforementioned method steps to implement a data processing method provided in the above embodiments. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, Digital Video Discs (DVDs), Compact Disc Read-Only Memory (CD-ROMs), microdrives, and magneto-optical disks, read-only memory (ROMs), random access memory (RAMs), erasable programmable read-only memory (EPROMs), electrically erasable programmable read-only memory (EEPROMs), dynamic random access memory (DRAMs), video random access memory (VRAMs), flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of media or device suitable for storing instructions and / or data.

[0186] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned related steps to implement a data processing method provided in the above embodiments.

[0187] The computer-readable storage medium, computer program product, or chip provided in this application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0188] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0189] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0190] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, The method is applied to electronic devices, including: Using the target big model, the initial response content corresponding to the target question is obtained. The initial response content includes at least one target tag, which is used to replace the corresponding multimedia data. Replace at least one target tag in the initial response content with the corresponding multimedia data to obtain the target response content corresponding to the target question; Output the target response content.

2. The method according to claim 1, characterized in that, The step of replacing at least one target tag in the initial response content with corresponding multimedia data to obtain the target response content corresponding to the target question includes: Query the target mapping relationship related to the target tag, the target mapping relationship being used to characterize the correspondence between the target tag and multimedia data; Based on the target mapping relationship, at least one target tag in the initial response content is replaced with the corresponding multimedia data to obtain the target response content corresponding to the target question.

3. The method according to claim 1, characterized in that, The process of obtaining the initial response content corresponding to the target question through the target large model includes: In response to obtaining the target question, at least one search enhancement content corresponding to the target question is obtained, and the search enhancement content includes at least one target tag; The target question and the at least one search enhancement content are input into the target big model, and the target big model processes the target question and the at least one search enhancement content to output the initial response content.

4. The method according to claim 3, characterized in that, The process of processing the target question and the at least one search enhancement content using the target large model to output the initial response content includes: Obtain a prompt text template, which is used to prompt that the target tags in the enhanced retrieval content be retained when generating response content; The target question and the enhanced search content are filled into the prompt text template and processed to obtain the target prompt text; The target prompt text is processed by the target large model to output the initial response content.

5. The method according to claim 3, characterized in that, The step of obtaining at least one enhanced search content corresponding to the target question in response to obtaining the target question includes: In response to obtaining the target question, a query vector corresponding to the target question is determined; The query vector is matched with multiple candidate vectors in the enhanced content database to obtain at least one target vector, which is a candidate vector that successfully matches the query vector. The search enhancement content corresponding to the at least one target vector is determined as at least one search enhancement content corresponding to the target question.

6. The method according to claim 1, characterized in that, The output of the target response content includes: Determine the target display format for the target response content; The target response content is adjusted based on the target display format to output the adjusted target response content.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: Acquire at least one multimodal data, and extract text data and multimedia data from the multimodal data, wherein the text data is used to explain the multimedia data; Determine the tags corresponding to the multimedia data, and establish a mapping relationship between the multimedia data and the tags corresponding to the multimedia data; The mapping relationship between the multimedia data and the corresponding tags of the multimedia data is stored.

8. The method according to claim 7, characterized in that, Determining the tag corresponding to the multimedia data includes: Extract semantic information corresponding to the multimedia data from the text data; The semantic information corresponding to the multimedia data is added to the tag corresponding to the multimedia data, or the semantic information corresponding to the multimedia data is added to the context of the tag corresponding to the multimedia data.

9. The method according to claim 7, characterized in that, The method further includes: The multimedia data is replaced based on the tags corresponding to the multimedia data to obtain the preprocessed multimedia data. The preprocessed multimedia data and text data are divided into blocks to obtain multiple candidate data blocks, which are portions of the multimodal data. The candidate data blocks are stored in the enhanced content database.

10. The method according to claim 9, characterized in that, The preprocessed multimedia data and text data are divided into blocks to obtain multiple candidate data blocks, including: Based on the order of the preprocessed multimedia data and the text data in the multimodal data, and the semantic similarity between the preprocessed multimedia data and the text data, the preprocessed multimedia data and the text data are divided into blocks to obtain the multiple candidate data blocks.

Images