Data processing method, device, medium, and program product

Abstract

The embodiment of the present specification provides a data processing method, device, medium and program product, and the embodiment of the present specification relates to the technical field of artificial intelligence, wherein the data processing method comprises: performing multi-modal encoding processing on a target problem to obtain a multi-modal problem code, and determining multi-modal data corresponding to the target problem; determining target reference data corresponding to the target problem from the multi-modal data according to the multi-modal problem code, wherein the target reference data is at least one kind of modal data in the multi-modal data; and using a problem processing model to process the target problem according to the target reference data to obtain a problem processing result corresponding to the target problem; and the problem that the data processing result is inaccurate due to the limited knowledge learned by the neural network model is avoided.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to a data processing method. One or more embodiments of this specification also relate to a computing device, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the continuous development of computer technology and artificial intelligence technology, neural network models can be used to process data and obtain corresponding data processing results in scenarios that require data processing.

[0003] Current neural network models, due to the limited knowledge they have learned, may produce inaccurate data processing results. Therefore, improving the accuracy of data processing results has become an urgent problem to be solved. Summary of the Invention

[0004] In view of the above, embodiments of this specification provide a data processing method. One or more embodiments of this specification also relate to a data processing apparatus, a computing device, an electronic device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a data processing method is provided, comprising:

[0006] The target problem is subjected to multimodal coding to obtain a multimodal problem code, and the multimodal data corresponding to the target problem is determined.

[0007] Based on the multimodal problem encoding, target reference data corresponding to the target problem is determined from the multimodal data, wherein the target reference data is at least one modality data in the multimodal data;

[0008] Using a problem-solving model, the target problem is processed based on the target reference data to obtain the problem-solving result corresponding to the target problem.

[0009] According to a second aspect of the embodiments of this specification, a data processing apparatus is provided, comprising:

[0010] The encoding module is configured to perform multimodal encoding processing on the target problem, obtain the multimodal problem code, and determine the multimodal data corresponding to the target problem;

[0011] The data determination module is configured to determine target reference data corresponding to the target problem from the multimodal data based on the multimodal problem encoding, wherein the target reference data is at least one modality data in the multimodal data;

[0012] The problem processing module is configured to use a problem processing model to process the target problem based on the target reference data, and obtain the problem processing result corresponding to the target problem.

[0013] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising:

[0014] Memory and processor;

[0015] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0016] According to a fourth aspect of the embodiments of this specification, an electronic device is provided, comprising:

[0017] A memory and a processor, the memory and the processor being connected via a bus;

[0018] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0019] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the data processing method described above.

[0020] According to a sixth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the data processing method described above.

[0021] This specification provides one or more embodiments of a data processing method. During data processing, this method performs multimodal encoding on a target problem to obtain a multimodal problem code. Using this multimodal problem code, it determines at least one modality of target reference data corresponding to the target problem from the multimodal data. This target reference data can be external reference knowledge consulted by the problem processing model during the processing of the target problem. Then, using the problem processing model, the target problem is processed based on the at least one modality of target reference data, thereby obtaining accurate problem processing results. This avoids the problem of inaccurate data processing results due to the limited knowledge learned by the neural network model. Attached Figure Description

[0022] Figure 1 This is an application scenario diagram of a data processing method provided in one embodiment of this specification;

[0023] Figure 2 This is a flowchart illustrating a data processing method provided in one embodiment of this specification;

[0024] Figure 3 This is a schematic diagram of a bimodal distribution in a data processing method provided in one embodiment of this specification;

[0025] Figure 4 This is a flowchart illustrating the processing procedure of a data processing method provided in one embodiment of this specification.

[0026] Figure 5 This is a schematic diagram of the structure of a data processing apparatus provided in one embodiment of this specification;

[0027] Figure 6 This is a structural block diagram of a computing device provided in one embodiment of this specification;

[0028] Figure 7 This is a structural block diagram of an electronic device provided in one embodiment of this specification. Detailed Implementation

[0029] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0030] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0031] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0032] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0033] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0034] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0035] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0036] Gaussian Mixture Model (GMM) is a probabilistic model used to represent complex distributions in a dataset. It assumes that the data is composed of several Gaussian distributions combined with certain weights. In classification tasks, especially binary classification problems, GMMs can be used to model the data distribution for each class, and then classification can be performed based on these models.

[0037] The EM (Expectation-Maximization) algorithm is an iterative method suitable for solving problems with latent variables, such as in Gaussian Mixture Models (GMMs) where the Gaussian component from which each data point originates is unknown. The EM algorithm is used in GMMs to learn better parameters of the model from the data, thereby accurately constructing a GMM with a Gaussian mixture distribution.

[0038] LLMs (Large Language Models): These are large language models. LLMs are models built using deep learning techniques that can generate human-like text from text input.

[0039] VLMs (Vision-Language Models): These are visual language models, which are built using deep learning technology and can generate human-like text from images and words.

[0040] RAG (Retrieval-Augmented Generation): This can be called retrieval-enhanced generation; RAG refers to the technique of large models using documents retrieved from external knowledge bases to improve the quality of their own answers.

[0041] Multimodal: refers to the integration and processing of data from different senses or information sources. In the field of artificial intelligence, multimodal models can simultaneously understand and process multiple types of data input, such as text, images, sound, and video.

[0042] Multiscale: In image processing and analysis, this refers to the process of manipulating images at different scale levels to accommodate features of different sizes and details. This multiscale processing method involves image scaling.

[0043] Visual document information refers to document information that includes both visual and textual elements. This includes textual information as well as visual information that conveys information through shapes, spatial arrangements, and other visual formats, such as tables, flowcharts, and line graphs.

[0044] With the continuous development of computer technology and artificial intelligence, neural network models can be used to process data and obtain corresponding results in scenarios requiring data processing. However, current neural network models, due to the limited knowledge they have learned, may produce inaccurate results. For example, when using large models for semantic reasoning, their limited knowledge prevents them from accurately deriving the answer to the question.

[0045] To address the aforementioned issues, this specification provides a text-based retrieval enhancement generation scheme. This scheme typically uses OCR (Optical Character Recognition) to convert text information in an image into text, allowing a neural network model to infer answers based on this text. However, this scheme neglects the visual features of the image itself. For example, line graphs not only contain numerical information about data points but also convey trends and patterns through line variations; this information is often lost during OCR processing. Therefore, multimodal retrieval enhancement generation schemes are often limited by the generation model and can only accept a single image as input, or when multiple images can be input, they are severely affected by noise interference, impacting the quality and accuracy of the responses.

[0046] To address the aforementioned technical problems, this specification provides a data processing method. This specification also relates to a data processing apparatus, a computing device, an electronic device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0047] Considering the large number of model parameters in large models and the limited computing resources of mobile terminals, the data processing method provided in this application can be applied to, for example, Figure 1 The application scenarios shown are not limited to these. In, for example... Figure 1 In the application scenario shown, the large model is deployed on server 10. Server 10 can connect to one or more client devices 20 via a local area network (LAN), wide area network (WAN), internet connection, or other types of data network. These client devices 20 may include, but are not limited to, smartphones, tablets, laptops, PDAs, personal computers, smart home devices, and in-vehicle devices. Client devices 20 can interact with users through a graphical user interface to access the large model, thereby implementing the methods provided in the embodiments of this specification.

[0048] In the embodiments of this specification, the system consisting of client device 20 and server 10 can perform the following steps: client device 20 performs the operation of uploading query to server 10, and server 10 can perform the steps of retrieving reference data and generating answer; wherein the step of retrieving reference data includes: performing multimodal encoding on query to obtain query encoding of multiple modalities; and using the query encoding of multiple modalities to retrieve target reference data from multimodal data; the step of generating answer includes: inputting the reference data and query into a multimodal large model (i.e., a question processing model) to perform answer reasoning and obtain the answer (i.e., the question processing result) corresponding to the query.

[0049] It should be noted that, provided that the client device's operating resources can meet the deployment and operation conditions of the large model, the embodiments of this application can be performed on the client device.

[0050] See Figure 2 , Figure 2 A flowchart of a data processing method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0051] Step 202: Perform multimodal coding on the target problem to obtain the multimodal problem code, and determine the multimodal data corresponding to the target problem.

[0052] The target question can be understood as a question that requires reasoning using a problem-solving model; for example, a query that is processed using a large model. This method can receive the target question sent by the client, which can be sent by the client during the process of the user performing a problem-solving operation based on the problem-solving interface.

[0053] The problem-solving interface can be understood as a human-computer interaction interface used to process the target problem. For example, the problem-solving interface can be a webpage, an application interface, etc.; the problem-solving operation can be understood as an operation such as uploading or inputting information related to the target problem.

[0054] Multimodal coding can be understood as a process of encoding a target problem into multiple modalities. The multimodal problem encoding can be the encoded data obtained after performing multimodal coding on the target problem. For example, this method can encode the target problem into a first modal problem encoding and a second modal problem encoding.

[0055] The first modality problem code can be any one of text modality problem code, image modality problem code, and audio modality problem code, and the second modality problem code can be any one of text modality problem code, image modality problem code, and audio modality problem code; the first modality problem code and the second modality problem code correspond to different modalities.

[0056] Multimodal data can be understood as various types of data, such as text data, image data, and audio data. In one or more embodiments provided in the specification, the multimodal data includes first modal data and second modal data; the first modal data can be any one of image data, text data, and audio data, and the second modal data can be any one of image data (e.g., target document image), text data (e.g., target image text), and audio data; the modal types of the first modal data and the second modal data are different; the text data can be obtained by extracting text from image data or audio data; the image data can be obtained by visualizing audio data.

[0057] This text modal problem encoding can be understood as problem encoding with the same encoding format as text modal encoding; this text modal encoding is coded data obtained after encoding text data; the encoding format can be encoding size, encoding dimension, encoding size, etc. In other words, this method can encode text data to obtain text modal encoded data; and, by performing encoding processing corresponding to the text data on the target problem, it obtains text modal problem encoding with the same encoding format as text modal encoding.

[0058] This image modality problem coding can be understood as problem coding with the same coding format as image modality coding; image modality coding is coded data obtained after encoding image data; the coding format can be coding size, coding dimension, coding size, etc. In other words, this method can encode image data to obtain image modality coded data; and, by performing coding processing corresponding to the image data on the target problem, it obtains image modality problem coding with the same coding format as image modality coding.

[0059] This audio modal problem encoding can be understood as problem encoding with the same encoding format as audio modal coding; this audio modal coding is the encoded data obtained after encoding and processing audio data; the encoding format can be encoding size, encoding dimension, encoding size, etc. In other words, this method can encode and process audio data to obtain audio modal encoded data; and, by performing encoding processing corresponding to the audio data on the target problem, it obtains audio modal problem encoding with the same encoding format as audio modal coding.

[0060] In one or more embodiments provided in this specification, the first modal problem is encoded as an image modal problem, the second modal problem is encoded as a text modal problem, the first modal data is a target document image, and the second modal data is target image text;

[0061] The process of performing multimodal coding on the target problem to obtain a multimodal problem code includes:

[0062] The target problem is encoded using an image modality coding module to obtain the image modality problem code, and the target problem is encoded using a text modality coding module to obtain the text modality problem code.

[0063] The image modality coding module can be understood as a module used to perform image modality coding processing on the target problem; the image modality coding module can be an image modality coding model, a neural network model, or one or more network layers; for example, the image modality coding module can be an embedding model, which can be a multimodal embedding model used to perform image modality coding processing on the target problem.

[0064] A text modality coding module can be understood as a module used to perform text modality coding processing on a target problem; the text modality coding module can be a text modality coding model, a neural network model, or one or more network layers; for example, the text modality coding module can be a text embedding model, which is used to perform text modality coding processing on the target problem.

[0065] The image modal problem coding can be understood as the encoding obtained after performing image modal coding processing on the target problem; the image modal problem encoding can be a vector or a matrix.

[0066] The text modal problem encoding can be understood as the encoding obtained after performing text modal encoding on the target problem; the text modal problem encoding can be a vector or a matrix.

[0067] The target document image can be understood as the image obtained after image conversion of the target document; the target image text can be the text extracted from the target document image.

[0068] Taking the application of the data processing method provided in this specification in the scenario of visual document information retrieval enhancement generation as an example, the data processing method is explained; wherein, the image modality coding module can be a multimodal embedding model, and the text modality coding module can be a text embedding model.

[0069] Based on this, after receiving a query from a user, this method retrieves reference data (also known as recall) based on that query. During the recall phase, the query is encoded into a query text vector (i.e., text modal question encoding) using a text embedding model, and into a query image vector (i.e., image modal question encoding) using a multimodal embedding model. It should be noted that both the query text vector and the query image vector are high-dimensional vectors. This facilitates the accurate determination of target reference data from multimodal data based on the subsequent image modal question encoding and text modal question encoding.

[0070] Step 204: Based on the multimodal problem encoding, determine the target reference data corresponding to the target problem from the multimodal data, wherein the target reference data is at least one modality data in the multimodal data.

[0071] The target reference data can be understood as the reference knowledge or external reference data required by the problem-solving model in the process of processing the target problem; the target reference data can be text data and / or image data.

[0072] In one or more embodiments provided in this specification, the multimodal problem coding is a first modal problem coding and a second modal problem coding;

[0073] The step of determining the target reference data corresponding to the target problem from the multimodal data based on the multimodal problem encoding includes steps one to three:

[0074] Step 1: Determine the first modal data and the second modal data from the multimodal data, wherein the second modal data is data generated using the first modal data that has a different modality from the first modal data.

[0075] Specifically, this method can determine the multimodal data stored in the data storage unit, and determine the first modality data and the second modality data from the multimodal data; wherein, the data storage unit can be understood as a unit that stores multimodal data, for example, the data storage unit can be a database, a server, a local disk, etc.

[0076] Following the previous example, this method can determine document images and text from a document database (i.e., a data storage unit).

[0077] In one or more embodiments provided in this specification, determining the multimodal data corresponding to the target problem includes:

[0078] From multiple target document files, at least one page of document is determined in each target document file, and the multiple pages of document are converted into images to obtain multiple target document images, wherein the multiple target document files contain information for processing the target problem;

[0079] Text extraction is performed on the multiple target document images to obtain multiple target image texts;

[0080] The plurality of target document images and the plurality of target image texts are determined as the multimodal data corresponding to the target question.

[0081] The target document file can be understood as a file containing text and images. For example, the target document file is a rich text or multimodal document; the image can be a table, line chart, or other image.

[0082] Continuing with the previous example, the problem-solving model in this method, during the processing of the query (i.e., the target question), struggles to reflect these latest changes given the constantly updating and evolving information on a fixed training dataset. Consequently, the model's output (i.e., the problem-solving result) may contain seemingly reasonable but inaccurate or fabricated information. Therefore, RAG technology can be utilized to enhance generation through visual document information retrieval. This allows access to external knowledge bases or documents, enabling the model to process new information not included in the training data or detailed information specific to a particular domain, thus better understanding and responding to complex queries.

[0083] When performing visual document information retrieval enhancement generation, a multimodal document parsing operation needs to be performed beforehand. This operation involves converting document files in the document database into multiple target document images and multiple target image texts. These document files may contain knowledge or information for processing the target problem; this allows subsequent problem-solving models to accurately analyze the target problem based on this external knowledge and obtain accurate problem-solving results.

[0084] The specific method of multimodal document parsing is as follows: The files in the document database (i.e., the target document files) in this method can be composed of various formats, such as PDF, PPT, DOC and other types of documents. For this document, each page in the document can be uniformly converted into a target document image in JPG format and stored in the document database; where one page of the document corresponds to one target document image.

[0085] Then, for multiple target document images, OCR technology can be used to parse them into text-location pairs, that is, the text has a corresponding text location, which can be represented as a bounding box. The text is divided into text chunks on a page-by-page basis, thus obtaining multiple text chunks (i.e., multiple target image texts); each text chunk contains one or more text-location pairs.

[0086] As can be seen from the above embodiments, this method can extract target image text from target document images, thereby improving the quality and accuracy of question answering for multimodal document databases.

[0087] Step 2: Using the first modal problem coding, determine the first modal reference data from the first modal data, and using the second modal problem coding, determine the second modal reference data from the second modal data.

[0088] The first modal reference data can be understood as the data related to the target problem in the first modal data. This first modal reference data can be the data that the problem-solving model needs to refer to when processing the target problem.

[0089] The second modal reference data can be understood as the data in the second modal data that is related to the target problem. This second modal reference data can be the data that the problem-solving model needs to refer to when processing the target problem.

[0090] Specifically, the step of determining first modal reference data from the first modal data using the first modal problem coding, and determining second modal reference data from the second modal data using the second modal problem coding, includes:

[0091] The first modality reference data is determined from the target document image using the image modality problem coding, and the second modality reference data is determined from the target image text using the text modality problem coding.

[0092] Following the previous example, in the process of retrieval enhancement generation, this method can determine the target document image related to the query from the target document images stored in the document database based on the image modality problem encoding; and determine the text block related to the query from the text block (target image text) stored in the document database based on the text modality problem encoding.

[0093] In one or more embodiments provided in this specification, determining first modal reference data from the first modal data using the first modal problem coding, and determining second modal reference data from the second modal data using the second modal problem coding, includes:

[0094] The first modal data is encoded using a first modal coding model to obtain a first modal code, and the second modal data is encoded using a second modal coding model to obtain a second modal code.

[0095] The similarity between the first modal problem code and the first modal code is calculated to obtain a first similarity between the first modal problem code and the first modal code, and based on the first similarity, first modal reference data is determined from the first modal data;

[0096] The similarity between the second modal problem code and the second modal code is calculated to obtain a second similarity between the second modal problem code and the second modal code. Based on the second similarity, second modal reference data is determined from the second modal data.

[0097] The first modality coding model can be a model used to encode the first modality data, which can be a model used to encode the target document image, and the first modality coding can be an embedding model; the second modality coding model can be a model used to encode the second modality data, which can be a model used to encode the target image text, and the second modality coding can be an embedding model.

[0098] Here, the similarity can be understood as a numerical value used to represent the degree of similarity between two codes. The similarity can be cosine similarity or distance similarity; the first similarity can be the similarity between the first modal problem code and the first modal code; the second similarity can be the similarity between the second modal problem code and the second modal code.

[0099] Following the previous example, multimodal hybrid recall can be performed during the recall phase; the query will calculate similarity scores (i.e., first similarity and second similarity) from the vectors after embedding text and images respectively, and recall vectors based on the similarity scores.

[0100] Specifically, multimodal hybrid recall aims to combine the results of text retrieval and multimodal retrieval to obtain accurate reference data (i.e., target document images) and filter out noise to the greatest extent possible.

[0101] The specific steps for obtaining multimodal retrieval results (e.g., image recall results) are as follows:

[0102] 1. Employ a multimodal embedding model to embed the target document images, thereby encoding each target document image into a high-dimensional vector, and obtaining the document image vector embedding. Visual .

[0103] 2. Determine the Query image vector Embedding obtained after the Query is encoded through the multimodal Embedding model. Q And determine the document image vector obtained after the target document image is encoded by the image embedding model.

[0104] 3. Calculate the cosine similarity between the query image vector and the document image vector, and determine the target document image corresponding to the cosine similarity as the image recall result.

[0105] The image recall result is... Visual The calculation method can be found in the following formula:

[0106] Recall Visual =Similarity(Embedding) Q Embedding Visual )

[0107] Where Similarity is the cosine similarity calculation function; this Recall Visual This represents the target document image obtained by performing the recall operation.

[0108] 4. Use Gaussian Mixture Distribution (GMM) to perform binary classification on the image recall results to obtain the final image recall results (i.e., first modality reference data).

[0109] This method, through observation, determines that the similarity scores of image recall results conform to a bimodal distribution; this bimodal distribution (i.e., Gaussian mixture distribution) can be referenced. Figure 3 , Figure 3This is a schematic diagram of a bimodal distribution in a data processing method provided in one embodiment of this specification; wherein, Figure 3 The solid lines in the image represent target document images with low similarity; Figure 3 The dashed lines in the image represent target document images with high similarity.

[0110] Based on this, this method uses a Gaussian mixture distribution (GMM) to perform binary classification on the target document image, classifying images with higher similarity (i.e., those matching the given similarity criteria) into categories. Figure 3 The target document image (distributed by dashed lines) is used as the image recall result. Visual The calculation formula is as follows.

[0111] Recall Visual =GMM(Recall) Visual )

[0112] Here, GMM indicates that a Gaussian mixture distribution processing is performed; it should be noted that the GMM fitting distribution is determined using the EM algorithm.

[0113] The specific steps to obtain text retrieval results (e.g., text recall results) are as follows:

[0114] 1. Use a text embedding model to embed text blocks, so that each text block is encoded into a high-dimensional vector, thus obtaining the image text vector embedding. Textual .

[0115] 2. Determine the Query text vector Embedding obtained after the Query is encoded through the text embedding model. Q And determine the image text vector obtained after the target image text is encoded by the text embedding model.

[0116] 3. Calculate the cosine similarity between the query image vector and the image text vector, and determine the target image text corresponding to the cosine similarity as the text recall result. Textual .

[0117] The text recall result is Recall. Textual The calculation method can be found in the following formula:

[0118] Recall Textual =Similarity(Embedding) Q Embedding Textual )

[0119] 4. Use Gaussian Mixture Model (GMM) to perform binary classification on the text recall results to obtain the final text recall results (i.e., second modality reference data).

[0120] This method uses a Gaussian mixture distribution (GMM) to perform binary classification of text in target images, classifying text based on high similarity (i.e., similarity matching the given criteria). Figure 3 The target image text (distributed by dashed lines) is used as the final text recall result. Textual The calculation method is shown in the following formula.

[0121] Result Textual =GMM(Recall) Textual )

[0122] Step 3: Based on the first modal reference data and the second modal reference data, determine the target reference data corresponding to the target problem.

[0123] Specifically, this method can determine the first modal reference data and the second modal reference data as the target reference data corresponding to the target problem.

[0124] In one or more embodiments provided in this specification, there are multiple first modal reference data;

[0125] The step of determining the target reference data corresponding to the target problem based on the first modal reference data and the second modal reference data includes:

[0126] Determine target first modal reference data from multiple first modal reference data, wherein the target first modal reference data is the first modal reference data used to generate each second modal reference data;

[0127] The target first mode reference data is determined as the target reference data.

[0128] Following the previous example, after obtaining the text recall results and image recall results, this method can calculate the union of the text recall results and image recall results; specifically:

[0129] Since the target image text (i.e., the text recall result) is extracted from the target document image (i.e., the image recall result), it is necessary to determine the target document image (i.e., the target first modality reference data) from multiple target document images to generate the second modality reference data, and then define the target document image that generates the second modality reference data as the target reference data (i.e., the target document image). In other words, after obtaining the reference text (i.e., the second modality reference data) and multiple reference images (i.e., multiple first modality reference data) through two modalities of retrieval, this method needs to calculate the union of the reference text and the reference images as the final result (i.e., the target reference data). Since the reference text is extracted from the reference image, the union of the two can be the reference image from which the reference text was extracted (i.e., the target first modality reference data).

[0130] As can be seen from the above embodiments, in the process of information retrieval enhancement, this method determines the target reference data from multimodal data through various filtering methods, thereby ensuring the accuracy of the target reference data and avoiding the problem of low recall rate at the retrieval end affecting the quality of the answer.

[0131] Step 206: Using the problem processing model, process the target problem according to the target reference data to obtain the problem processing result corresponding to the target problem.

[0132] The problem-solving model can be understood as a model that processes the target problem. For example, the problem-solving model can be a multimodal large model, a large language model, a large model, a deep learning model, etc., without any specific restrictions.

[0133] The problem processing result can be understood as the result obtained by the problem processing model after processing the target problem. The problem processing result can be the answer to the problem or the response text to the target problem.

[0134] In one or more embodiments provided in this specification, the step of using a problem-solving model to process the target problem based on the target reference data and obtaining a problem-solving result corresponding to the target problem includes:

[0135] The target reference data and the target problem are input into the problem processing model, and in the problem processing model, the associated reference data corresponding to the target problem is determined from the target reference data;

[0136] The target problem is processed using the associated reference data to obtain the corresponding problem processing result.

[0137] The associated reference data can be understood as the part of the target reference data that is related to the target problem. For example, when the target reference data consists of multiple target document images, the associated reference data is the associated document image that is related to the target problem among the multiple target document images.

[0138] Following the previous example, in the process of multimodal and multiscale retrieval enhancement generation, this method inputs multiple recalled target document images and queries into a multimodal large model. In the multimodal large model, image analysis is performed on each target document image to determine the associated document images related to the query from multiple target document images. Then, the multimodal large model performs semantic reasoning on the query based on the associated document images to obtain the answer (i.e., the question processing result) corresponding to the query.

[0139] As can be seen from the above embodiments, in the process of processing the target problem using the problem processing model, this method can efficiently determine the associated reference data corresponding to the target problem from a small number of target reference data, reducing the computational pressure on the problem processing model; and, by using the associated reference data to process the target problem, accurate problem processing results can be obtained.

[0140] In one or more embodiments provided in this specification, the target reference data is a plurality of target document images;

[0141] The step of inputting the target reference data and the target problem into the problem processing model, and determining the associated reference data corresponding to the target problem from the target reference data in the problem processing model, includes steps one and two:

[0142] Step 1: Stitch the multiple target document images together to obtain the model input document image, and input the model input document image and the target question into the problem processing model.

[0143] Specifically, this method takes into account the scale limitation of the input data of the problem processing model, which can only input one or two images. Therefore, this method stitches multiple target document images together to obtain a model input document image. Then, the model input document image and the target question are input into the problem processing model, thereby overcoming the problem of limited context length of multimodal model in multimodal retrieval enhancement generation.

[0144] In one or more embodiments provided in this specification, the step of stitching together the plurality of target document images to obtain a model input document image includes:

[0145] Adjust the images of each target document according to the preset image size to obtain multiple adjusted document images;

[0146] Multiple adjusted document images are stitched together to obtain a stitched document image. The stitched document image is then adjusted according to the input image size of the problem processing model to obtain the model input document image.

[0147] The preset image size can be set according to the actual application scenario. For example, the preset image size can be in the form of 2x2, 3x3, 3x4, etc.

[0148] The input image size can be understood as the number of pixels or the size of the image that the problem-solving model can process. The input image size can be set according to the actual application scenario.

[0149] Following the previous example, this method includes a multi-scale generation stage in the process of multimodal and multi-scale retrieval enhancement generation; in this multi-scale generation stage, the model input generated by the multi-scale generation operation is a stitching of images of multiple recall results.

[0150] Specifically, the target document images recalled during the recall phase can be adjusted to image sizes of 2x2, 3x3, or 3x4 to obtain adjusted document images (with image sizes of 2x2, 3x3, or 3x4); then, multiple adjusted document images are stitched together into a single stitched image (i.e., a stitched document image).

[0151] After obtaining the stitched image, it can be reduced to the maximum pixel value (i.e., input image size) that the multimodal large model can input, thus obtaining the model input document image. input The specific calculation formula is as follows:

[0152] Image input =Resize(Concat(x1,x2,...,x i MaximumPixels

[0153] Here, x represents the input image, Max Pixels represents the maximum number of input pixels accepted by the current generative model, Resize represents size adjustment, and Concat represents image stitching.

[0154] As can be seen from the above embodiments, this method can adjust the stitched document image obtained by stitching multiple target document images according to the input image size of the problem processing model, thereby obtaining the model input document image and avoiding the problem of limited pixels in the model input image.

[0155] Step 2: In the problem processing model, determine the associated document image corresponding to the target problem from the input document images of the model.

[0156] In one or more embodiments provided in this specification, determining the associated document image corresponding to the target question from the plurality of target document images includes:

[0157] From the input document image of the model, determine the document image region corresponding to each target document image, and extract features from the target document image region to obtain the target document image region features, wherein the target document image region is any one of multiple document image regions;

[0158] Determine the feature similarity between the features of the target document image region and the features of the question corresponding to the target question, and based on the feature similarity, determine the associated document image region corresponding to the target question from the plurality of document image regions;

[0159] The associated document image region is identified as the associated document image.

[0160] In this context, the document image region can be understood as the image region or image block corresponding to each target document image in the model input document image.

[0161] The target document image region feature can be understood as a feature used to represent the image information contained in the target document image region. For example, the target document image region feature can be a matrix or a vector. The question feature can be understood as a feature used to represent the information contained in the target question. The question feature can be a matrix or a vector.

[0162] Feature similarity is a similarity measure that represents the degree of similarity between the features of the target document image region and the features of the question; this feature similarity can be cosine similarity or distance similarity.

[0163] Following the previous example, in the multi-scale generation stage, the generation of the answer to the question can be carried out in multiple rounds. Specifically, in each round of processing of the generative model, the generative model (i.e., the multimodal large model) will first summarize the information in the stitched image that is useful for answering the question; then, the generative model will select the patch (document image region) that it believes needs to be further enlarged from the upper left, upper right, lower left, and lower right of the input image.

[0164] Taking a 4x4 input image as an example:

[0165] First, in each round of processing, the multimodal large model will determine a stitched image region in the stitched image and perform feature extraction on the stitched image region selected in this round to obtain the stitched image region features. Here, one stitched image region corresponds to one recalled target document image; this feature extraction operation can be understood as summarizing the input image.

[0166] Secondly, in each round of processing, the multimodal large model will calculate the feature similarity between the features of the stitched image region and the query features;

[0167] Finally, the multimodal large model, based on feature similarity, determines the stitched image region associated with the query from multiple stitched image regions. Selected .

[0168] For example, select the 2x2 image in the top left corner as the selected image. Selected .

[0169] In the multi-round processing of multimodal large models, the selected image region will be processed. Selected Magnification is performed.

[0170] For example, if you select the top left corner for further viewing, the image area in the top left corner will be magnified to the maximum input pixels of the multimodal large model, and the pixels of each image will be magnified by an average of 4 times.

[0171] The specific method for enlarging an image is shown in the following formula:

[0172] Image Selected =Resize(Concat(x1,x2,...,x m ), MaximumPixels

[0173] Where x_m represents the selected image that needs to be further enlarged.

[0174] It should be noted that after completing the zoom-in operation in step 418, the zoomed-in image area will be... Selected Historical information (i.e., features of the stitched image region), query data, etc., are used as input data for the multimodal large model again.

[0175] In one or more embodiments provided in this specification, determining the feature similarity between the features of the target document image region and the features of the question corresponding to the target question, and determining the associated document image region corresponding to the target question from the plurality of document image regions based on the feature similarity, includes:

[0176] Determine the problem features corresponding to the target problem, and calculate the feature similarity between the target document image region features and the problem features;

[0177] The maximum feature similarity is determined from multiple feature similarities, and the document image region corresponding to the maximum feature similarity is determined from the multiple document image regions;

[0178] The document image region corresponding to the maximum feature similarity is determined as the associated document image region corresponding to the target question.

[0179] Following the example above, in each round of processing, the multimodal large model will calculate the feature similarity between the features of the stitched image region and the query features;

[0180] Then, the multimodal large model determines the maximum feature similarity and selects the stitched image region (i.e., the document image region) corresponding to the maximum feature similarity as the stitched image region Image associated with the query. Selected (i.e., the image region associated with the document).

[0181] As can be seen from the above embodiments, in the process of determining the associated document image region, this method can utilize the maximum feature similarity to select from multiple document image regions, thereby accurately identifying the associated document image region related to the target problem and providing accuracy for subsequent problem processing results.

[0182] In one or more embodiments provided in this specification, the step of using the associated reference data to process the target problem and obtain the problem processing result corresponding to the target problem includes:

[0183] Based on the region features of the associated document image and the target document image, the answer to the target question is inferred to obtain the answer to the target question.

[0184] Using the previous example, when a picture is finally selected, the multimodal large model will generate the final answer based on this picture and all historical information (i.e., the target document image region features). The historical information includes the model's thinking process in each step of the selection.

[0185] Based on the above embodiments, it can be seen that this method solves the problems of limited context length of multimodal models, limited pixels of input images, and low recall rate at the retrieval end in multimodal retrieval enhancement generation, which affect the quality of answers. By viewing the input image at different scales through image stitching, the model can pay more attention to valuable information, provide users with more robust evaluation results, and improve the quality of answers from retrieval enhancement generation technology.

[0186] This specification provides one or more embodiments of a data processing method. During data processing, this method performs multimodal encoding on a target problem to obtain a multimodal problem code. Using this multimodal problem code, it determines at least one modality of target reference data corresponding to the target problem from the multimodal data. This target reference data can be external reference knowledge consulted by the problem processing model during the processing of the target problem. Then, using the problem processing model, the target problem is processed based on the at least one modality of target reference data, thereby obtaining accurate problem processing results. This avoids the problem of inaccurate data processing results due to the limited knowledge learned by the neural network model.

[0187] The following is in conjunction with the appendix Figure 4 Taking the application of the data processing method provided in this specification in a scenario of enhanced visual document information retrieval and generation as an example, the data processing method will be further explained. Figure 4 The flowchart of a data processing method provided in one embodiment of this specification is shown. The method specifically includes three parts: multimodal document parsing and embedding, multimodal hybrid recall, and multiscale generation.

[0188] For details on multimodal document parsing and embedding, please refer to [link / reference]. Figure 4 Steps 402 to 410 in the process.

[0189] Step 402: Document database storage.

[0190] Specifically, the files in the document database (i.e., the target document files) in this method can be composed of various formats, such as PDF, PPT, DOC and other document types.

[0191] For this document, each page can be uniformly converted into a target document image in JPG format and stored in the document database; one page of the document corresponds to one target document image.

[0192] Step 404: Multimodal Embedding.

[0193] Multimodal embedding refers to directly embedding each page of the image. Specifically, a multimodal embedding model can be used to embed the target document images, thereby encoding each target document image into a high-dimensional vector, resulting in the document image vector embedding. Visual .

[0194] Step 406: Document parsing.

[0195] For multiple target document images, OCR technology can be used to parse them into text-location pairs, that is, the text has a corresponding text location, and the text location can be represented as a bounding box.

[0196] Step 408: Slice.

[0197] Specifically, the text is divided into text chunks on a page-by-page basis, resulting in multiple text chunks; each text chunk contains one or more text-position pairs.

[0198] Step 410: Text Embedding.

[0199] Specifically, text embedding refers to embedding the text after OCR of an image to obtain the image text vector. Textual .

[0200] The multimodal mixed recall stage can be referred to in step 412 below.

[0201] Step 412: Multimodal hybrid recall.

[0202] After receiving a query from a user, this method retrieves reference data based on that query (also known as multimodal hybrid recall); the specific execution steps include steps one through four as follows:

[0203] Step one: In the multimodal hybrid recall phase, this method performs multimodal encoding on the query, as follows:

[0204] 1. The query (i.e., the text question) is encoded into a high-dimensional vector (i.e., the query image vector below) through multimodal embedding, thereby aligning with the high-dimensional vector after image encoding;

[0205] 2. The Query is encoded into a high-dimensional vector (i.e., the Query text vector below) through text embedding, thereby aligning it with the high-dimensional vector of the OCR text.

[0206] Step two: In the multimodal hybrid recall stage, this method performs image retrieval to obtain image recall results. The specific execution method is as follows:

[0207] 1. Determine the Query image vector Embedding obtained after the Query is encoded through the multimodal Embedding model. Q And determine the document image vector Embedding obtained after encoding the target document image through the image Embedding model. Visual .

[0208] 2. Calculate the cosine similarity between the query image vector and the document image vector, and determine the target document image corresponding to the cosine similarity as the image recall result.

[0209] The image recall result is... Visual The calculation method can be found in the following formula:

[0210] Recall Visual =Similarity(Embedding) Q Embedding Visual )

[0211] Where Similarity is the cosine similarity calculation function; this Recall Visual This represents the target document image obtained by performing the recall operation.

[0212] 3. Use Gaussian Mixture Distribution (GMM) to perform binary classification on the image recall results to obtain the final image recall results.

[0213] This method, through observation, determines that the similarity scores of image recall results conform to a bimodal distribution; this bimodal distribution (i.e., Gaussian mixture distribution) can be referenced. Figure 3 , Figure 3 This is a schematic diagram of a bimodal distribution in a data processing method provided in one embodiment of this specification; wherein, Figure 3 The solid lines in the image represent target document images with low similarity; Figure 3 The dashed lines in the image represent target document images with high similarity.

[0214] Based on this, this method uses a Gaussian mixture distribution (GMM) to perform binary classification on the target document image, classifying images with higher similarity (i.e., those matching the given similarity criteria) into categories. Figure 3 The target document image (distributed by dashed lines) is used as the image recall result. Visual The calculation formula is as follows.

[0215] Result Visual =GMM(Recall) Visual )

[0216] Step three: In the multimodal hybrid recall stage, this method performs text retrieval to obtain text recall results. The specific execution method is as follows:

[0217] 1. Determine the query text vector Embedding obtained after the query is encoded through the text embedding model. QAnd determine the image text vector Embedding obtained after encoding the target image text through the text Embedding model. Textual .

[0218] 2. Calculate the cosine similarity between the query image vector and the image text vector, and determine the target image text corresponding to the cosine similarity as the text recall result. Textual .

[0219] The text recall result is Recall. Textual The calculation method can be found in the following formula:

[0220] Recall Textual =Similarity(Embedding) Q Embedding Textual )

[0221] 3. Use Gaussian Mixture Model (GMM) to perform binary classification on the text recall results to obtain the final text recall results.

[0222] This method uses a Gaussian mixture distribution (GMM) to perform binary classification of text in target images, classifying text based on high similarity (i.e., similarity matching the given criteria). Figure 3 The target image text (distributed by dashed lines) is used as the final text recall result. Textual The calculation method is shown in the following formula.

[0223] Result Textual =GMM(Recall) Textual )

[0224] Step four: After obtaining the text recall results and the image recall results, this method can calculate the union of the text recall results and the image recall results.

[0225] Specifically: Since the text chunk is extracted from the target document image, it is necessary to determine the target document image from multiple target document images to extract the chunk, and then determine the target document image from which the chunk is extracted as the final recall result.

[0226] The multi-scale generation stage can be referred to steps 414 to 420 below.

[0227] Step 414: Stitch the images together.

[0228] Specifically, in the multi-scale generation stage, the model input generated by the multi-scale generation operation is a stitch-together image of multiple recall results.

[0229] First, the multiple target document images recalled in the multimodal hybrid recall stage can be adjusted according to image sizes such as 2x2, 3x3, or 3x4 to obtain adjusted document images (with image sizes of 2x2, 3x3, or 3x4).

[0230] Secondly, multiple adjusted document images are stitched together into a single stitched image;

[0231] Finally, the stitched image is reduced to the maximum pixel value that the multimodal large model can input, thus obtaining the model input document image. input The specific calculation formula is as follows:

[0232] Image input =Resize(Concat(x1,x2,...,x i ), MaximumPixels

[0233] Here, x represents the input image, Max Pixels represents the maximum number of input pixels accepted by the current generative model, Resize represents size adjustment, and Concat represents image stitching.

[0234] Step 416: Summarize and select.

[0235] In the multi-scale generation stage, the generation of the answer to the question can be carried out in multiple rounds. Specifically, in each round of processing of the generative model, the generative model (i.e., the multimodal large model) will first summarize the information in the stitched image that is useful for answering the question; then, the generative model will select the patch (document image region) that it believes needs to be further enlarged from the upper left, upper right, lower left and lower right of the input image.

[0236] Taking a 4x4 input image as an example:

[0237] First, in each round of processing, the multimodal large model will determine a stitched image region in the stitched image and perform feature extraction on the stitched image region selected in this round to obtain the stitched image region features. Here, one stitched image region corresponds to one recalled target document image; this feature extraction operation can be understood as summarizing the input image.

[0238] Secondly, in each round of processing, the multimodal large model will calculate the feature similarity between the features of the stitched image region and the query features;

[0239] Finally, the multimodal large model will select the stitched image region with the highest feature similarity as the stitched image region associated with the query. Selected .

[0240] For example, select the 2x2 image in the top left corner as the selected image. Selected .

[0241] Step 418: Scale up.

[0242] Specifically, in the multi-round processing of a multimodal large model, the selected image region will be processed. Selected Magnification is performed.

[0243] For example, if you select the top left corner for further viewing, the image area in the top left corner will be magnified to the maximum input pixels of the multimodal large model, and the pixels of each image will be magnified by an average of 4 times.

[0244] The specific method for enlarging an image is shown in the following formula:

[0245] Image Selected =Resize(Concat(x1,x2,...,x m ), MaximumPixels

[0246] Where x_m represents the selected image that needs to be further enlarged.

[0247] It should be noted that after completing the zoom-in operation in step 418, the zoomed-in image area will be... Selected Historical information (i.e., features of the stitched image region), query data, etc., are used as input data for the multimodal large model again.

[0248] Step 420: Answer.

[0249] When a final image is selected, the multimodal big data model will generate the final answer to the query based on that image and all historical information.

[0250] Based on the above steps, it can be seen that the data processing method in this specification is a visual document information retrieval enhancement generation method based on multimodal and multi-scale queries. This method improves the quality and accuracy of question answering for multimodal document databases, thereby enhancing the satisfaction and user experience of building a search engine for its own database, especially rich text document databases containing rich visual information.

[0251] This method addresses the limitations of multimodal model context length and input image pixel count in multimodal retrieval enhancement generation by employing a multi-scale visual query approach. It also solves the problem of low recall affecting response quality by using GMM-based multimodal hybrid document search. Furthermore, by viewing input images at different scales through image stitching, the model focuses more on valuable information, providing users with more robust evaluation results and improving the quality of responses generated by retrieval enhancement technology.

[0252] Corresponding to the above method embodiments, this specification also provides data processing apparatus embodiments. Figure 5 A schematic diagram of the structure of a data processing apparatus according to one embodiment of this specification is shown. Figure 5 As shown, the device includes:

[0253] The encoding module 502 is configured to perform multimodal encoding processing on the target problem, obtain the multimodal problem code, and determine the multimodal data corresponding to the target problem;

[0254] The data determination module 504 is configured to determine target reference data corresponding to the target problem from the multimodal data based on the multimodal problem encoding, wherein the target reference data is at least one modality data in the multimodal data;

[0255] Problem processing module 506 is configured to use a problem processing model to process the target problem based on the target reference data, and obtain the problem processing result corresponding to the target problem.

[0256] Optionally, the multimodal problem encoding is a first modal problem encoding and a second modal problem encoding;

[0257] The data determination module 504 is further configured to:

[0258] First modal data and second modal data are determined from the multimodal data, wherein the second modal data is data generated using the first modal data that has a different modality from the first modal data;

[0259] Using the first modal problem coding, first modal reference data is determined from the first modal data, and using the second modal problem coding, second modal reference data is determined from the second modal data;

[0260] Based on the first modal reference data and the second modal reference data, the target reference data corresponding to the target problem is determined.

[0261] Optionally, the data determination module 504 is further configured to:

[0262] The first modal data is encoded using a first modal coding model to obtain a first modal code, and the second modal data is encoded using a second modal coding model to obtain a second modal code.

[0263] The similarity between the first modal problem code and the first modal code is calculated to obtain a first similarity between the first modal problem code and the first modal code, and based on the first similarity, first modal reference data is determined from the first modal data;

[0264] The similarity between the second modal problem code and the second modal code is calculated to obtain a second similarity between the second modal problem code and the second modal code. Based on the second similarity, second modal reference data is determined from the second modal data.

[0265] Optionally, there may be multiple first modal reference data.

[0266] The data determination module 504 is further configured to:

[0267] Determine target first modal reference data from multiple first modal reference data, wherein the target first modal reference data is the first modal reference data used to generate each second modal reference data;

[0268] The target first mode reference data is determined as the target reference data.

[0269] Optionally, the first modal problem encoding is an image modal problem encoding, the second modal problem encoding is a text modal problem encoding, the first modal data is a target document image, and the second modal data is target image text;

[0270] The encoding module 502 is further configured to:

[0271] The target problem is encoded using an image modality coding module to obtain the image modality problem code, and the target problem is encoded using a text modality coding module to obtain the text modality problem code.

[0272] The data determination module 504 is further configured to:

[0273] The first modality reference data is determined from the target document image using the image modality problem coding, and the second modality reference data is determined from the target image text using the text modality problem coding.

[0274] Optionally, the encoding module 502 is further configured to:

[0275] From multiple target document files, at least one page of document is determined in each target document file, and the multiple pages of document are converted into images to obtain multiple target document images, wherein the multiple target document files contain information for processing the target problem;

[0276] Text extraction is performed on the multiple target document images to obtain multiple target image texts;

[0277] The plurality of target document images and the plurality of target image texts are determined as the multimodal data corresponding to the target question.

[0278] Optionally, the problem processing module 506 is further configured to:

[0279] The target reference data and the target problem are input into the problem processing model, and in the problem processing model, the associated reference data corresponding to the target problem is determined from the target reference data;

[0280] The target problem is processed using the associated reference data to obtain the corresponding problem processing result.

[0281] Optionally, the target reference data may be multiple target document images;

[0282] The problem handling module 506 is also configured to:

[0283] The multiple target document images are stitched together to obtain the model input document image, and the model input document image and the target question are input into the problem processing model.

[0284] In the problem processing model, associated document images corresponding to the target problem are determined from the document images input to the model.

[0285] Optionally, the problem processing module 506 is further configured to:

[0286] Adjust the images of each target document according to the preset image size to obtain multiple adjusted document images;

[0287] Multiple adjusted document images are stitched together to obtain a stitched document image. The stitched document image is then adjusted according to the input image size of the problem processing model to obtain the model input document image.

[0288] Optionally, the problem processing module 506 is further configured to:

[0289] From the input document image of the model, determine the document image region corresponding to each target document image, and extract features from the target document image region to obtain the target document image region features, wherein the target document image region is any one of multiple document image regions;

[0290] Determine the feature similarity between the features of the target document image region and the features of the question corresponding to the target question, and based on the feature similarity, determine the associated document image region corresponding to the target question from the plurality of document image regions;

[0291] The associated document image region is identified as the associated document image.

[0292] Optionally, the problem processing module 506 is further configured to:

[0293] Determine the problem features corresponding to the target problem, and calculate the feature similarity between the target document image region features and the problem features;

[0294] The maximum feature similarity is determined from multiple feature similarities, and the document image region corresponding to the maximum feature similarity is determined from the multiple document image regions;

[0295] The document image region corresponding to the maximum feature similarity is determined as the associated document image region corresponding to the target question.

[0296] Optionally, the problem processing module 506 is further configured to:

[0297] Based on the region features of the associated document image and the target document image, the answer to the target question is inferred to obtain the answer to the target question.

[0298] This specification provides one or more embodiments of a data processing apparatus. During data processing, this apparatus can perform multimodal encoding on a target problem to obtain a multimodal problem code. Using this multimodal problem code, it determines at least one modality of target reference data corresponding to the target problem from the multimodal data. This target reference data can be external reference knowledge referenced by the problem processing model during the processing of the target problem. Then, using the problem processing model, the target problem is processed based on the target reference data of at least one modality, thereby obtaining an accurate problem processing result. This avoids the problem of inaccurate data processing results caused by the limited knowledge learned by the neural network model.

[0299] The above is an illustrative scheme of a data processing apparatus according to this embodiment. It should be noted that the technical solution of this data processing apparatus and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing apparatus, please refer to the description of the technical solution of the data processing method described above.

[0300] Figure 6 A structural block diagram of a computing device 600 provided according to one embodiment of this specification is shown.

[0301] The computing device 600 includes:

[0302] Memory 610 and processor 620;

[0303] The memory 610 is used to store computer programs / instructions, and the processor 620 is used to execute the computer programs / instructions, which, when executed by the processor 620, implement the steps of the data processing method.

[0304] In one or more embodiments of this specification, the computing device can be understood as an integrated smart terminal, including but not limited to a server, desktop computer, PC (Personal Computer), all-in-one model machine, mobile phone, tablet computer or other portable smart terminal, etc., and the computing device may have the model described in the above embodiments of this application pre-installed.

[0305] Specifically, this computing device can pre-install various types of models, including but not limited to models in natural language processing, visual processing, speech processing, code processing, and multimodal task processing, thus providing diverse model selection. In different product forms, this computing device can support one or more model usage methods, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference, and application. In some product forms, this computing device also supports model management, including but not limited to multi-type model management (supporting the management of discriminative, generative, and other model types), model version control (supporting the control of different model versions), and model evaluation (evaluating model performance and effectiveness based on model evaluation tools). In other product forms, this computing device can also create applications based on models, providing API (Application Programming Interface) calling capabilities. Users can call models into created applications through the API interface, and application management tools are also provided for application management and monitoring.

[0306] Furthermore, the computing device may also include data management (supporting the creation and management of model tuning datasets), a training center (providing abundant training resources to help users learn and master AI (Artificial Intelligence) technology), and basic control capabilities (providing enterprise-level basic control capabilities to ensure the security and efficient operation of the system). Through the above functions, it provides a comprehensive and integrated device for AI development, training, deployment, and application.

[0307] Figure 7 A structural block diagram of an electronic device 700 provided according to one embodiment of this specification is shown.

[0308] A memory 710 and a processor 720 are connected via a bus 730;

[0309] The memory 710 is used to store computer programs / instructions, and the processor 720 is used to execute the computer programs / instructions, which, when executed by the processor 720, implement the steps of the method.

[0310] Specifically, the components of the electronic device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and the database 750 is used to store data.

[0311] Electronic device 700 also includes access device 740, which enables electronic device 700 to communicate via one or more networks 760. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. Access device 740 may include one or more of any type of wired or wireless network interface (e.g., network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, Wi-MAX (Worldwide Interoperability for Microwave Access) interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth interface, Near Field Communication (NFC) interface, and so on.

[0312] In one embodiment of this specification, the above-described components of the electronic device 700 and Figure 7 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 7 The block diagram of the electronic device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0313] Electronic device 700 can be any type of stationary or mobile electronic device, including mobile computers or mobile electronic devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable electronic devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary electronic devices such as desktop computers or personal computers (PCs). Electronic device 700 can also be a mobile or stationary server.

[0314] The above is an illustrative scheme of an electronic device according to this embodiment. It should be noted that the technical solution of this electronic device and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the electronic device, please refer to the description of the technical solution of the data processing method described above.

[0315] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0316] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computer-readable storage medium embodiments are basically similar to the data processing method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the data processing method embodiments.

[0317] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0318] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the data processing method described above.

[0319] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0320] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0321] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0322] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0323] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method, comprising: The target problem is subjected to multimodal coding to obtain a multimodal problem code, and the multimodal data corresponding to the target problem is determined. Based on the multimodal problem encoding, target reference data corresponding to the target problem is determined from the multimodal data, wherein the target reference data is at least one modality data in the multimodal data; Using a problem-solving model, the target problem is processed based on the target reference data to obtain the problem-solving result corresponding to the target problem.

2. The data processing method according to claim 1, wherein the multimodal problem coding is a first modal problem coding and a second modal problem coding; The step of determining the target reference data corresponding to the target problem from the multimodal data based on the multimodal problem encoding includes: First modal data and second modal data are determined from the multimodal data, wherein the second modal data is data generated using the first modal data that has a different modality from the first modal data; Using the first modal problem coding, first modal reference data is determined from the first modal data, and using the second modal problem coding, second modal reference data is determined from the second modal data; Based on the first modal reference data and the second modal reference data, the target reference data corresponding to the target problem is determined.

3. The data processing method according to claim 2, wherein determining first modal reference data from the first modal data using the first modal problem coding, and determining second modal reference data from the second modal data using the second modal problem coding, comprises: The first modal data is encoded using a first modal coding model to obtain a first modal code, and the second modal data is encoded using a second modal coding model to obtain a second modal code. The similarity between the first modal problem code and the first modal code is calculated to obtain a first similarity between the first modal problem code and the first modal code, and based on the first similarity, first modal reference data is determined from the first modal data; The similarity between the second modal problem code and the second modal code is calculated to obtain a second similarity between the second modal problem code and the second modal code. Based on the second similarity, second modal reference data is determined from the second modal data.

4. The data processing method according to claim 2, wherein there are multiple first modal reference data; The step of determining the target reference data corresponding to the target problem based on the first modal reference data and the second modal reference data includes: Determine target first modal reference data from multiple first modal reference data, wherein the target first modal reference data is the first modal reference data used to generate each second modal reference data; The target first mode reference data is determined as the target reference data.

5. The data processing method according to claim 2, wherein the first modal problem encoding is image modal problem encoding, the second modal problem encoding is text modal problem encoding, the first modal data is a target document image, and the second modal data is target image text; The process of performing multimodal coding on the target problem to obtain a multimodal problem code includes: The target problem is encoded using an image modality coding module to obtain the image modality problem code, and the target problem is encoded using a text modality coding module to obtain the text modality problem code. The step of determining first modal reference data from the first modal data using the first modal problem coding, and determining second modal reference data from the second modal data using the second modal problem coding, includes: The first modality reference data is determined from the target document image using the image modality problem coding, and the second modality reference data is determined from the target image text using the text modality problem coding.

6. The data processing method according to any one of claims 1 to 5, wherein determining the multimodal data corresponding to the target problem includes: From multiple target document files, at least one page of document is determined in each target document file, and the multiple pages of document are converted into images to obtain multiple target document images, wherein the multiple target document files contain information for processing the target problem; Text extraction is performed on the multiple target document images to obtain multiple target image texts; The plurality of target document images and the plurality of target image texts are determined as the multimodal data corresponding to the target question.

7. The data processing method according to any one of claims 1 to 5, wherein the step of using a problem processing model to process the target problem based on the target reference data to obtain a problem processing result corresponding to the target problem includes: The target reference data and the target problem are input into the problem processing model, and in the problem processing model, the associated reference data corresponding to the target problem is determined from the target reference data; The target problem is processed using the associated reference data to obtain the corresponding problem processing result.

8. The data processing method according to claim 7, wherein the target reference data comprises multiple target document images; The step of inputting the target reference data and the target problem into the problem processing model, and determining the associated reference data corresponding to the target problem from the target reference data in the problem processing model, includes: The multiple target document images are stitched together to obtain the model input document image, and the model input document image and the target question are input into the problem processing model. In the problem processing model, associated document images corresponding to the target problem are determined from the document images input to the model.

9. The data processing method according to claim 8, wherein the step of stitching together the plurality of target document images to obtain the model input document image includes: Adjust the images of each target document according to the preset image size to obtain multiple adjusted document images; Multiple adjusted document images are stitched together to obtain a stitched document image. The stitched document image is then adjusted according to the input image size of the problem processing model to obtain the model input document image.

10. The data processing method according to claim 8, wherein determining the associated document image corresponding to the target question from the plurality of target document images comprises: From the input document image of the model, determine the document image region corresponding to each target document image, and extract features from the target document image region to obtain the target document image region features, wherein the target document image region is any one of multiple document image regions; Determine the feature similarity between the features of the target document image region and the features of the question corresponding to the target question, and based on the feature similarity, determine the associated document image region corresponding to the target question from the plurality of document image regions; The associated document image region is identified as the associated document image.

11. The data processing method according to claim 10, wherein determining the feature similarity between the features of the target document image region and the features of the question corresponding to the target question, and determining the associated document image region corresponding to the target question from the plurality of document image regions based on the feature similarity, comprises: Determine the problem features corresponding to the target problem, and calculate the feature similarity between the target document image region features and the problem features; The maximum feature similarity is determined from multiple feature similarities, and the document image region corresponding to the maximum feature similarity is determined from the multiple document image regions; The document image region corresponding to the maximum feature similarity is determined as the associated document image region corresponding to the target question.

12. The data processing method according to claim 10, wherein the step of using the associated reference data to process the target problem and obtain the problem processing result corresponding to the target problem includes: Based on the region features of the associated document image and the target document image, the answer to the target question is inferred to obtain the answer to the target question.

13. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 12.

14. An electronic device comprising: A memory and a processor, the memory and the processor being connected via a bus; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 12.

15. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 12.

16. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 12.

Images