Data processing method, data processing model training method, and data processing system

Abstract

Provided in the present disclosure are a data processing method, a data processing model training method, and a data processing system. The data processing method comprises: determining initial visual data and prompt text associated with the initial visual data; inputting the initial visual data and the prompt text into a data processing model, and using the data processing model to perform both data segmentation and data volume compression on the initial visual data, so as to obtain a plurality of pieces of sub visual data and target visual data; on the basis of a target visual feature of the target visual data, performing feature compression on a plurality of sub visual features of the plurality of pieces of sub visual data, so as to obtain an initial visual feature of the initial visual data; and on the basis of the initial visual feature and the prompt text, obtaining the result of performing data processing on the initial visual data on the basis of the prompt text. In this way, the efficiency of data processing is improved, and the usage of a video memory in a data processing process is reduced.

Description

Data processing method, data processing model training method, and data processing system TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computers, and in particular to a data processing method, system and device, and a data processing model training method. BACKGROUND

[0002] When a scheme combining OCR (Optical Character Recognition, which refers to optical character recognition technology that can automatically recognize and convert the text content in an image into a text format) and a large language model (LLM, Large Language Model) is used for document image analysis, a large amount of visual information (such as icon, layout, etc.) will be lost; using a multimodal large model that does not rely on OCR to directly understand the document image can both recognize the text in the picture and understand the visual information in the picture.

[0003] However, in a multimodal scene including text data and image data, the text data is usually short and highly abstract, while the image data is the opposite, which can form visual features with rich information and longer length. That is, the existing multimodal large model generates too many visual features when encoding the document image, resulting in too slow inference speed and too much memory occupation; this makes it difficult for these multimodal large models to be used in understanding scenes of a whole document containing multiple pages. SUMMARY

[0004] Therefore, the present disclosure provides a data processing method and a data processing model training method. One or more embodiments of the present disclosure also relate to a data processing system, a data processing device, a computing device, a computer-readable storage medium, and a computer program product, to solve the technical defects in the prior art caused by the image data containing too many visual features, resulting in too slow inference speed and too much memory occupation.

[0005] According to a first aspect of the present disclosure, a data processing method is provided, comprising:

[0006] determining initial visual data and prompt text associated with the initial visual data;

[0007] inputting the initial visual data and the prompt text into a data processing model, and using the data processing model to respectively perform data cutting and data volume compression on the initial visual data to obtain a plurality of sub-visual data and target visual data;

[0008] According to a target visual feature of the target visual data, a plurality of sub-visual features of the plurality of sub-visual data are compressed to obtain an initial visual feature of the initial visual data;

[0009] According to the initial visual feature and the prompt text, a data processing result of the initial visual data based on the prompt text is obtained.

[0010] According to a second aspect of the present disclosure, a data processing model training method comprises:

[0011] An initial visual sample data, a prompt text sample associated with the initial visual sample data, and a target data processing result are determined, wherein the target data processing result is a data processing result of the initial visual sample data based on the prompt text sample;

[0012] The initial visual sample data and the prompt text sample are input into an initial data processing model, and the initial data processing model is used to respectively perform data cutting and data volume compression on the initial visual sample data to obtain a plurality of sub-visual sample data and a target visual sample data;

[0013] According to a target visual sample feature of the target visual sample data, a plurality of sub-visual sample features of the plurality of sub-visual sample data are compressed to obtain an initial visual sample feature of the initial visual sample data;

[0014] According to the initial visual sample feature and the prompt text sample, a predicted data processing result of the initial visual sample data based on the prompt text sample is obtained.

[0015] According to the predicted data processing result and the target data processing result, the initial data processing model is trained to obtain a data processing model.

[0016] According to a third aspect of the present disclosure, a data processing system is provided, comprising one or more computers and one or more storage devices, the one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement a data processing model, the data processing model comprising:

[0017] A first data processing network configured to perform data cutting and data volume compression on determined initial data to obtain a plurality of sub-data and target data,

[0018] According to a target feature of the target data, a plurality of sub-features of the plurality of sub-data are compressed to obtain an initial feature of the initial data;

[0019] a second data processing network configured to obtain a data processing result of the initial data based on the prompt data according to the initial feature and the determined prompt data of the initial data association.

[0020] According to a fourth aspect of the present disclosure, a data processing apparatus is provided, comprising:

[0021] a determining module configured to determine initial visual data and prompt text associated with the initial visual data;

[0022] a processing module configured to input the initial visual data and the prompt text into a data processing model, and perform data cutting and data volume compression on the initial visual data respectively by using the data processing model to obtain a plurality of sub-visual data and target visual data;

[0023] a compression module configured to perform feature compression on a plurality of sub-visual features of the plurality of sub-visual data according to a target visual feature of the target visual data to obtain an initial visual feature of the initial visual data;

[0024] an obtaining module configured to obtain a data processing result of the initial visual data based on the prompt text according to the initial visual feature and the prompt text.

[0025] According to a fifth aspect of the present disclosure, a computing device is provided, comprising:

[0026] a memory and a processor;

[0027] the memory is configured to store computer programs / instructions, and the processor is configured to execute the computer programs / instructions, which realize the steps of the above-mentioned data processing method and data processing model training method when executed by the processor.

[0028] According to a sixth aspect of the present disclosure, a computer readable storage medium is provided, which stores computer programs / instructions, which realize the steps of the above-mentioned data processing method and data processing model training method when executed by the processor.

[0029] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising computer programs / instructions, which realize the steps of the above-mentioned data processing method and data processing model training method when executed by the processor.

[0030] An embodiment of the present specification provides a data processing method. The data processing method utilizes a data processing model to process initial visual data to obtain corresponding sub-visual data and target visual data. Since the initial visual data includes multiple sub-visual data, the multiple sub-visual features of the multiple sub-visual data contain detailed visual information of the initial visual data, and the target visual data contains global visual information of the initial visual data. In the case of feature compression of the multiple sub-visual features according to the target visual feature, the initial visual feature fused with global and local features is obtained. The initial visual feature contains global context information and detailed information from each sub-visual data, and the number of the initial visual feature is less than the number of the multiple sub-visual features. According to the initial visual feature and the prompt text, a data processing result of the initial visual data based on the prompt text is obtained. By obtaining the compressed initial visual feature with reduced number, the inference speed of the data processing model is improved, that is, the processing efficiency of data processing is improved. Generally, the visual data is processed by the video memory. By processing the compressed initial visual feature, the video memory occupation in the data processing process is reduced, thereby being suitable for more multi-modal scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0031] FIG. 1 is a scene schematic diagram of a data processing method according to an embodiment of the present specification;

[0032] FIG. 2 is a flowchart of a data processing method according to an embodiment of the present specification;

[0033] FIG. 3 is a processing process schematic diagram of feature aggregation processing according to an embodiment of the present specification;

[0034] FIG. 4 is a processing process flowchart of a data processing method applied to a multi-page document parsing scenario according to an embodiment of the present specification;

[0035] FIG. 5 is a flowchart of a data processing model training method according to an embodiment of the present specification;

[0036] FIG. 6 is a structural schematic diagram of a data processing system according to an embodiment of the present specification;

[0037] FIG. 7 is a structural schematic diagram of a data processing apparatus according to an embodiment of the present specification;

[0038] FIG. 8 is a structural block diagram of a computing device according to an embodiment of the present specification. DETAILED DESCRIPTION

[0039] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present description. However, the present description can be practiced without the specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to obscure the present description. Some portions of the following description are presented in terms of algorithms, symbols, or information flow representations of operations on data streams within a computer memory. These algorithmic descriptions and representations are used by those skilled in the art to convey the substance of their work efficiently. A programmer, as is well-understood by those skilled in the art, only needs to implement the substance of the algorithm in their own programming language in order to fully and effectively carry out the operations.

[0040] The terminology used in this description is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments of the present description. As used in this description and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0041] It will be understood that, although the terms first, second, etc. can be used herein to describe various information, these terms are not intended to denote a temporal or chronological order. Rather, these terms are used only as a shorthand notation to first, second, etc. information. For example, a first item discussed below could be termed a second item, and, similarly, a second item could be termed a first item without departing from the scope of the present description. As used herein, the term "if' can be construed to mean "when" or "upon" or "in response to determining" terms that indicate a logical relationship between an act and an event or a condition. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments of the present description.

[0042] In addition, it should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in one or more embodiments of the present description are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0043] In one or more embodiments of the present description, a large model refers to a deep learning model with a large number of model parameters, usually containing hundreds of millions, tens of billions, hundreds of billions, thousands of billions or even tens of billions of model parameters. The large model can also be called a foundation model. Through large-scale unlabeled corpus pre-training, a pre-training model with hundreds of millions of parameters is output. Such a model can adapt to a wide range of downstream tasks and has good generalization ability. For example, large language models, multi-modal pre-training models, etc.

[0044] In practical applications, a large model can be applied to different tasks by fine-tuning a pre-trained model with a small amount of samples. The large model can be widely applied in natural language processing (NLP) and computer vision, and can be applied to computer vision tasks such as visual question answering (VQA), image captioning (IC), image generation, and natural language processing tasks such as text-based sentiment classification, text summarization generation, and machine translation. The main application scenarios of the large model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, and the like.

[0045] First, the nomenclature related to one or more embodiments of the present specification is explained.

[0046] Token: a unit of text or visual input encoded by a language model.

[0047] Cross Attention: Cross Attention layer, one of the commonly used deep learning networks.

[0048] Document image: A document image generally refers to a digital image of a paper document obtained by a scanner, a camera or other image capture device, or a digital image of various electronic documents such as book pages, web documents, newspaper articles, tables, etc.; The document image usually contains not only text information but also rich visual information.

[0049] Using LLM for multi-page document understanding will cause a large amount of token consumption, whether for existing OCR combined with LLM solutions or multi-modal large model solutions; The existing multi-modal large model occupies a large number of visual tokens when understanding a single image, and causes large memory occupation and slow inference time when understanding multiple pages, and has weak multi-page document understanding capability.

[0050] Existing solutions include: filtering valuable visual features by calculating feature similarity to reduce visual features, and integrating these visual features with original features, but the valuable visual features filtered may not cover all areas of the image, resulting in loss of text information.

[0051] The data processing method provided by the present disclosure implements compression of visual features of a document image by using global visual features of the document image as a guide to ensure that information in all areas is compressed and retained, greatly reducing the number of visual features while retaining most of the information.

[0052] In the specification, a data processing method and a data processing model training method are provided. The specification also relates to a data processing system, a data processing device, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail in the following embodiments.

[0053] Referring to FIG. 1, FIG. 1 shows a scenario diagram of a data processing method according to an embodiment of the specification.

[0054] Specifically, considering that the model parameter amount of a large model is large and the computing resources of a mobile terminal are limited, the data processing method provided in the embodiments of the present application can be applied to the application scenario shown in FIG. 1, but is not limited thereto. In the application scenario shown in FIG. 1, the large model is deployed in a server 104, which can be connected to one or more end-side devices 102 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The end-side devices 102 can include, but are not limited to, smartphones, tablet computers, notebook computers, palm computers, personal computers, smart home devices, vehicle-mounted devices, etc. The end-side devices 102 can interact with users through a graphical user interface to call the large model and implement the method provided in the present disclosure.

[0055] In the present disclosure, the system composed of the end-side devices 102 and the server 104 can perform the following steps: the end-side device 102 is configured to send initial visual data and prompt text associated with the initial visual data to the server 104; the server 104 is configured to receive the initial visual data and the prompt text sent by the client, input the initial visual data and the prompt text into a data processing model, use the data processing model to respectively perform data cutting and data volume compression on the initial visual data, obtain a plurality of sub-visual data and target visual data; determine a plurality of sub-visual features of the plurality of sub-visual data and a target visual feature of the target visual data; perform feature compression on the plurality of sub-visual features according to the target visual feature to obtain an initial visual feature of the initial visual data; and obtain a data processing result of the initial visual data based on the prompt text according to the initial visual feature and the prompt text.

[0056] It should be noted that in the case where the running resources of the end-side device can meet the deployment and running conditions of the large model, the embodiments of the present application can be performed in the end-side device.

[0057] The data processing method provided by the present disclosure utilizes a data processing model to perform data processing on input initial visual data, to obtain corresponding multiple sub-visual data and target visual data. Since the initial visual data includes multiple sub-visual data, multiple sub-visual features of the multiple sub-visual data contain detailed visual information of the initial visual data, and the target visual data contains global visual information of the initial visual data. In the case of performing feature compression on the multiple sub-visual features according to a target visual feature, an initial visual feature that fuses global and local features is obtained. The initial visual feature contains global context information and detailed information from each sub-visual data, and the number of the initial visual feature is less than the number of the multiple sub-visual features. According to the initial visual feature and prompt text, a data processing result of the initial visual data based on the prompt text is obtained. By obtaining the compressed initial visual feature with reduced number, the inference speed of the data processing model is improved, that is, the processing efficiency of data processing is improved. Generally, the visual data is processed by the video memory. By processing the compressed initial visual feature, the video memory occupation in the data processing process can be reduced, and the method can be applied to more multi-modal scenarios.

[0058] Referring to FIG. 2, FIG. 2 shows a flowchart of a data processing method according to an embodiment of the present disclosure, which specifically includes the following steps.

[0059] The data processing method can be applied to a data processing system including one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement a data processing model.

[0060] Step 202: determining initial visual data and prompt text associated with the initial visual data.

[0061] The initial visual data can be understood as an image with rich text information. The initial visual data includes but is not limited to scanned images of document pages, charts, web screenshots, natural images with text, and video frames, etc. The initial visual data is a high-resolution image, that is, the initial visual data has more pixels, so it can display more details and finer image quality. For example, the initial visual data is a 3840x2160 pixel picture. The prompt text can be understood as an input text used to guide the data processing model to generate a specific type of output.

[0062] In the present disclosure, the prompt text is associated with the initial visual data to obtain a data processing result related to the initial visual data according to the prompt text. For example, when the initial visual data includes a document image of a paper document, the prompt text can be "analyze the abstract part of the paper". According to the prompt text, the parsed text of the abstract part of the paper is obtained.

[0063] In actual application, a user can upload initial visual data and input prompt text associated with the initial visual data to a server through a client, and the server receives the initial visual data and the prompt text. The specific implementation is described as follows:

[0064] The determination of the initial visual data and the prompt text associated with the initial visual data includes:

[0065] The initial visual data and the prompt text associated with the initial visual data sent by the client are received, wherein the initial visual data is determined through upload operation and / or input operation of a user interaction interface of the client.

[0066] In actual application, the user interaction interface of the client can include a control for uploading a document, a control for inputting a web page link, and the like. For example, the user can upload a document stored locally on the client through a click operation on the control for uploading the document, and the document image of the document is the initial visual data. Alternatively, the user can determine a corresponding web page through a click operation on the control for inputting the web page link, and the web page content of the web page is the initial visual data.

[0067] The data processing method provided by the present disclosure realizes that the server receives the initial visual data and the prompt text sent by the client through the interaction operation of the client and the server. The user interaction interface of the client can greatly facilitate the user interaction operation and improve the user experience.

[0068] Step 204: inputting the initial visual data and the prompt text into a data processing model, and performing data cutting and data volume compression on the initial visual data by using the data processing model to obtain a plurality of sub-visual data and target visual data.

[0069] The initial visual data includes the plurality of sub-visual data, that is, the plurality of sub-visual data constitutes the initial visual data. In the case where the initial visual data is an image, the plurality of sub-visual data can be understood as a plurality of sub-images of the image without overlapping parts. The resolution of the initial visual data is greater than the resolution of the target visual data. In the present disclosure, data volume compression is realized by reducing the resolution of the visual data, that is, the target visual data can be understood as visual data with reduced resolution of the initial visual data.

[0070] The data processing model can be understood as a multi-modal large model capable of processing data in visual and text modalities. The data processing model is configured to perform corresponding data processing on initial visual data based on input prompt text to obtain a data processing result. For example, when the prompt text is used to analyze the initial visual data, the analysis text for the initial visual data is obtained.

[0071] Specifically, when the initial visual data and the prompt text are obtained, the initial visual data and the prompt text are input into the data processing model. The data processing model performs corresponding data processing on the initial visual data based on the prompt text. First, the data processing model can perform data cutting and data volume compression on the initial visual data respectively to obtain a plurality of sub-visual data and target visual data.

[0072] In one or more embodiments of the present specification, a plurality of sub-visual data is obtained by cutting the initial visual data, and target visual data is obtained by performing data volume compression on the initial visual data. The specific implementation is described as follows:

[0073] The data processing model includes a visual data processing network, and the visual data processing network includes a data preprocessing unit.

[0074] The data processing model respectively performs data cutting and data volume compression on the initial visual data to obtain a plurality of sub-visual data and target visual data, including:

[0075] The data preprocessing unit is used to cut the initial visual data to obtain the plurality of sub-visual data;

[0076] The data preprocessing unit is used to perform data volume compression on the initial visual data to obtain the target visual data.

[0077] The data preprocessing unit is configured to process the initial visual data so that the processed data can be processed by the data processing model. The number of the plurality of sub-visual data can be set according to actual conditions, for example, four sub-visual data or six sub-visual data are obtained by cutting the initial visual data. The target visual data is obtained by performing data volume compression on the initial visual data. The target visual data is visual data containing global information, for example, an image with a resolution of 3840x2160 pixels is reduced to an image with a resolution of 504x504 pixels, or an image with a resolution of 1920x1080 pixels is reduced to an image with a resolution of 256x256 pixels.

[0078] Specifically, the initial visual data is cut into multiple sub-visual data by the data preprocessing unit, for example, a document image is cut into 6 sub-images, and the initial visual data can be reduced to obtain target visual data, that is, the initial visual data is reduced to a low-resolution global image, for example, an image with a resolution of 3840x2160 pixels is reduced to an image with a resolution of 504x504 pixels.

[0079] In a specific implementation, the resolution of the initial visual data is changed by adjusting the size of the initial visual data, because a high-resolution image means that the image contains a large number of pixel points, each of which may carry visual information such as color, brightness, texture, etc. With the increase of the resolution, the details in the image are more abundant, and more visual features are needed to describe these details. Therefore, the smaller the resolution of the initial visual data is, the fewer the extracted visual features are, and the smaller the data amount is. Two implementation manners of adjusting the size of the initial visual data to change the resolution are provided in the disclosure. One manner is to change the size of the initial visual data by cutting the initial visual data, and the local information of the initial visual data is contained in the multiple sub-visual data obtained by cutting. Another manner is to directly reduce the initial visual data, and the global information of the initial visual data is contained in the target visual data obtained by reducing.

[0080] Of course, in actual application, the data preprocessing unit can be arranged outside the data processing model, that is, after receiving the initial visual data, the data preprocessing unit is used to cut and compress the data amount of the initial visual data respectively to obtain multiple sub-visual data and target visual data, and then the multiple sub-visual data and the target visual data are input into the data processing model.

[0081] The data processing method provided by the disclosure obtains multiple sub-visual data and target visual data by cutting and compressing the data amount of the initial visual data by the data preprocessing unit, which provides data support for subsequent data processing of the data processing model.

[0082] In one or more embodiments of the disclosure, feature extraction is performed on the multiple sub-visual data in the data processing model to obtain multiple sub-visual features of the multiple sub-visual data, and feature extraction is performed on the target visual data to obtain target visual features. Specifically, the data encoding unit of the visual data processing network is used to encode the sub-visual data and the target visual data to obtain the sub-visual features and the target visual features. The specific implementation is as follows:

[0083] The data processing model includes a visual data processing network, and the visual data processing network includes a data encoding unit.

[0084] After the plurality of sub-visual data and the target visual data are obtained, the method further includes:

[0085] The plurality of sub-visual data are encoded by using the data encoding unit to obtain sub-visual features of each sub-visual data in the plurality of sub-visual data.

[0086] The target visual data are encoded by using the data encoding unit to obtain target visual features of the target visual data.

[0087] The visual data processing network can be understood as a network in the data processing model configured to process the initial visual data, and the output of the visual data processing network is the initial visual features of the initial visual data. The data encoding unit can be understood as a network unit of an encoder structure, configured to encode the visual data to obtain corresponding visual features.

[0088] Specifically, the plurality of sub-visual data are encoded by using the data encoding unit to obtain sub-visual features of each sub-visual data in the plurality of sub-visual data. Each sub-visual feature contains detailed visual information of a local part of the initial visual data. The target visual data are encoded by using the data encoding unit to obtain target visual features of the target visual data. The target visual features contain global visual information of the initial visual data.

[0089] The data processing method provided by the present disclosure extracts a plurality of sub-visual features and target visual features of higher dimensions through the encoding of the plurality of sub-visual data and the target visual data by the data encoding unit. The plurality of sub-visual features can better capture detailed information of the initial visual data, and the target visual features can better capture global layout information of the initial visual data.

[0090] Step 206: According to the target visual features of the target visual data, the plurality of sub-visual features of the plurality of sub-visual data are compressed to obtain the initial visual features of the initial visual data.

[0091] Specifically, in the case where the plurality of sub-visual features and the target visual features are obtained, the initial visual features with a smaller number of visual features are obtained by compressing the plurality of sub-visual features. When the initial visual features and the text features of the prompt text are used for result prediction, the inference speed of the data processing model can be improved, and the data processing efficiency can be improved.

[0092] In one or more embodiments of the present specification, the data compression unit of the visual data processing network performs initial compression processing on each of the plurality of sub-visual features and the target visual feature, and then performs target compression processing on the compressed sub-visual features using the compressed target visual feature obtained by compression, to obtain the initial visual feature of the initial visual data. The specific implementation is described as follows:

[0093] The data processing model comprises a visual data processing network, and the visual data processing network comprises a data compression unit;

[0094] The feature compression of the plurality of sub-visual features of the plurality of sub-visual data according to the target visual feature of the target visual data to obtain the initial visual feature of the initial visual data comprises:

[0095] The data compression unit performs initial compression processing on each of the plurality of sub-visual features of the plurality of sub-visual data and the target visual feature of the target visual data, to obtain a plurality of compressed sub-visual features and a compressed target visual feature;

[0096] The target compression processing of the plurality of compressed sub-visual features according to the compressed target visual feature to obtain the initial visual feature of the initial visual data.

[0097] Specifically, the initial compression processing can be understood as feature aggregation, and the feature aggregation is performed on each sub-visual feature and the target visual feature to realize compression of a single visual feature; the target compression processing can be understood as feature compression, and the feature compression is performed on the plurality of compressed sub-visual features using the compressed target visual feature after the feature aggregation to realize compression in the number of visual features.

[0098] In actual application, the data compression unit comprises a feature aggregation module; the feature aggregation module is used to perform feature aggregation on each sub-visual feature and the target visual feature to obtain the plurality of compressed sub-visual features and the compressed target visual feature. The specific implementation is described as follows:

[0099] The data compression unit performs initial compression processing on each of the plurality of sub-visual features of the plurality of sub-visual data and the target visual feature of the target visual data, to obtain a plurality of compressed sub-visual features and a compressed target visual feature, comprising:

[0100] The feature aggregation module is used to perform feature aggregation processing on each of the plurality of sub-visual features, to obtain the plurality of compressed sub-visual features;

[0101] The feature aggregation module is used to perform the feature aggregation processing on the target visual feature, to obtain the compressed target visual feature.

[0102] The feature aggregation module includes a convolutional neural network, which is configured to aggregate the sub-visual features and the target visual feature in a horizontal direction, so as to reduce the number of the sub-visual features and the target visual feature.

[0103] Specifically, taking an initial visual data as a document image, 6 sub-images with a resolution of 504*504 pixels are obtained through cutting processing, and a global image with a resolution of 504*504 pixels is obtained through downsizing processing.

[0104] The data encoding unit encodes the 6 sub-images and the global image respectively, and obtains 1296 sub-visual features for each sub-image and 1296 target visual features for the global image. Since most of the texts in the document image are organized from left to right and from top to bottom, the text semantics of the document image in the horizontal direction is always coherent. In order to maintain the visual structure information while reducing the visual features of the sub-images and the global image, the feature aggregation module can use a convolutional kernel with a size of 1*4 and a step of 4 to integrate 4 visual features in the horizontal direction, and obtain compressed sub-visual features and compressed target visual features.

[0105] As shown in FIG. 3, FIG. 3 shows a processing process diagram of a feature aggregation process according to an embodiment of the present specification. Through the feature aggregation process of the feature aggregation module, such as using a convolutional kernel with a size of 1*4 to cover 4 positions (i.e. 4 small boxes in the feature map of FIG. 3) in the horizontal direction of the feature map, the features corresponding to the 4 positions are aggregated through an aggregation operation (such as weighted summation). In this way, the number of sub-visual features and target visual features can be reduced by 4 times, that is, the number of compressed sub-visual features corresponding to the 6 sub-images and the number of compressed target visual features of the global image are reduced to 324 respectively.

[0106] In addition, it should be noted that the feature aggregation module also uses a fully connected layer to convert the dimension of the visual features, so as to align the visual features with the text feature space of the data processing model. The converted visual features can also be regarded as text features of different parts of text information in the initial visual data. Through the conversion of the dimension of the visual features by the fully connected layer, the text features of the prompt text in the same feature space and the converted visual features can be spliced in the subsequent multi-modal data processing module. The compressed target visual features contain the overall text layout information of the initial visual data, and the compressed sub-visual features capture the detailed text information of each sub-visual data.

[0107] The data processing method provided by the present disclosure reduces the number of visual features of each sub-visual feature and the target visual feature through the feature aggregation module for the feature aggregation processing of each sub-visual feature and the target visual feature, and the feature aggregation module can convert the dimension of the visual feature, so as to facilitate the alignment of the visual feature and the text feature space of the data processing model.

[0108] In one or more embodiments of the present specification, the data compression unit further comprises a feature compression module; by combining a plurality of compressed sub-visual features, a combined sub-visual feature is obtained, and the feature compression module realizes feature compression of the initial visual data by cross-attention processing of the compressed target visual feature and the combined sub-visual feature. The specific implementation is as follows:

[0109] According to the compressed target visual feature, the plurality of compressed sub-visual features are target compressed to obtain the initial visual feature of the initial visual data, comprising:

[0110] Using the feature compression module, according to the position identifier of each sub-visual data in the plurality of sub-visual data, the plurality of compressed sub-visual features are combined to obtain a combined sub-visual feature, wherein the position identifier of each sub-visual data is determined by a preset position numbering sequence and a position relationship between the sub-visual data and the initial visual data;

[0111] The cross-attention processing is performed on the compressed target visual feature and the combined sub-visual feature to obtain the initial visual feature of the initial visual data.

[0112] Wherein, the position identifier can be understood as an identifier marking the position of each sub-visual data in the initial visual data; the preset position numbering sequence can be understood as the sequence from left to right and from top to bottom.

[0113] Specifically, after cutting the initial visual data, according to the position of each sub-visual data obtained by cutting in the initial visual data, each sub-visual data is marked with a corresponding position identifier in the order from left to right and from top to bottom; in this case, according to the position identifier, each sub-visual data is reorganized to be placed in its correct position in the initial visual data, and a reorganized global map is obtained, which is consistent with the resolution of the initial visual data.

[0114] In a specific implementation, the plurality of sub-visual data correspond to a plurality of compressed sub-visual features, and the features can be represented in the form of a feature map, so that by using the position identifiers of the sub-visual data, the plurality of compressed sub-visual features can be spliced and combined according to their relative positions in the initial visual data to form a global feature map corresponding to the positions of the initial visual data, i.e., the global feature map is a combined sub-visual feature, which not only contains detailed information of the plurality of compressed sub-visual features, but also retains the spatial position relationship of the plurality of compressed sub-visual features in the initial visual data.

[0115] By performing cross-attention processing on the compressed target visual feature and the combined sub-visual feature, an initial visual feature corresponding to the initial visual data after feature compression is obtained.

[0116] The data processing method provided by the present disclosure can splice and combine the plurality of compressed sub-visual features according to the correct position relationship through the position identifiers of the sub-visual data, so that the combined sub-visual feature formed has a corresponding position relationship with the compressed target visual feature, which facilitates subsequent cross-attention processing.

[0117] In one or more embodiments of the present disclosure, the cross-attention network is used to compress the compressed target visual feature as a query and the recombined combined sub-visual feature as a compression object, so that the initial visual feature of the initial visual data will not change according to the number of cuts. The specific implementation is as follows:

[0118] The cross-attention processing on the compressed target visual feature and the combined sub-visual feature to obtain the initial visual feature of the initial visual data includes:

[0119] A query vector is determined according to the compressed target visual feature, and a key-value vector is determined according to the combined sub-visual feature;

[0120] According to the query vector and the key-value vector, an attention weight of each compressed sub-visual feature in the combined sub-visual feature is determined;

[0121] According to the attention weight, a weighted sum processing is performed on the plurality of compressed sub-visual features in the combined sub-visual feature to obtain the initial visual feature of the initial visual data.

[0122] In the above embodiment, the compressed target visual feature and each compressed sub-visual feature are obtained by feature aggregation. However, the number of compressed sub-visual features increases with the increase of the number of cuts. By increasing the number of cuts, the initial visual data is cut into more sub-visual data. Therefore, the number of initial visual features of the initial visual data is not fixed on the basis that the initial visual features are determined by the compressed target visual feature and each compressed sub-visual feature. The possibility of generating too many visual features is increased. Therefore, the initial visual features of the initial visual data can be compressed into a fixed number of visual features by processing of the cross-attention network.

[0123] Specifically, the compressed target visual feature is taken as a query, and the combined sub-visual feature is taken as a key and a value. By calculating the similarity of the query and the key, the similarity reflects the correlation or importance between the query and the combined sub-visual feature and between each compressed sub-visual feature. According to the similarity, an attention weight is assigned to each compressed sub-visual feature in the combined sub-visual feature. Therefore, the value (a plurality of compressed sub-visual features in the combined sub-visual feature) is weighted and summed according to the attention weight, the feature compression of the plurality of compressed sub-visual features is realized, and the initial visual feature of the initial visual data is obtained.

[0124] In the above example, the resolution of each of the plurality of sub-visual data obtained by data processing of the initial visual data and the target visual data is 506*506 pixels in the data preprocessing unit. Therefore, in the case that the resolution of the sub-visual data and the target visual data is fixed by data preprocessing, the number of each sub-visual feature and the target visual feature obtained by the data encoding unit is 1296. Therefore, by processing of the feature aggregation module and the feature compression module in the data compression unit, the number of the initial visual feature of the output initial visual data can be kept fixed and will not change with the different number of cuts. In the case that the compressed target visual feature (containing global information) is taken as a guide, the feature compression process can cover all areas of the initial visual data and retain more comprehensive visual information.

[0125] In one or more embodiments, the compressed target visual feature can be partitioned, and the feature compression is performed on the plurality of compressed sub-visual features corresponding to each partition target visual feature to obtain each partition initial visual feature. The initial visual feature of the initial visual data is obtained by using each partition initial visual feature. The specific implementation is described as follows:

[0126] The cross-attention processing is performed on the compressed target visual feature and the combined sub-visual feature to obtain the initial visual feature of the initial visual data, and the cross-attention processing includes:

[0127] partition the compression target visual feature to obtain a plurality of partition target visual features;

[0128] determine, according to the association relationship among the plurality of partition target visual features, the combination sub-visual features and the initial visual data, a plurality of compressed sub-visual features corresponding to each partition target visual feature from the combination sub-visual features;

[0129] determine a query vector according to the each partition target visual feature, and determine a key-value vector according to the plurality of compressed sub-visual features corresponding to the each partition target visual feature;

[0130] determine, according to the query vector and the key-value vector, an attention weight of each compressed sub-visual feature in the plurality of compressed sub-visual features corresponding to the each partition target visual feature;

[0131] perform weighted summation processing on the plurality of compressed sub-visual features corresponding to the each partition target visual feature according to the attention weight to obtain each partition initial visual feature;

[0132] obtain the initial visual feature of the initial visual data according to the each partition initial visual feature.

[0133] In actual application, in the case of feature compression based on layout information for a plurality of compressed sub-visual features, the plurality of compressed sub-visual features with higher correlation in the same layout area can be weighted and summed, such as the plurality of compressed sub-visual features corresponding to the title area part in a document image. Therefore, the compression target visual feature is partitioned to obtain a plurality of partition target visual features. Since the compression target visual feature and the combination sub-visual feature both correspond to the original initial visual data, there is a corresponding mapping relationship between the compression target visual feature and the combination sub-visual feature according to the position information of the initial visual data, that is, for each partition target visual feature, a plurality of compressed sub-visual features corresponding to each partition target visual feature can be determined from the combination sub-visual features.

[0134] Each partition target visual feature is used as a query vector, and the plurality of compressed sub-visual features corresponding to each partition target visual feature is used as a key-value vector. In the above embodiments, the partition initial visual feature of each partition target visual feature is obtained in a similar manner. For example, the compression target visual feature is divided into four regions. By combining the partition initial visual features of the four regions, the initial visual feature of the initial visual data can be determined.

[0135] The data processing method provided by the disclosure, by partitioning the compression target visual features, taking each partitioned compression target visual feature as a guide, performing feature compression on a plurality of compression sub-visual features corresponding to each partitioned target visual feature and having higher correlation, and retaining visual position-based text information.

[0136] Step 208: obtaining a data processing result of the initial visual data based on the prompt text according to the initial visual feature and the prompt text.

[0137] Specifically, for the same initial visual data, the data processing result output by the data processing model is different according to different prompt texts, and therefore the data processing result obtained by the data processing model is a data processing result of the initial visual data based on the prompt text.

[0138] For example, when the initial visual data is a table of contents page of a book, the prompt text can be "how many chapters does the book have", or the prompt text can also be "what is the book about with high probability", and the obtained data processing result is different according to different prompt texts.

[0139] In one or more embodiments, the data processing model includes a multi-modal data processing network; in the multi-modal data processing network, the initial visual feature and the prompt text are subjected to multi-modal data processing to obtain a data processing result of the initial visual data based on the prompt text.

[0140] The data processing result of the initial visual data based on the prompt text is obtained according to the initial visual feature and the prompt text, including:

[0141] The initial visual feature and the prompt text are input into the multi-modal data processing network, and the data processing result of the initial visual data based on the prompt text is obtained by using the multi-modal data processing network.

[0142] In practical applications, the multi-modal data processing network includes a feature extraction unit, a feature fusion unit, and a data prediction unit.

[0143] The initial visual feature and the prompt text are input into the multi-modal data processing network, and the data processing result of the initial visual data based on the prompt text is obtained by using the multi-modal data processing network.

[0144] The prompt text is input into the feature extraction unit, and the text feature of the prompt text is obtained by using the feature extraction unit.

[0145] The initial visual feature and the text feature are fused by using the feature fusion unit to obtain a fusion feature.

[0146] The fusion feature is predicted by using the data prediction unit to obtain the data processing result of the initial visual data based on the prompt text.

[0147] In the application, the multi-modal data processing network can be a multi-modal large language model, the text feature of the prompt text contains semantic information of the prompt text, the fusion processing can be understood as splicing processing, that is, the initial visual feature and the text feature are spliced, and the spliced feature is determined as the fusion feature. Of course, in actual application, the feature fusion can also be realized based on an attention mechanism, which is not limited herein.

[0148] Specifically, the feature extraction unit can be a network unit of an encoder structure. The feature extraction unit is used to extract features of the prompt text to obtain text features containing semantic information of the prompt text. The initial visual feature and the text feature are spliced to obtain a fusion feature containing both visual information of the initial visual data and semantic information of the prompt text. The fusion feature is used to generate a data processing result of the initial visual data based on the prompt text by using the data prediction unit.

[0149] The data processing method provided by the application realizes data processing of the initial visual data based on the prompt text input by a user through a feature extraction unit, a feature fusion unit and a data prediction unit, and accurately predicts a data processing result. In the case of the initial visual feature being a compressed visual feature, the processing efficiency of the multi-modal data processing network can be improved.

[0150] In one or more embodiments of the present application, the initial visual data includes a plurality of initial visual data, and the prompt text is a first prompt text associated with the plurality of initial visual data. In this case, the data processing result of each initial visual data based on the first prompt text can be obtained according to the first prompt text.

[0151] The data processing result of the initial visual data based on the prompt text is obtained according to the initial visual feature and the prompt text, including:

[0152] The data processing result of each initial visual data based on the first prompt text is obtained according to the initial visual feature of each initial visual data in the plurality of initial visual data and the first prompt text.

[0153] In the case where the data processing method is applied to a video understanding scenario, the video includes multiple video frames, and the multiple initial visual data can be understood as the multiple video frames. In the case where the data processing method is applied to a multi-page document understanding scenario, the multiple initial visual data can be understood as multi-page document images. The data processing method is described in detail below by taking the multiple initial visual data as multi-page document images as an example.

[0154] Specifically, in the case where the multiple initial visual data are multi-page document images, the first prompt text can be "analyze the content of the multi-page document". The initial visual data of each document image in the multi-page document images is obtained by using the visual data processing network of the data processing model. The initial visual data of each document image and the first prompt text are input into the multi-modal data processing network. Based on the first prompt text, the analysis text of each document image is obtained.

[0155] In actual applications, because the data processing method provided by the present disclosure can reduce the number of visual features of the initial visual data and obtain a fixed number of initial visual features, the data processing method can be better applied to scenarios of data processing on multiple initial visual data, and the problems of slow inference speed and large memory occupation of the data processing model caused by too many visual features of a single initial visual data can be avoided.

[0156] The data processing method provided by the present disclosure can obtain initial visual features of each initial visual data in multiple initial visual data. In the case where the initial visual features are compressed features, the problem of slow data processing speed is avoided, and the practicality of the data processing model is improved.

[0157] In one or more embodiments of the present disclosure, the initial visual data includes multiple initial visual data, the prompt text is a second prompt text associated with a target initial visual data in the multiple initial visual data, and a data processing result of the target initial visual data in the multiple initial visual data is obtained based on the second prompt text. The specific implementation is described as follows:

[0158] After determining the initial visual data and the prompt text associated with the initial visual data, the method further includes:

[0159] Determining a data sequence identifier of each initial visual data in the multiple initial visual data;

[0160] The data processing result of the initial visual data based on the prompt text is obtained according to the initial visual features and the prompt text, and includes:

[0161] According to the initial visual feature of each initial visual data in the plurality of initial visual data, the data sequence identifier of the initial visual data, and the second prompt text, a data processing result of the target visual data based on the second prompt text is obtained.

[0162] In the above example, the target visual data can be the second page of the document image, and the second prompt text can be "analyze the content of the second page of the document in the plurality of initial visual data".

[0163] Specifically, in the case where the initial visual data includes a plurality of initial visual data, a unique data sequence identifier can be determined for each initial visual data in the plurality of initial visual data. The data sequence identifier can be represented in the form of a number, a letter, or a combination of a number and a letter. Through the data sequence identifier, the initial visual data corresponding to the data sequence identifier can be determined from the plurality of initial visual data. The data sequence identifier can be assigned to each initial visual data by the arrangement order of the plurality of initial visual data. In the case where the initial visual data itself includes a page number, the page number can be determined as the data sequence identifier of the corresponding initial visual data, which is not limited herein.

[0164] In actual application, the data processing result of the target visual data based on the second prompt text is obtained according to the initial visual feature of each initial visual data in the plurality of initial visual data, the data sequence identifier of the initial visual data, and the second prompt text, including:

[0165] According to the data sequence identifier of each initial visual data, the initial visual features of the initial visual data are spliced to obtain the sequence visual features of the initial visual data.

[0166] According to the sequence visual features of the initial visual data and the second prompt text, a data processing result of the target visual data based on the second prompt text is obtained.

[0167] Specifically, the initial visual features of the initial visual data are obtained by using the visual data processing network, the initial visual features of the initial visual data can be distinguished by using the data sequence identifiers of the initial visual data, and when the initial visual features of the initial visual data are spliced, the splicing is performed according to the order of the data sequence identifiers in the original input. The spliced fusion features are obtained by splicing the multiple initial visual features spliced according to the data sequence identifiers and the text features corresponding to the second prompt text, and the fusion features include the semantic information of the second prompt text. Therefore, according to the fusion features, the initial visual features of the target visual data in the multiple initial visual features are understood and predicted based on the semantic information of the second prompt text, and the data processing result of the target visual data based on the second prompt text is obtained.

[0168] In actual application, the data sequence identifiers can be represented in the form of Token, and the data sequence identifiers can be added before the initial visual data to distinguish the initial visual features of different initial visual data and understand the sequence relationship of the initial visual data in the multiple initial visual data.

[0169] The data processing method provided by the present disclosure can determine the data sequence identifiers of the initial visual data when processing the multiple initial visual data, understand and predict the target visual data in the multiple initial visual data based on the second prompt text and the data sequence identifiers, realize personalized processing, provide data processing results meeting the actual needs of users, and improve user experience.

[0170] The following exemplary descriptions are made for several common application scenarios of the data processing method:

[0171] In one or more embodiments of the present disclosure, the data processing method can be applied to a data analysis scene, and the second prompt text is a prompt text of a data analysis task.

[0172] The data processing result of the target visual data based on the second prompt text is obtained, including:

[0173] The analysis text of the target visual data based on the prompt text of the data analysis task is obtained.

[0174] Continuing with the above example, the prompt text of the data analysis task can be "analyze the second page document" and "what does the second page document talk about", and the target visual data is the second document image in the multiple page document images. The data processing result is the analysis text obtained by analyzing the target visual data, such as "the content of the second page document is mainly XXX".

[0175] In one or more embodiments of the present specification, the data processing method can be applied to a data search scenario, and the second prompt text is a prompt text of a data search task.

[0176] The obtaining of the data processing result based on the second prompt text for the target visual data comprises:

[0177] The data search result based on the prompt text of the data search task for the target visual data is obtained, wherein the data search result comprises a data sequence identifier of the target visual data.

[0178] Continuing with the above example, the prompt text of the data search task can be "which page mentions the content about a part", at this time the multi-modal data processing network determines that the fourth initial visual feature includes information about the content of a part by understanding and analyzing each initial visual feature, and therefore determines that the fourth page document image mentions the content about a part, determines the fourth page document image as the target visual data, and outputs the data sequence identifier of the fourth page document image to obtain the data search result as "the fourth page".

[0179] In actual applications, in the case where the data sequence identifier of each initial visual data is in the form of Token and is spliced before each initial visual feature, the data sequence identifier spliced with the fourth initial visual feature is outputted.

[0180] In one or more embodiments of the present specification, the data processing method can be applied to a data search scenario, and the second prompt text is a prompt text of a data search task.

[0181] The obtaining of the data processing result based on the second prompt text for the target visual data comprises:

[0182] The data answer result based on the prompt text of the data answer task for the target visual data is obtained, wherein the data answer result comprises a data sequence identifier of the target visual data and an analysis text of the target visual data.

[0183] Continuing with the above example, the prompt text of the data search task can be "according to the input document, answer the question about b content, and give specific explanation and basis", at this time the multi-modal data processing network determines that the document pages about b content are the third page and the fourth page by understanding and analyzing each initial visual feature, determines the third page and the fourth page of the document image as the target visual data, and obtains the data answer result of the target visual data corresponding to the prompt text, such as the data answer result can be "according to the content of the 3rd page and the 4th page, it can be known that XXX".

[0184] The data processing method provided by the present disclosure can obtain multiple and different data processing results of target visual data based on prompt text according to different prompt text in each data processing task, and can be applied to different data processing scenarios, thereby improving the applicability.

[0185] In one or more embodiments of the present disclosure, after obtaining the data processing result, the data processing result is returned to the client, and the data processing result is displayed on the user interaction interface of the client. The specific implementation is described as follows.

[0186] The method further comprises the following steps:

[0187] The data processing result is returned to the client, and the data processing result is displayed on the user interaction interface of the client.

[0188] The data processing method provided by the present disclosure displays the data processing result on the user interaction interface of the client, timely feeds back the processing result of the input data, and improves the user interaction experience.

[0189] The data processing method provided by the present disclosure uses the global target visual feature as a guide and the recombined combined sub-visual feature as a compression object, ensures that the information of all regions is compressed and retained, can retain most of the text information, and can compress the initial visual data of different sizes into a fixed number of initial visual features; the initial visual features of the initial visual data are obtained by compression, the token occupation is reduced, the inference speed of the data processing model is accelerated, and the resources required in the inference process are saved, that is, the processing of the initial visual data is realized with faster inference time and less video memory occupation.

[0190] Referring to FIG. 4, FIG. 4 shows a processing process flowchart of the data processing method applied to the multi-page document parsing scene according to an embodiment of the present disclosure.

[0191] A plurality of high-resolution document pictures (i.e., multiple initial visual data in the above embodiment) and prompt text are determined, and each high-resolution document picture in the plurality of high-resolution document pictures is independently image encoded to obtain visual features corresponding to each high-resolution document picture. FIG. 4 shows a flowchart of processing a second high-resolution document picture.

[0192] Specifically, the high-resolution document picture is cut into multiple low-resolution (504x504) sub-pictures (i.e., sub-visual data in the above embodiment) by image cutting, for example, 6 low-resolution sub-pictures (hereinafter referred to as sub-pictures) are obtained by cutting, and the high-resolution document picture is changed into a low-resolution global picture (hereinafter referred to as a global picture) through a reduction process.

[0193] The obtained sub-pictures and global picture are input into a visual encoder (i.e., a data encoding unit in the above embodiment), and 1296 visual features of each sub-picture and global picture are obtained through the encoding process of the visual encoder; the visual features output by the visual encoder are input into a visual-to-text module (i.e., a feature aggregation module in the above embodiment), and the visual-to-text module aggregates the visual features of the sub-pictures and global picture in the horizontal aspect through a convolutional neural network, and reduces the visual features of the sub-pictures and global picture to 324; specifically, the visual-to-text module merges four visual features in the horizontal direction through a 1x4 kernel, reduces the sequence length, and retains the spatial information, and the full connection layer in the visual-to-text module maps the processed visual features to a language embedding space.

[0194] The high-resolution global compression module (i.e., a feature compression module in the above embodiment) is used to reorganize the feature maps of the visual features of the multiple sub-pictures according to the position identifier of the sub-pictures (the position identifier is used to represent the position of the sub-picture in the high-resolution document picture), to ensure that the visual data corresponding to each sub-picture is placed at the correct position corresponding to the high-resolution document picture, and to obtain a high-resolution combined feature map (i.e., a combined sub-visual feature in the above embodiment).

[0195] The low-resolution global feature map (i.e., a compressed target visual feature in the above embodiment) of the global picture output by the visual-to-text module is partitioned, for the visual feature map of each partition, the low-resolution global picture feature of the partition is taken as a query, the reorganized sub-picture feature corresponding to the position is taken as a key and a value, and a cross attention network (Cross Attention) is used for compression, such as calculating the attention weight of the reorganized sub-picture feature corresponding to the position, and performing weighted summation based on the attention weight, and the feature obtained after the weighted summation is determined as the partition initial visual feature of the partition; in the case of obtaining the partition initial visual feature of each partition, the initial visual feature of the single high-resolution document picture is obtained; in the same way, the initial visual feature of each high-resolution document picture in the multiple high-resolution document pictures is obtained, and a text sequence token (which can be understood as a data sequence identifier in the above embodiment) can be spliced in front of the initial visual feature of each high-resolution document picture, where x is a sequence number.

[0196] As shown in FIG. 4, the low-resolution global map feature map is divided into 4 regions, and for the features of the shadow region in the low-resolution global map feature map, based on the position of the features of the shadow region in the original high-resolution document picture, it is determined that the features of the shadow part in the high-resolution combined feature map are corresponding to the features of the shadow region in the low-resolution global map feature map.

[0197] The prompt text (usually an instruction to "parse document content") and the initial visual features of each high-resolution document picture are input into the large language model (i.e., the multi-modal data processing network in the above embodiment), the text features of the prompt text are obtained through the large language model, and the text features are fused with the initial visual features of each high-resolution document picture and the text ordinal number-based fusion features are obtained.

[0198] Specifically, the MAM (Multi-Modal Adaption Module) of the large language model is a multi-modal adaptation module, and its purpose is to enable the large language model to adapt to different modal inputs (such as vision and text) and effectively interact between them. The specific steps are that the MAM uses a linear layer to convert the input visual features, and the conversion process generates a pair of visual key-value pairs (key and value); the MAM uses another linear layer to process the input text features, and this processing also generates a pair of text key-value pairs (key and value); the key-value pairs of vision and text are compared and fused through self-attention mechanism inside the model, which allows the model to pass information between different modalities, thereby achieving a deeper understanding; and based on the prompt text, the parsed text of multiple high-resolution document pictures is obtained.

[0199] The data processing method provided by the present disclosure can reduce the number of visual features by 20% to achieve 80% of the effect of the data processing model in single-page document understanding, and at the same time, the data processing model has the ability to understand multi-page documents. In the evaluation of multi-page document understanding, the data processing model can also achieve better performance, that is, the inference speed is faster while ensuring the accuracy of the output results.

[0200] Referring to FIG. 5, FIG. 5 shows a flowchart of a data processing model training method according to an embodiment of the present disclosure, which specifically includes the following steps.

[0201] Step 502: Determine the initial visual sample data, the prompt text sample associated with the initial visual sample data, and the target data processing result.

[0202] Among them, the target data processing result is the data processing result of the initial visual sample data based on the prompt text sample.

[0203] Step 504: inputting the initial visual sample data and the prompt text sample into an initial data processing model, performing data cutting and data volume compression on the initial visual sample data respectively by using the initial data processing model, and obtaining a plurality of sub-visual sample data and target visual sample data.

[0204] The initial visual sample data includes the plurality of sub-visual sample data, and a resolution of the initial visual sample data is greater than a resolution of the target visual sample data.

[0205] Step 506: performing feature compression on a plurality of sub-visual sample features of the plurality of sub-visual sample data according to a target visual sample feature of the target visual sample data, and obtaining an initial visual sample feature of the initial visual sample data.

[0206] Step 508: obtaining a predicted data processing result of the initial visual sample data based on the prompt text sample according to the initial visual sample feature and the prompt text sample.

[0207] In one or more embodiments of the present specification, the initial visual sample data includes a plurality of initial visual sample data, the prompt text sample is a first prompt text sample associated with the plurality of initial visual sample data, and the target data processing result is a data processing result of the plurality of initial visual sample data based on the first prompt text sample.

[0208] The obtaining of the predicted data processing result of the initial visual sample data based on the prompt text sample according to the initial visual sample feature and the prompt text sample includes:

[0209] The obtaining of the predicted data processing result of the plurality of initial visual sample data based on the first prompt text sample according to the initial visual sample feature of each initial visual sample data in the plurality of initial visual sample data and the first prompt text sample.

[0210] For specific implementation, reference can be made to the above-mentioned embodiments, which will not be described here again.

[0211] In one or more embodiments of the present specification, the initial visual sample data includes a plurality of initial visual sample data, the prompt text sample is a second prompt text sample associated with target visual sample data in the plurality of initial visual sample data, and the target data processing result is a data processing result of the target visual sample data based on the second prompt text sample.

[0212] The determining of the initial visual sample data, the prompt text sample associated with the initial visual sample data, and the target data processing result further includes:

[0213] determine a data sequence identifier of each of the plurality of initial visual sample data;

[0214] The obtaining, according to the initial visual sample features and the prompt text sample, of a predicted data processing result based on the prompt text sample for the initial visual sample data comprises:

[0215] The obtaining, according to the initial visual sample features of each of the plurality of initial visual sample data and the second prompt text sample, of a predicted data processing result based on the second prompt text sample for the target visual sample data.

[0216] For details, refer to the above embodiments, which will not be repeated here.

[0217] Step 510: training the initial data processing model according to the predicted data processing result and the target data processing result to obtain a data processing model.

[0218] In one or more embodiments of the present specification, a loss function is calculated based on the predicted data processing result obtained by the initial data processing model and the target data processing result, and the model parameters of the initial data processing model are adjusted according to the loss function to obtain a data processing model, and the specific implementation is as follows:

[0219] The training of the initial data processing model according to the predicted data processing result and the target data processing result to obtain a data processing model comprises:

[0220] determining a loss function between the predicted data processing result and the target data processing result;

[0221] adjusting the model parameters of the initial data processing model according to the loss function to obtain the data processing model.

[0222] The loss function is used to quantify the difference between the predicted data processing result and the target data processing result, and the loss function includes but is not limited to mean square error, cross-entropy loss, etc.

[0223] Specifically, in the case where the loss function between the predicted data processing result and the target data processing result is determined, an optimization algorithm is used to minimize this loss function, so as to adjust the model parameters of the initial data processing model. The optimization algorithm can be gradient descent, stochastic gradient descent, etc., which is not limited here; in the case where a training end condition (such as the loss value no longer significantly decreases or the preset number of iterations is reached) is reached, the data processing model is obtained.

[0224] In actual application, the data processing model is trained according to a training framework of “single-picture text analysis pre-training-multi-picture text analysis and search pre-training-multi-task instruction fine-tuning”, that is, in order to ensure that the visual features of the document picture can also retain most of the text information after compression, the data processing model is first pre-trained on a single document picture for a data analysis task; then, in order to enhance the understanding ability of the data processing model for multiple document pictures, the model is pre-trained on multiple document pictures for data analysis and data search tasks; and then, the instruction fine-tuning data of single-page document question answering and multi-page document question answering is collected for multi-task fine-tuning training.

[0225] The data processing model training method provided by the present disclosure considers the performance of single-page document understanding and multi-page document understanding through the training framework of “single-picture text analysis pre-training-multi-picture text analysis and search pre-training-multi-task instruction fine-tuning”, ensures that the data processing model can have the understanding ability for multiple document pages, and enhances the generalization ability and adaptability of the data processing model according to the multi-task fine-tuning training, thereby providing an efficient and accurate solution for the document processing task in actual application.

[0226] Corresponding to the method embodiments described above, the present specification also provides data processing system embodiments. FIG. 6 shows a structural schematic diagram of a data processing system according to an embodiment of the present specification. The system includes one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement a data processing model, the data processing model including:

[0227] The first data processing network 602 is configured to perform data cutting and data volume compression on the determined initial data, to obtain a plurality of sub-data and target data, to perform feature compression on a plurality of sub-features of the plurality of sub-data according to a target feature of the target data, and to obtain an initial feature of the initial data.

[0228] The second data processing network 604 is configured to obtain a data processing result of the initial data based on the prompt data according to the initial feature and the determined prompt data associated with the initial data.

[0229] Optionally, the first data processing network 602 is further configured to:

[0230] The data preprocessing unit is configured to determine initial data and prompt data associated with the initial data, and to perform data cutting and data volume compression on the initial data to obtain a plurality of sub-data and target data;

[0231] The data encoding unit is configured to determine a plurality of sub-features of the plurality of sub-data and a target feature of the target data.

[0232] a data compression unit, configured to perform feature compression on the plurality of sub-features according to the target feature, to obtain initial features of the initial data.

[0233] The first data processing network can be understood as the visual data processing network in the data processing method of the above embodiment; the second data processing network can be understood as the multi-modal data processing network in the data processing method of the above embodiment; the data preprocessing unit can be understood as the data preprocessing unit in the data processing method of the above embodiment, which is configured to implement the specific content of the data preprocessing unit in the data processing method of the above embodiment, which will not be described here; similarly, the specific implementation of the data encoding unit can be referred to the content of the data encoding unit in the data processing method of the above embodiment; the specific implementation of the data compression unit can be referred to the content of the data compression unit in the data processing method of the above embodiment.

[0234] The data processing system provided by the embodiment of the present specification utilizes the data processing model to perform data processing on the input initial visual data, to obtain a plurality of sub-visual data and target visual data. Since the initial visual data includes a plurality of sub-visual data, the plurality of sub-visual features of the plurality of sub-visual data contain detailed visual information of the initial visual data, and the target visual data contains global visual information of the initial visual data. In the case of compressing the plurality of sub-visual features according to the target visual feature, the initial visual feature fused with global and local features is obtained, which contains global context information and detailed information from each sub-visual data, and the number of the initial visual feature is less than the number of the plurality of sub-visual features before compression. According to the initial visual feature and the prompt text, the data processing result of the initial visual data based on the prompt text is obtained. By obtaining the compressed initial visual feature with reduced number, the inference speed of the data processing model is improved, that is, the processing efficiency of data processing is improved. Generally, the visual data is processed by the video memory. By processing the compressed initial visual feature, the video memory occupation in the data processing process can be reduced, thereby being suitable for more multi-modal scenarios.

[0235] The above is a schematic scheme of a data processing system of the embodiment. It should be noted that the technical scheme of the data processing system belongs to the same concept as the technical scheme of the data processing method described above. The details of the technical scheme of the data processing system which are not described in detail can be referred to the description of the technical scheme of the data processing method.

[0236] Corresponding to the method embodiments described above, the specification also provides data processing device embodiments. FIG. 7 shows a structural schematic diagram of a data processing device according to an embodiment of the specification. As shown in FIG. 7, the device includes:

[0237] A determination module 702 configured to determine initial visual data and prompt text associated with the initial visual data;

[0238] A processing module 704 configured to input the initial visual data and the prompt text into a data processing model, and use the data processing model to respectively perform data cutting and data volume compression on the initial visual data, to obtain a plurality of sub-visual data and target visual data;

[0239] A compression module 706 configured to perform feature compression on a plurality of sub-visual features of the plurality of sub-visual data according to a target visual feature of the target visual data, to obtain an initial visual feature of the initial visual data;

[0240] An obtaining module 708 configured to obtain a data processing result of the initial visual data based on the prompt text according to the initial visual feature and the prompt text.

[0241] Optionally, the processing module 704 is further configured to:

[0242] perform cutting processing on the initial visual data using the data preprocessing unit to obtain the plurality of sub-visual data;

[0243] perform data volume compression processing on the initial visual data using the data preprocessing unit to obtain the target visual data.

[0244] The device further includes:

[0245] An extraction module configured to perform encoding processing on the plurality of sub-visual data using the data encoding unit to obtain a sub-visual feature of each sub-visual data in the plurality of sub-visual data;

[0246] perform encoding processing on the target visual data using the data encoding unit to obtain a target visual feature of the target visual data.

[0247] Optionally, the compression module 706 is further configured to:

[0248] perform initial compression processing on each sub-visual feature in the plurality of sub-visual features of the plurality of sub-visual data and the target visual feature of the plurality of sub-visual data using the data compression unit, respectively, to obtain a plurality of compressed sub-visual features and a compressed target visual feature;

[0249] According to the compressed target visual feature, target compression processing is performed on the plurality of compressed sub-visual features to obtain initial visual features of the initial visual data.

[0250] Optionally, the compression module 706 is further configured to:

[0251] The feature aggregation module is used to perform feature aggregation processing on each sub-visual feature in the plurality of sub-visual features respectively to obtain the plurality of compressed sub-visual features.

[0252] The feature aggregation module is used to perform the feature aggregation processing on the target visual feature to obtain the compressed target visual feature.

[0253] Optionally, the compression module 706 is further configured to:

[0254] The feature compression module is used to combine the plurality of compressed sub-visual features according to position identifiers of each sub-visual data in the plurality of sub-visual data to obtain combined sub-visual features, wherein the position identifiers of the each sub-visual data are determined through a preset position number sequence and a position relationship between the each sub-visual data and the initial visual data.

[0255] Cross-attention processing is performed on the compressed target visual feature and the combined sub-visual feature to obtain initial visual features of the initial visual data.

[0256] Optionally, the compression module 706 is further configured to:

[0257] A query vector is determined according to the compressed target visual feature, and a key-value vector is determined according to the combined sub-visual feature.

[0258] According to the query vector and the key-value vector, an attention weight of each compressed sub-visual feature in the combined sub-visual feature is determined.

[0259] According to the attention weight, weighted sum processing is performed on the plurality of compressed sub-visual features in the combined sub-visual feature to obtain initial visual features of the initial visual data.

[0260] Optionally, the compression module 706 is further configured to:

[0261] The compressed target visual feature is partitioned to obtain a plurality of partitioned target visual features.

[0262] According to the association relationship among the plurality of partitioned target visual features, the combined sub-visual feature, and the initial visual data, a plurality of compressed sub-visual features corresponding to each partitioned target visual feature are determined from the combined sub-visual feature.

[0263] determine a query vector according to the target visual feature of each partition, and determine a key value vector according to the plurality of compressed sub-visual features corresponding to the target visual feature of each partition;

[0264] determine an attention weight of each compressed sub-visual feature in the plurality of compressed sub-visual features corresponding to the target visual feature of each partition according to the query vector and the key value vector;

[0265] perform weighted sum processing on the plurality of compressed sub-visual features corresponding to the target visual feature of each partition according to the attention weight, to obtain an initial visual feature of each partition;

[0266] obtain an initial visual feature of the initial visual data according to the initial visual feature of each partition.

[0267] Optionally, the obtaining module 708 is further configured to:

[0268] input the initial visual feature and the prompt text into the multi-modal data processing network, and obtain a data processing result of the initial visual data based on the prompt text by using the multi-modal data processing network.

[0269] Optionally, the obtaining module 708 is further configured to:

[0270] input the prompt text into the feature extraction unit, and obtain a text feature of the prompt text by using the feature extraction unit;

[0271] perform fusion processing on the initial visual feature and the text feature by using the feature fusion unit, to obtain a fusion feature;

[0272] perform prediction processing on the fusion feature by using the data prediction unit, to obtain the data processing result of the initial visual data based on the prompt text.

[0273] Optionally, the obtaining module 708 is further configured to:

[0274] obtain a data processing result of each initial visual data based on the first prompt text by using the initial visual feature of each initial visual data in the plurality of initial visual data and the first prompt text.

[0275] The apparatus further includes an identification determination module configured to determine a data sequence identifier of each initial visual data in the plurality of initial visual data.

[0276] Optionally, the obtaining module 708 is further configured to:

[0277] According to the initial visual features of each initial visual data in the plurality of initial visual data, the data sequence identifier of each initial visual data, and the second prompt text, a data processing result of the target visual data based on the second prompt text is obtained.

[0278] Optionally, the obtaining module 708 is further configured to:

[0279] According to the data sequence identifier of each initial visual data, the initial visual features of each initial visual data are spliced to obtain sequence visual features of each initial visual data.

[0280] According to the sequence visual features of each initial visual data and the second prompt text, a data processing result of the target visual data based on the second prompt text is obtained.

[0281] Optionally, the obtaining module 708 is further configured to:

[0282] An analysis text of the target visual data based on the prompt text of the data analysis task is obtained.

[0283] Optionally, the obtaining module 708 is further configured to:

[0284] A data search result of the target visual data based on the prompt text of the data search task is obtained, wherein the data search result includes a data sequence identifier of the target visual data.

[0285] Optionally, the obtaining module 708 is further configured to:

[0286] A data answer result of the target visual data based on the prompt text of the data question and answer task is obtained, wherein the data answer result includes a data sequence identifier of the target visual data and an analysis text of the target visual data.

[0287] Optionally, the determining module 702 is further configured to:

[0288] An initial visual data and a prompt text associated with the initial visual data sent by a client are received, wherein the initial visual data is determined through an upload operation and / or an input operation of a user interaction interface of the client.

[0289] The apparatus further includes:

[0290] The sending module is configured to return the data processing result to the client to display the data processing result through the user interaction interface of the client.

[0291] The data processing method provided by the present disclosure uses global target visual features as guidance and recombined combined sub-visual features as compression objects, ensures that the information of all regions is compressed and retained, can retain most of the text information, and can compress initial visual data of different sizes into a fixed number of initial visual features; the initial visual features of the initial visual data are obtained through compression, the token occupation is reduced, the inference speed of the data processing model is accelerated, and the resources required in the inference process are saved, that is, the processing of the initial visual data is realized with faster inference time and less video memory occupation.

[0292] The above is a schematic scheme of the data processing device of the present embodiment. It should be noted that the technical scheme of the data processing device belongs to the same concept as the technical scheme of the data processing method described above, and the details of the technical scheme of the data processing device that are not described in detail can be referred to the description of the technical scheme of the data processing method.

[0293] The present disclosure also provides a data processing model training device, which comprises:

[0294] A determination module is configured to determine initial visual sample data, prompt text samples associated with the initial visual sample data, and target data processing results, wherein the target data processing results are data processing results of the initial visual sample data based on the prompt text samples;

[0295] A processing module is configured to input the initial visual sample data and the prompt text samples into an initial data processing model, and use the initial data processing model to perform data cutting and data volume compression on the initial visual sample data respectively, to obtain a plurality of sub-visual sample data and target visual sample data;

[0296] A compression module is configured to perform feature compression on a plurality of sub-visual sample features of the plurality of sub-visual sample data according to target visual sample features of the target visual sample data, to obtain initial visual sample features of the initial visual sample data;

[0297] An obtaining module is configured to obtain a predicted data processing result of the initial visual sample data based on the prompt text samples according to the initial visual sample features and the prompt text samples;

[0298] A training module is configured to train the initial data processing model according to the predicted data processing result and the target data processing result, to obtain a data processing model.

[0299] Optionally, the training module is further configured to:

[0300] determine a loss function between the predicted data processing result and the target data processing result;

[0301] adjust model parameters of the initial data processing model according to the loss function, to obtain the data processing model.

[0302] Optionally, the obtaining module is further configured to:

[0303] obtain a predicted data processing result of the target visual sample data based on the second prompt text sample, according to the initial visual sample features of each initial visual sample data in the plurality of initial visual sample data and the second prompt text sample.

[0304] The apparatus further includes an identification determining module configured to determine a data sequence identifier of each initial visual sample data in the plurality of initial visual sample data.

[0305] Optionally, the obtaining module is further configured to:

[0306] obtain a predicted data processing result of the target visual sample data based on the second prompt text sample, according to the initial visual sample features of each initial visual sample data in the plurality of initial visual sample data and the second prompt text sample.

[0307] The data processing model training method provided by the present disclosure has the performance of single-page document understanding and multi-page document understanding through the three-stage training framework, ensures that the data processing model has the understanding ability of multi-page documents, and enhances the generalization ability and adaptability of the data processing model according to the multi-task fine-tuning training, thereby providing an efficient and accurate solution for document processing tasks in actual applications.

[0308] The above is a schematic scheme of the data processing model training apparatus of the present embodiment. It should be noted that the technical scheme of the data processing model training apparatus belongs to the same concept as the technical scheme of the data processing model training method described above, and the details of the technical scheme of the data processing model training apparatus that are not described in detail can be referred to the description of the technical scheme of the data processing model training method.

[0309] FIG. 8 shows a structural block diagram of a computing device 800 according to an embodiment of the present specification. The components of the computing device 800 include, but are not limited to, a memory 810 and a processor 820. The processor 820 is connected to the memory 810 through a bus 830, and a database 850 is arranged to store data.

[0310] The computing device 800 also includes an access device 840 that enables the computing device 800 to communicate via one or more networks 860. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or combinations of such networks, such as the Internet. The access device 840 can include one or more of any type of network interface (for example, a network interface card (NIC)) such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC).

[0311] In one embodiment of the present specification, the above-mentioned components of the computing device 800 and other components not shown in FIG. 8 can also be connected to each other, for example, through a bus. It should be understood that the computing device structure block diagram shown in FIG. 8 is only for the purpose of example, and is not a limitation on the scope of the present specification. Other components can be added or replaced as needed by those skilled in the art.

[0312] The computing device 800 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (for example, a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (for example, a smartphone), a wearable computing device (for example, a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 800 can also be a mobile or stationary server.

[0313] Among them, the processor 820 is configured to execute the following computer program / instructions, which when executed by the processor, implement the steps of the above-mentioned data processing method, data processing model training method.

[0314] The various embodiments in the specification are described in progressive manner, and the same or similar parts among the various embodiments can be referred to each other, and each embodiment mainly describes the difference from other embodiments. Especially, for the computing device embodiment, since it is basically similar to the data processing method, the data processing model training method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the data processing method, the data processing model training method embodiment.

[0315] An embodiment of the specification further provides a computer readable storage medium storing computer programs / instructions, which are executed by a processor to implement the steps of the above data processing method and data processing model training method.

[0316] The various embodiments in the specification are described in progressive manner, and the same or similar parts among the various embodiments can be referred to each other, and each embodiment mainly describes the difference from other embodiments. Especially, for the computer readable storage medium embodiment, since it is basically similar to the data processing method, the data processing model training method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the data processing method, the data processing model training method embodiment.

[0317] An embodiment of the specification further provides a computer program product including computer programs / instructions, which are executed by a processor to implement the steps of the above data processing method and data processing model training method.

[0318] The above is a schematic scheme of the computer program product of the embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above data processing method and data processing model training method belong to the same concept, and the details of the technical scheme of the computer program product which are not described in detail can be referred to the description of the technical scheme of the above data processing method and data processing model training method.

[0319] The above describes specific embodiments of the specification. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps in a claim can be performed in an order different than the order in which the acts or steps are recited, and still accomplish the desired results. Also, the process depicted in the accompanying figures can not require the particular order shown, or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0320] The computer readable medium can include any entity or apparatus capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, software distribution medium, etc. It should be noted that the computer readable medium can include appropriate additions or subtractions according to the requirements of patent practice. For example, according to the patent practice in some regions, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0321] It should be noted that, for the foregoing method embodiments, in order to facilitate description, they are all described as a combination of a series of actions, but those skilled in the art should know that the present disclosure is not limited to the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present disclosure.

[0322] In the above embodiments, the description of each embodiment is focused on, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0323] The preferred embodiments of the present specification disclosed above are only used to help explain the present specification. The alternative embodiments do not describe all the details and limit the invention to the specific embodiments described. Obviously, according to the content of the present disclosure, many modifications and changes can be made. The present specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present disclosure, so that those skilled in the art can well understand and use the present specification. The present specification is limited by the claims and their entire scope and equivalents.

Claims

1. A data processing method comprising: determining initial visual data and prompt text associated with the initial visual data; inputting the initial visual data and the prompt text into a data processing model, using the data processing model to respectively perform data cutting and data volume compression on the initial visual data, and obtaining a plurality of sub-visual data and target visual data; performing feature compression on a plurality of sub-visual features of the plurality of sub-visual data according to a target visual feature of the target visual data, and obtaining an initial visual feature of the initial visual data; obtaining a data processing result of the initial visual data based on the prompt text according to the initial visual feature and the prompt text.

2. The data processing method of claim 1, wherein the data processing model comprises a visual data processing network, and the visual data processing network comprises a data preprocessing unit; the using the data processing model to respectively perform data cutting and data volume compression on the initial visual data, and obtaining a plurality of sub-visual data and target visual data comprises: using the data preprocessing unit to perform cutting processing on the initial visual data, and obtaining the plurality of sub-visual data; using the data preprocessing unit to perform data volume compression processing on the initial visual data, and obtaining the target visual data.

3. The data processing method of claim 1, wherein the data processing model comprises a visual data processing network, and the visual data processing network comprises a data encoding unit; after the obtaining a plurality of sub-visual data and target visual data, further comprising: using the data encoding unit to perform encoding processing on the plurality of sub-visual data, and obtaining a sub-visual feature of each sub-visual data in the plurality of sub-visual data; using the data encoding unit to perform encoding processing on the target visual data, and obtaining a target visual feature of the target visual data.

4. The data processing method of claim 1, wherein the data processing model comprises a visual data processing network, and the visual data processing network comprises a data compression unit; the performing feature compression on a plurality of sub-visual features of the plurality of sub-visual data according to a target visual feature of the target visual data, and obtaining an initial visual feature of the initial visual data comprises: using the data compression unit to respectively perform initial compression processing on each sub-visual feature in the plurality of sub-visual features of the plurality of sub-visual data and the target visual feature of the target visual data, and obtaining a plurality of compressed sub-visual features and a compressed target visual feature; performing target compression processing on the plurality of compressed sub-visual features according to the compressed target visual feature, and obtaining the initial visual feature of the initial visual data.

5. The data processing method of claim 4, wherein the data compression unit comprises a feature aggregation module; the using the data compression unit to respectively perform initial compression processing on each sub-visual feature in the plurality of sub-visual features of the plurality of sub-visual data and the target visual feature of the target visual data, and obtaining a plurality of compressed sub-visual features and a compressed target visual feature comprises: The feature aggregation module is used for performing feature aggregation processing on each of the plurality of sub-visual features respectively, to obtain the plurality of compressed sub-visual features. The feature aggregation module is used for performing the feature aggregation processing on the target visual feature, to obtain the compressed target visual feature.

6. The data processing method of claim 4 or 5, wherein the data compression unit further comprises a feature compression module; The target visual feature is compressed according to the compressed target visual feature, to obtain the initial visual feature of the initial visual data. The feature compression module is used for combining the plurality of compressed sub-visual features according to the position identifier of each of the plurality of sub-visual data, to obtain a combined sub-visual feature, wherein the position identifier of each of the sub-visual data is determined according to a preset position number sequence and a position relationship between each of the sub-visual data and the initial visual data. The compressed target visual feature and the combined sub-visual feature are subjected to cross-attention processing, to obtain the initial visual feature of the initial visual data.

7. The data processing method of claim 6, wherein the cross-attention processing of the compressed target visual feature and the combined sub-visual feature to obtain the initial visual feature of the initial visual data comprises: determining a query vector according to the compressed target visual feature, and determining a key-value vector according to the combined sub-visual feature; determining an attention weight of each compressed sub-visual feature in the combined sub-visual feature according to the query vector and the key-value vector; performing weighted summation processing on the plurality of compressed sub-visual features in the combined sub-visual feature according to the attention weight, to obtain the initial visual feature of the initial visual data.

8. The data processing method of claim 6, wherein the cross-attention processing of the compressed target visual feature and the combined sub-visual feature to obtain the initial visual feature of the initial visual data comprises: partitioning the compressed target visual feature to obtain a plurality of partitioned target visual features; determining, from the combined sub-visual feature, a plurality of compressed sub-visual features corresponding to each partitioned target visual feature according to an association relationship among the plurality of partitioned target visual features, the combined sub-visual feature, and the initial visual data; determining a query vector according to each partitioned target visual feature, and determining a key-value vector according to the plurality of compressed sub-visual features corresponding to each partitioned target visual feature; determining an attention weight of each compressed sub-visual feature in the plurality of compressed sub-visual features corresponding to each partitioned target visual feature according to the query vector and the key-value vector; performing weighted summation processing on the plurality of compressed sub-visual features corresponding to each partitioned target visual feature according to the attention weight, to obtain a partitioned initial visual feature of each partitioned target visual feature; obtaining the initial visual feature of the initial visual data according to the partitioned initial visual features of each partitioned target visual feature.

9. The data processing method of claim 1, wherein the data processing model comprises a multi-modal data processing network. obtaining, according to the initial visual feature and the prompt text, a data processing result based on the prompt text for the initial visual data, including: inputting the initial visual feature and the prompt text into the multi-modal data processing network, and obtaining, by using the multi-modal data processing network, the data processing result based on the prompt text for the initial visual data.

10. The data processing method of claim 9, wherein the multi-modal data processing network comprises a feature extraction unit, a feature fusion unit, and a data prediction unit. obtaining, according to the initial visual feature and the prompt text, a data processing result based on the prompt text for the initial visual data, including: inputting the prompt text into the feature extraction unit, and obtaining, by using the feature extraction unit, a text feature of the prompt text; performing fusion processing on the initial visual feature and the text feature by using the feature fusion unit, to obtain a fusion feature; performing prediction processing on the fusion feature by using the data prediction unit, to obtain the data processing result based on the prompt text for the initial visual data.

11. The data processing method of claim 1, wherein the initial visual data comprises a plurality of initial visual data, and the prompt text is a first prompt text associated with the plurality of initial visual data. obtaining, according to the initial visual feature and the prompt text, a data processing result based on the prompt text for the initial visual data, including: obtaining, according to the initial visual feature of each initial visual data in the plurality of initial visual data and the first prompt text, a data processing result based on the first prompt text for the each initial visual data.

12. The data processing method of claim 11, wherein the initial visual data comprises a plurality of initial visual data, and the prompt text is a second prompt text associated with target initial visual data in the plurality of initial visual data. After determining the initial visual data and the prompt text associated with the initial visual data, the method further comprises: determining a data sequence identifier of each initial visual data in the plurality of initial visual data; obtaining, according to the initial visual feature of each initial visual data in the plurality of initial visual data, the data sequence identifier of the each initial visual data, and the second prompt text, a data processing result based on the second prompt text for the target visual data.

13. The data processing method of claim 12, wherein the obtaining, according to the initial visual feature of each initial visual data in the plurality of initial visual data, the data sequence identifier of the each initial visual data, and the second prompt text, a data processing result based on the second prompt text for the target visual data, comprises: ​ According to the data sequence identification of the initial visual data, the initial visual features of the initial visual data are spliced to obtain sequence visual features of the initial visual data; According to the sequence visual features of the initial visual data and the second prompt text, a data processing result of the target visual data based on the second prompt text is obtained.

14. The data processing method of claim 12, wherein the second prompt text is a prompt text of a data parsing task; The data processing result of the target visual data based on the second prompt text is obtained, including: An analysis text of the target visual data based on the prompt text of the data parsing task is obtained.

15. The data processing method of claim 12, wherein the second prompt text is a prompt text of a data searching task; The data processing result of the target visual data based on the second prompt text is obtained, including: A data searching result of the target visual data based on the prompt text of the data searching task is obtained, wherein the data searching result includes the data sequence identification of the target visual data.

16. The data processing method of claim 12, wherein the second prompt text is a prompt text of a data question answering task; The data processing result of the target visual data based on the second prompt text is obtained, including: A data answer result of the target visual data based on the prompt text of the data question answering task is obtained, wherein the data answer result includes the data sequence identification of the target visual data and an analysis text of the target visual data.

17. The data processing method of claim 1, wherein the initial visual data and the prompt text associated with the initial visual data are determined, including: The initial visual data and the prompt text associated with the initial visual data sent by a client are received, wherein the initial visual data is determined through an upload operation and / or an input operation of a user interaction interface of the client; After the data processing result of the initial visual data based on the prompt text is obtained according to the initial visual features and the prompt text, the method further includes: The data processing result is returned to the client to display the data processing result through the user interaction interface of the client.

18. A data processing model training method, including: An initial visual sample data, a prompt text sample associated with the initial visual sample data, and a target data processing result are determined, wherein the target data processing result is a data processing result of the initial visual sample data based on the prompt text sample; The initial visual sample data and the prompt text sample are input into an initial data processing model, and the initial data processing model is used to respectively perform data cutting and data volume compression on the initial visual sample data to obtain a plurality of sub-visual sample data and a target visual sample data. According to a target visual sample feature of the target visual sample data, a plurality of sub-visual sample features of the plurality of sub-visual sample data are compressed to obtain an initial visual sample feature of the initial visual sample data; According to the initial visual sample feature and the prompt text sample, a prediction data processing result of the initial visual sample data based on the prompt text sample is obtained; According to the prediction data processing result and the target data processing result, the initial data processing model is trained to obtain a data processing model.

19. The data processing model training method of claim 18, wherein the initial data processing model is trained according to the prediction data processing result and the target data processing result to obtain a data processing model, comprising: determining a loss function between the prediction data processing result and the target data processing result; adjusting model parameters of the initial data processing model according to the loss function to obtain the data processing model.

20. The data processing model training method of claim 18, wherein the initial visual sample data comprises a plurality of initial visual sample data, the prompt text sample is a first prompt text sample associated with the plurality of initial visual sample data, and the target data processing result is a data processing result of the plurality of initial visual sample data based on the first prompt text sample; the prediction data processing result of the initial visual sample data based on the prompt text sample is obtained according to the initial visual sample feature of each initial visual sample data in the plurality of initial visual sample data and the first prompt text sample.

21. The data processing model training method of claim 18, wherein the initial visual sample data comprises a plurality of initial visual sample data, the prompt text sample is a second prompt text sample associated with target visual sample data in the plurality of initial visual sample data, and the target data processing result is a data processing result of the target visual sample data based on the second prompt text sample; after the initial visual sample data, the prompt text sample associated with the initial visual sample data, and the target data processing result are determined, the method further comprises: determining a data sequence identifier of each initial visual sample data in the plurality of initial visual sample data; the prediction data processing result of the initial visual sample data based on the prompt text sample is obtained according to the initial visual sample feature of each initial visual sample data in the plurality of initial visual sample data and the second prompt text sample.

21. The data processing model training method of claim 18, wherein the initial visual sample data comprises a plurality of initial visual sample data, the prompt text sample is a second prompt text sample associated with target visual sample data in the plurality of initial visual sample data, and the target data processing result is a data processing result of the target visual sample data based on the second prompt text sample; after the initial visual sample data, the prompt text sample associated with the initial visual sample data, and the target data processing result are determined, the method further comprises: determining a data sequence identifier of each initial visual sample data in the plurality of initial visual sample data; the prediction data processing result of the initial visual sample data based on the prompt text sample is obtained according to the initial visual sample feature of each initial visual sample data in the plurality of initial visual sample data and the second prompt text sample. 22.A data processing system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to implement a data processing model, the data processing model comprising: a first data processing network configured to perform data slicing and data volume compression on determined initial data to obtain a plurality of sub-data and target data, perform feature compression on a plurality of sub-features of the plurality of sub-data according to a target feature of the target data to obtain initial features of the initial data; a second data processing network configured to obtain a data processing result on the initial data based on prompt data associated with the initial data according to the initial features and the prompt data. 23.The data processing system of claim 22, wherein the first data processing network comprises: a data preprocessing unit configured to determine initial data, prompt data associated with the initial data, and perform data slicing and data volume compression on the initial data to obtain a plurality of sub-data and target data; a data encoding unit configured to determine a plurality of sub-features of the plurality of sub-data and a target feature of the target data; a data compression unit configured to perform feature compression on the plurality of sub-features according to the target feature to obtain initial features of the initial data. 24.A computing device comprising: a memory and a processor; the memory is configured to store computer programs / instructions, and the processor is configured to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method of any one of claims 1-21. 25.A computer readable storage medium storing computer programs / instructions, which, when executed by a processor, implement the steps of the method of any one of claims 1-21. 26.A computer program product comprising computer programs / instructions, which, when executed by a processor, implement the steps of the method of any one of claims 1-21.

Images