A data processing method and electronic device

Abstract

The application discloses a data processing method and an electronic device, and relates to the technical field of computers. The method comprises the following steps: acquiring input text and input images; generating text labels corresponding to the input text; determining the image categories of the input images, and determining activated visual feature extraction models according to the image categories of the input images; extracting visual features of the input images by using the activated visual feature extraction models, and generating image labels corresponding to the input images based on the visual features output by the activated visual feature extraction models; and inputting the text labels and the image labels into a pre-trained language model, so that the pre-trained language model generates answers corresponding to the input text and the input images. The application improves the processing performance of the pre-trained language model on different types of image data.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a data processing method and an electronic device. Background Technology

[0002] In related technologies, a single visual feature extraction model (with the same model structure and parameters) is typically used to process all types of input images. However, different types of images have drastically different feature distributions and task requirements. If a uniform visual feature extraction model is used to process these different types of data, the model may not be able to fully extract the features required for the specific task, leading to performance degradation.

[0003] Therefore, how to improve the processing performance of pre-trained language models on different types of data is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of this application is to provide a data processing method, apparatus and device, storage medium and computer program product that improves the processing performance of pre-trained language models for different types of data.

[0005] To achieve the above objectives, this application provides a data processing method, comprising: acquiring input text and an input image; generating text tags corresponding to the input text; determining the image category of the input image and determining an activated visual feature extraction model based on the image category of the input image; extracting visual features from the input image using the activated visual feature extraction model and generating image tags corresponding to the input image based on the visual features output by the activated visual feature extraction model; and inputting the text tags and image tags into a pre-trained language model so that the pre-trained language model generates answers corresponding to the input text and the input image.

[0006] To achieve the above objectives, this application provides a data processing apparatus, comprising: an acquisition module for acquiring input text and an input image; a first generation module for generating text tags corresponding to the input text; a determination module for determining the image category of the input image and determining an activated visual feature extraction model based on the image category of the input image; a second generation module for extracting visual features from the input image using the activated visual feature extraction model and generating image tags corresponding to the input image based on the visual features output by the activated visual feature extraction model; and an input module for inputting the text tags and image tags into a pre-trained language model so that the pre-trained language model generates responses corresponding to the input text and the input image.

[0007] To achieve the above objectives, this application provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the data processing method described above.

[0008] To achieve the above objectives, this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data processing method described above.

[0009] To achieve the above objectives, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the data processing method described above.

[0010] The data processing method based on a pre-trained language model provided in this application determines the category of the input image and activates the corresponding visual feature extraction model according to the category. This dynamic selection mechanism enables the system to adopt the most suitable feature extraction strategy for different types of images, thereby significantly improving the accuracy and relevance of feature extraction. Next, the activated visual feature extraction model is used to extract visual features from the input image, and image tags are generated based on these features. These image tags, along with the text tags corresponding to the input text, are input into the pre-trained language model, enabling the pre-trained language model to more accurately understand and process multimodal data. Therefore, this application not only improves the processing performance of the pre-trained language model for different types of image data but also enhances its overall performance in multimodal data fusion scenarios, thus achieving more efficient and accurate multimodal data processing. This application also discloses a data processing device, an electronic device, a computer-readable storage medium, and a computer program product, which can achieve the same technical effects.

[0011] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application. Attached Figure Description

[0012] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart illustrating a data processing method according to an exemplary embodiment.

[0014] Figure 2 This is a flowchart illustrating another data processing method according to an exemplary embodiment.

[0015] Figure 3 This is a multimodal large model structure diagram based on an adaptive visual encoder, provided in the application embodiments of this application.

[0016] Figure 4 A flowchart of a multimodal large model based on an adaptive visual encoder is provided in the application embodiments of this application.

[0017] Figure 5 This is a schematic diagram of the adaptive mode switching interface in the application embodiments provided in this application.

[0018] Figure 6 This is a schematic diagram of the manual mode switching interface in the application embodiments provided in this application.

[0019] Figure 7 This is a structural diagram of a data processing apparatus according to an exemplary embodiment.

[0020] Figure 8 This is a structural diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0022] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0023] Pre-training is a strategy for training deep learning models. Its core lies in using large-scale datasets to initially train the model, enabling it to learn general feature representations. This process is similar to the basic learning stage humans undergo before learning new knowledge, accumulating experience through extensive reading and observation.

[0024] Pre-trained language models generally refer to language model training tasks designed based on large-scale corpora (including language training materials such as sentences and paragraphs). A large-scale neural network algorithm structure is trained to learn and implement the model, resulting in a pre-trained language model with its parameters. Subsequent tasks can then use this model for feature extraction or task fine-tuning to achieve specific objectives. The idea behind pre-training is to first train a set of model parameters for one task, then use these parameters to initialize the network model parameters, and finally use the initialized network model to train other tasks, obtaining models adapted for those tasks. By pre-training on large-scale corpora, neural language representation models can learn powerful language representation capabilities, extracting rich syntactic and semantic information from text. Pre-trained language models can provide tokens containing rich semantic information and sentence-level features for downstream tasks. Fine-tuning can also be performed directly on pre-trained language models for downstream tasks, conveniently and quickly obtaining downstream-specific models.

[0025] The neural network algorithm structure used to train a pre-trained language model can be CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory), etc., or it can be a model built with attention networks, such as transformers, BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-trained Transformer), Clip (Contrastive Language–Image Pre-training), etc., which are not limited in this application. An attention network refers to a network model trained using an attention mechanism. This model extracts more important feature information from the input sequence by assigning different weights to each part of the input sequence, resulting in a more accurate output.

[0026] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] This application provides a data processing method through its embodiments. The method is described in detail below, along with its execution flow. See also... Figure 1 The flowchart illustrates a data processing method according to an exemplary embodiment.

[0028] S101: Obtain the input text and input image.

[0029] In multimodal large-scale model applications, input data typically includes both text and image modalities. For example, in an intelligent question-answering system, a user might upload an image containing specific information, such as a screenshot of a product manual, a document image with text, or a scene image, while simultaneously inputting a question text related to the image content. The system first needs to acquire this input data through appropriate interfaces. For text input, the system can receive the text information typed by the user through a text input box; for image input, it can receive the image file uploaded by the user through an image upload component. This input data is stored in the system's memory or temporary storage area for subsequent processing. Acquiring input data is the starting point of the entire multimodal processing flow; only by accurately acquiring the text and images provided by the user can the system provide the basic data support for subsequent operations such as feature extraction and fusion.

[0030] S102: Generate text tags corresponding to the input text.

[0031] In the field of natural language processing, text tokens are the basic units for processing text data. Tokenization is the process of breaking down an input text string into a series of tokens. These tokens can be words, sub-words, characters, etc., depending on the tokenization method used. For example, common tokenization methods include simple word segmentation based on spaces, which divides text into words as tokens; rule-based word segmentation, such as Chinese word segmentation tools that segment Chinese text into words according to certain grammatical rules; and model-based word segmentation, such as using BERT's WordPiece (a word segmentation algorithm) to decompose text into sub-word tokens. In this embodiment, a tokenization method suitable for a pre-trained language model can be used, such as using the tokenizer (segmenter / tokenizer) built into the pre-trained language model to tokenize the input text. For example, given the input text "This is a test text", BERT's tokenizer might break it down into a sequence of tokens like ["this", "is", "one", "segment", "test", "test", "text", "text"]. The purpose of generating text tokens is to convert the input text into a form that the model can understand and process, facilitating subsequent encoding and fusion operations in the pre-trained language model, thereby better utilizing textual information for multimodal tasks.

[0032] S103: Determine the image category of the input image, and determine the activated visual feature extraction model based on the image category of the input image.

[0033] The purpose of this step is to classify the input image to determine its category and select an appropriate visual feature extraction model based on the category. Image category refers to the content type represented by the input image. For example, images can be categorized as text images (such as scanned documents, handwritten notes, etc.), document images (document images containing structured content such as tables and charts), and natural scene images (such as landscape photos, street scene images, etc.). Once the image category is determined, the corresponding visual feature extraction model can be activated based on the category. In this embodiment, different visual feature extraction models are designed for different image categories. For example, an OCR (Optical Character Recognition) dedicated ViT (VisionTransformer) is used to process text images, a document analysis ViT is used to process document images, and a multi-object instance segmentation ViT is used to process natural scene images. In this way, the model most suitable for the feature extraction needs of different types of images can be used, thereby improving the accuracy and efficiency of image feature extraction and providing higher-quality image feature representations for subsequent multimodal fusion.

[0034] As a possible implementation method, determining the image category of the input image includes: using an image classification model to determine the image category of the input image.

[0035] Image classification models are trained deep learning models capable of identifying the category of an input image. They are typically based on convolutional neural network architectures, learning from large amounts of labeled image data to extract image features and perform classification.

[0036] The training process of an image classification model includes the following steps: 1. Data Collection and Labeling: Collect a large amount of image data of different categories and label each image with its category. For example, collect text images, document images, and natural scene images, and label them as "text," "document," and "scene," respectively. 2. Model Selection and Construction: Select a suitable CNN architecture, such as EfficientNet or ResNet (Residual Network), to build an image classification model. These models typically include an input layer, convolutional layers, pooling layers, and fully connected layers. 3. Model Training: Train the model using the labeled image data, and adjust the model parameters through the backpropagation algorithm to enable the model to accurately classify the input images. 4. Model Evaluation and Optimization: Evaluate the model's performance on a validation set and optimize the model as needed, such as adjusting hyperparameters and performing data augmentation.

[0037] In practical applications, an input image is fed into a trained image classification model, which outputs the probability value of the image belonging to each category. The category of the input image is then determined based on a set threshold or the highest probability principle.

[0038] For example, the following datasets were collected: text image datasets: textvqa (Text-based Visual Question Answering), ocr-vqa (Optical Character Recognition Visual Question Answering), and textocr (Text Optical Character Recognition); document image datasets: docvqa (Document Visual Question Answering), infovqa (Infographic Visual Question Answering), and chartqa (Chart Question Answering); natural scene image datasets: coco (Common Objects in Context), VQAv2 (Visual Question Answering Version 2), GQA (Graph Question Answering), and TDIUC (Task-Driven Image Understanding Challenge). These datasets were used to train the EfficientNet model. When given an image of a document containing a table, the model outputs the following probability distributions: text image: 0.1, document image: 0.8, natural scene image: 0.1. If the threshold is set to 0.45, the image is classified as a "document image".

[0039] Automated classification using image classification models can automatically identify the category of input images without manual intervention, thus improving processing efficiency. Deep learning models can accurately classify images, providing precise category information for subsequent visual feature extraction.

[0040] As a feasible implementation, the image classification model includes an input layer, a starting convolutional layer, multiple stage layers, a pooling layer, and a multi-label classification layer connected in sequence. The input layer is used to receive the input image. The starting convolutional layer includes a convolutional kernel of a preset size. The stage layers include multiple moving inverted bottleneck convolutional modules. The pooling layer is used to convert the feature map output by the last stage layer into a feature vector. The multi-label classification layer includes multiple parallel fully connected layers, which are used to output the probabilities of the corresponding different class labels.

[0041] In practical implementation, the input layer receives the input image, and the image size can be variable. The initial convolutional layer uses a 3×3 convolutional kernel to perform convolution operations on the input image with a stride of 2, resulting in 48 output channels. For example, for a 224×224 input image, after passing through the initial convolutional layer, the output feature map size is 112×112, with 48 channels. Stage layers: These contain multiple stage layers, each including multiple MBConv (Mobile Inverted Bottleneck Convolution) modules. For example, the stage layers of EfficientNetV2-S include: Stage 1: 4 MBConv blocks, increasing the number of channels from 48 to 64 with a stride of 1; Stage 2: 4 MBConv blocks, increasing the number of channels from 64 to 128 with a stride of 2; Stage 3: 6 MBConv blocks, increasing the number of channels from 128 to 160 with a stride of 2; Stage 4: 9 MBConv blocks, increasing the number of channels from 160 to 256 with a stride of 2.

[0042] The pooling layer uses Global Average Pooling (GAP) to convert the feature map output from the last stage layer into a feature vector. For example, for a 14×14×256 feature map, GAP will convert it into a 256-dimensional feature vector. The multi-label classification layer contains three parallel fully connected layers, corresponding to the three categories of "text image", "document image" and "natural scene image", respectively. Each fully connected layer outputs a probability value, representing the probability that the input image belongs to that category. The Sigmoid activation function is used to map the output value to the interval [0,1].

[0043] For an input image x, the probability p of each class label k is calculated independently using the sigmoid activation function: ;in, This indicates that the input image x belongs to the k-th category. This indicates that the input image x does not belong to the k-th category. Let x be the feature vector of the input image after global average pooling. , These are the weight vector and bias term related to the k-th category, respectively. Let T be the activation function, and let T denote the transpose.

[0044] As can be seen, image classification models can progressively extract high-level features of images and improve classification accuracy through multi-stage convolutional operations. These models can simultaneously output probabilities for multiple categories, making them suitable for multimodal image classification tasks. By optimizing the convolutional module and reducing the number of MBConv blocks in stage 4, high accuracy is maintained while reducing computational complexity and training time.

[0045] As another feasible implementation, determining the image category of the input image and determining the activated visual feature extraction model based on the image category of the input image includes: receiving a model determination instruction through an input interface and determining the activated visual feature extraction model based on the model determination instruction.

[0046] In practical implementation, a user-friendly graphical interface can be designed, allowing users to select or input model determination commands. For example, a drop-down menu can be provided, allowing users to select "Text Image Processing," "Document Image Processing," or "Natural Scene Image Processing." The user's model determination commands are received through an input interface. For example, if the user selects "Document Image Processing" from the drop-down menu, the corresponding visual feature extraction model is activated based on the user's input. For example, if the user selects "Document Image Processing," the document analysis ViT model is activated.

[0047] S104: Use the activated visual feature extraction model to extract visual features from the input image, and generate image labels corresponding to the input image based on the visual features output by the activated visual feature extraction model.

[0048] In this step, an activated visual feature extraction model is used to extract visual features from the input image. After extracting the visual features, they need to be converted into image tags for fusion with text tags. Image tag generation can be achieved by further processing the extracted visual features. For example, a fully connected layer can be used to map the visual features into a space with the same dimension as the text tags, generating vector representations similar to the text tags as image tags. Image tags generated in this way can be better fused with text tags in subsequent pre-trained language models. By using a dedicated visual feature extraction model to extract features from different categories of images and generate image tags, key information in the images can be captured more accurately, providing richer visual semantic information for multimodal tasks, thereby improving the overall system performance and effectiveness.

[0049] As a feasible implementation, if the activated visual feature extraction model includes a visual feature extraction model corresponding to text, and the visual feature extraction model corresponding to text includes an optical character recognition model based on a transformer architecture, then visual feature extraction of the input image is performed using the activated visual feature extraction model, including: dividing the input image into multiple non-overlapping image blocks, performing linear embedding on each image block to map each image block to an embedding vector of a preset dimension, and applying positional encoding to the embedding vector; inputting the positionally encoded embedding vector into the multilayer transformer encoding layer to extract visual features, inputting the extracted visual features into the transformer decoder, and generating a text sequence using cross-attention and autoregressive methods; and performing activation function and beam search processing on the text sequence to obtain the text recognition result as the visual features output by the visual feature extraction model corresponding to text.

[0050] In practical implementation, when the system identifies the input image as "text category," it activates an optical character recognition model (e.g., TroOCR) based on the Transformer architecture. First, the input image is uniformly divided into multiple non-overlapping 16×16 pixel image blocks. Each image block is mapped to a 768-dimensional continuous vector space through a linear embedding layer. Then, learnable positional encodings are appended to all vectors to preserve the original spatial information. Next, these embedded vectors with positional information are fed into a 12-layer Transformer encoder. Each layer consists of multi-head self-attention, residual connections and layer normalization, a feedforward network, and a second set of residual connections and layer normalization, extracting both global and local visual features. The encoded features are then fed into a Transformer decoder, which uses a cross-attention mechanism and an autoregressive approach to generate text sequences character by character. Finally, the string with the highest probability is output using the Softmax activation function and a Beam Search decoding strategy; this string is the model's final output of the "text visual features."

[0051] This implementation is entirely based on an end-to-end Transformer, achieving high-precision text recognition without the complex post-processing required by traditional CNN-CRNN (Convolutional Neural Network-Convolutional Recurrent Neural Network). Through high-resolution input (e.g., 896×896) and a local attention mechanism, it significantly improves the recognition accuracy of small fonts or curved text. Simultaneously, the model fully integrates printed and handwritten data during training, giving it strong generalization capabilities across various scenarios such as invoices, documents, and street scenes.

[0052] As a feasible implementation, if the activated visual feature extraction model includes a document-specific visual feature extraction model, and the document-specific visual feature extraction model includes a layout-aware multimodal document understanding model, then the activated visual feature extraction model is used to extract visual features from the input image, including: performing text embedding, image embedding, and position embedding on the input image respectively to obtain text vectors, image vectors, and position vectors respectively; using a multimodal transformer encoder to fuse the text vectors, image vectors, and position vectors to obtain multimodal features; performing a unified mask prediction task based on the multimodal features to obtain prediction results; wherein, the prediction results include text information, image information, and layout information; inputting the prediction results into a task-specific header, and using the document structure information or question-answering results output by the task-specific header as the visual features output by the document-specific visual feature extraction model.

[0053] In practice, when an image is identified as a "document category," the system invokes a layout-aware multimodal document understanding model (e.g., LayoutLMv3). This model receives three inputs in parallel: 1) a text sequence extracted using an OCR engine, which is then segmented using WordPiece to obtain text vectors; 2) an image vector obtained by encoding the entire document image using CNN or ViT; and 3) the two-dimensional coordinate information of each text fragment within the document, which is encoded using sine and cosine positions to form a position vector. These three vectors are concatenated and fed into a unified multimodal Transformer encoder, which aligns the text, image, and layout under a unified mask pre-training objective. Subsequently, the multimodal features output by the encoder are fed into a task-specific head: if the downstream task is document question answering, the answer fragment is output; if it is table parsing, structured JSON (JavaScript Object Notation) is output; and if it is layout restoration, a sequence of text blocks with coordinates is output. The above results constitute the final "document visual features" output by the document model.

[0054] This implementation method, by explicitly introducing coordinate and layout information, enables the model to accurately capture structured elements such as tables, headings, and paragraphs, significantly improving the metrics for document question answering and information extraction. Joint training of the unified mask language model and image mask modeling aligns textual and visual features in a shared semantic space, reducing the difficulty of cross-modal retrieval and understanding.

[0055] As a feasible implementation method, the pre-training process of the layout-aware multimodal document understanding model includes: constructing a training dataset and determining the annotation results of the training document images in the training dataset; wherein, the annotation results include text bounding boxes and semantic category labels; performing at least one cross-modal pre-training task based on the training dataset to train the layout-aware multimodal document understanding model to understand the relationship between text information, image information and layout information; wherein, the pre-training task includes any one or a combination of any of the following: cross-modal alignment matching task, unified mask prediction task, and document relationship reasoning task.

[0056] In practice, firstly, a large-scale training dataset with fine annotations is constructed. Each text region in a training document image is annotated not only with its location information (text bounding boxes) but also with semantic category labels describing its functional role (e.g., "heading," "paragraph," "table," "caption," etc.). These annotations collectively form the foundation for the model to learn document structure and semantics. Subsequently, at least one pre-training task designed to enhance cross-modal understanding is performed based on this dataset. These tasks include: a cross-modal alignment matching task, requiring the model to determine whether text fragments correspond to image regions to learn semantic relationships between modalities; a unified mask prediction task, randomly masking text words or image blocks, forcing the model to predict the masked content based on context (including other text, images, and layout) to learn deeply fused representations; and a document relationship reasoning task, requiring the model to infer the logical relationships between different document elements (such as a caption and the chart it describes) to understand the document's hierarchical structure and logic.

[0057] Through the above implementation methods, the model can be guided to learn deeper and more refined relationships between text, images, and layouts without altering its core Transformer architecture, thereby significantly improving the model's cross-modal understanding of complex documents. This pre-training strategy endows the model with strong domain adaptability and generalization capabilities because, by exposing it to massive and diverse labeled data and solving various challenging tasks, the model learns general prior knowledge of document structure, effectively handling document layouts and types not encountered during training. Finally, integrating this pre-trained model as a visual encoder into a multimodal large-scale model directly improves the accuracy and robustness of downstream tasks (such as document question answering and information extraction), achieving overall system performance optimization through a data- and task-driven approach.

[0058] As a feasible implementation, if the activated visual feature extraction model includes a visual feature extraction model corresponding to a natural scene, and the visual feature extraction model corresponding to a natural scene includes an unsupervised visual transformer model based on unlabeled knowledge distillation, then the activated visual feature extraction model is used to extract visual features from the input image, including: performing local and global cropping on the input image through a data augmentation module to obtain a first augmented image and a second augmented image, respectively; extracting a first image patch label sequence of the first augmented image through the student visual transformer backbone network, and extracting a second image patch label sequence of the second augmented image through the teacher visual transformer backbone network; calculating a self-distillation loss based on the first and second image patch label sequences; updating the teacher network parameters based on momentum using the self-distillation loss, and outputting the first image patch label sequence as the visual features output by the visual feature extraction model corresponding to the natural scene.

[0059] In practice, when an image is classified as a "natural scene," the system activates an unsupervised visual Transformer (e.g., DINOv2) based on unlabeled knowledge distillation. First, the input image undergoes two random transformations via a data augmentation module: 1) local cropping to obtain a first augmented image; 2) global cropping and enlargement to obtain a second augmented image. Subsequently, the student network ViT-Base receives the first augmented image and outputs a sequence of 197 image patch labels (including one class label); the teacher network ViT-Large receives the second augmented image and outputs a corresponding sequence of 197 labels. The two sequences are normalized, and the self-distillation loss is calculated to minimize the difference between the student and teacher network outputs. During training, the teacher network parameters smoothly follow the student network using a momentum update strategy, ensuring the stability of the feature space. During inference, only the class labels output by the student network are used as the 768-dimensional global features, i.e., the "natural scene visual features."

[0060] This implementation method uses unsupervised learning to learn general features of images without manual annotation, reducing data annotation costs. The self-distillation mechanism, through comparative learning between student and teacher networks, learns more stable and higher-quality features. The model can adapt to various natural scene images and has good generalization ability, thus providing general visual features for subsequent multimodal fusion.

[0061] As a feasible implementation, after extracting the first image patch label sequence of the first enhanced image through the student visual transformer backbone network, the method further includes: inputting the multi-scale first image patch label sequence into a feature pyramid network for feature fusion to obtain a fused feature map; inputting the fused feature map into a shared prediction head network for feature encoding; and inputting the encoded features output by the shared prediction head network into multiple parallel task branches to obtain instance segmentation results; wherein, the instance segmentation results include the target category, bounding box coordinates, and pixel-level mask; extracting corresponding instance region features from the first image patch label sequence based on the instance segmentation results; and correspondingly, outputting the first image patch label sequence as the visual features output by the visual feature extraction model corresponding to the natural scene, including: outputting instance region features as the visual features output by the visual feature extraction model corresponding to the natural scene.

[0062] In the specific implementation, firstly, the multi-scale first image patch label sequence is input into a feature pyramid network for feature fusion. This network generates a fused feature map that simultaneously considers high-resolution details and strong semantic information by fusing feature maps from different depths of the backbone network (shallow features retain rich spatial details, while deep features contain high-level semantic information), laying the foundation for accurate segmentation of objects of different sizes. Next, the fused feature map is input into a shared prediction head network for unified feature encoding. This design significantly reduces the number of parameters and improves computational efficiency through a weight-sharing mechanism. Subsequently, the encoded features are input in parallel into three task-specific branches: the classification branch outputs the target class probability distribution through a fully connected layer, the regression branch outputs the precise coordinates of the bounding box through a fully connected layer, and the masking branch generates pixel-level binary masks through a lightweight convolutional network. The three branches work together to output a complete instance segmentation result (including class, bounding box, and mask). Finally, based on this segmentation result, the corresponding instance region features are extracted from the first image patch label sequence output by the backbone network, and each detected object instance is transformed into a fixed-dimensional feature vector.

[0063] The above implementation, through the collaborative design of a feature pyramid network and multi-task parallel branches, significantly improves processing efficiency while ensuring instance segmentation accuracy. It can accurately identify objects of different scales and efficiently generate complete instance segmentation triples. By extracting instance-level region features from general visual features, dense pixel-level segmentation results are transformed into instanced feature representations with clear semantics. These features retain both the spatial structure and semantic information of objects and possess a well-organized data structure, making them more suitable for cross-modal fusion with text features. The entire processing flow, based on the DINOv2 self-supervised pre-trained model, achieves precise mapping from general visual representations to task-specific outputs with minimal computational overhead. This satisfies the efficiency requirements of multimodal systems while ensuring the reliability of instance segmentation in complex scenes, providing high-quality visual feature input for subsequent cross-modal understanding.

[0064] As a feasible implementation method, if the activated visual feature extraction models include a visual feature extraction model corresponding to text, a visual feature extraction model corresponding to a document, and a visual feature extraction model corresponding to a natural scene, then generating image tags corresponding to the input image based on the visual features output by the activated visual feature extraction models includes: mapping the visual features output by the visual feature extraction models corresponding to text, documents, and natural scenes to preset feature dimensions to obtain text sequence features, first spatial features, and second spatial features respectively; performing cross-modal attention calculation with the text sequence features as the query, the first spatial features as the key, and the second spatial features as the value to obtain a fused multimodal feature representation; and generating image tags corresponding to the input image based on the fused multimodal feature representation.

[0065] In practical implementation, when the system simultaneously activates three types of visual feature extraction models—text, document, and natural scene—image tag generation is achieved through an innovative cross-modal attention fusion mechanism. First, the heterogeneous features output by the three specialized models are dimensionally aligned: the text sequence features output by the text-corresponding model (e.g., TroOCR), the first spatial features containing layout information output by the document-corresponding model (e.g., LayoutLMv3), and the second spatial features of the instance region output by the natural scene-corresponding model (e.g., DINOv2+ segmentation head) are mapped to a unified preset feature dimension through independent linear projection layers, forming text sequence features, first spatial features, and second spatial features capable of cross-modal interaction. Subsequently, asymmetric cross-modal attention computation is employed: the text sequence features are used as query vectors, the document spatial features as key vectors, and the natural scene spatial features as value vectors. An attention mechanism enables the text features to actively query and aggregate relevant information from the two types of spatial features. This unique design allows text content to guide the selection and fusion of visual information, ultimately generating a fused multimodal feature representation. Based on this unified representation, image tags corresponding to the input image are generated through linear transformation or serialization processing.

[0066] Cross-modal attention is calculated as follows: ;in, The feature dimensions are preset for the fused dimensions, Q is the query, K is the build, V is the value, Softmax() is the normalization function, and CrossAttn() is the cross-modal attention calculation.

[0067] The above implementation first utilizes an asymmetric attention mechanism based on text as the query to actively guide and filter multi-source visual information through text semantics. This makes the fused features more focused on visual elements related to the text content, significantly improving the accuracy of cross-modal semantic alignment. Secondly, it fuses the structured layout features of the document with the instantiated object features of the natural scene through a unified attention framework. This preserves the document's layout structure information while incorporating entity information from the scene, forming a richer visual context representation. Finally, this unified fusion scheme effectively solves the alignment problem of multi-source heterogeneous visual features. The generated multimodal feature representation not only fully retains the output advantages of each specialized model but also possesses a regular and unified format, laying a solid foundation for the efficient understanding and generation of subsequent large language models.

[0068] As a feasible implementation method, generating image tags corresponding to input images based on visual features output by an activated visual feature extraction model includes: if only one activated visual extraction model exists, then the visual features output by the activated visual extraction model are used as image tags corresponding to the input images.

[0069] In practice, if only one visual feature extraction model is active, the visual features output by that model are directly used as the image tags corresponding to the input image. For example, if the activated visual feature extraction model corresponds to text, its output text recognition result is directly used as the image tag; if the activated visual feature extraction model corresponds to a document, its output document structure information or question-and-answer result is directly used as the image tag; if the activated visual feature extraction model corresponds to a natural scene, its output first image patch tag sequence is directly used as the image tag.

[0070] When only one model is activated, its output is directly used as the image label, reducing computational complexity and improving processing efficiency. This approach eliminates the need for additional feature fusion steps, directly utilizing the output of a single model, ensuring feature purity and consistency, and providing concise and efficient visual features for subsequent multimodal fusion.

[0071] As another feasible implementation, generating image tags corresponding to the input image based on the visual features output by the activated visual feature extraction model includes: if there are multiple activated visual extraction models, unifying the visual features output by the multiple activated visual extraction models into target visual features of a preset scale and preset resolution; determining the weights corresponding to the multiple activated visual extraction models respectively; weighting the target visual features based on the weights corresponding to the multiple activated visual extraction models to generate image tags corresponding to the input image.

[0072] In practical implementation, if multiple activated visual feature extraction models exist, a hierarchical feature fusion strategy is adopted. First, a spatial alignment layer is used, employing deformable convolutional layers to adjust the feature maps of different modules to a uniform spatial scale and resolution, thereby resolving the issue of inconsistent scales in the output feature maps of different ViT modules. Second, the weights corresponding to the multiple activated visual feature extraction models are determined, based on the output probabilities of the image content classification module. Finally, the target visual features are weighted based on the weights corresponding to the multiple activated visual feature extraction models to generate image labels corresponding to the input image.

[0073] This implementation method integrates the outputs of multiple visual feature extraction models, fully leveraging the strengths of different models. Through weighted fusion, the contribution of different models can be dynamically adjusted based on the content of the input image, thereby generating more accurate and comprehensive image labels. This approach is suitable for multimodal scenarios, better handles complex input images, and provides high-quality integrated visual features for subsequent multimodal fusion.

[0074] S105: Input the text and image tags into the pre-trained language model so that the pre-trained language model can generate answers corresponding to the input text and input image.

[0075] After generating text and image tags, this step involves inputting these two modalities into a pre-trained language model for fusion processing to generate an answer associated with the input text and image. In this embodiment, the pre-trained language model receives both text and image tags as input and interacts and fuses them through internal structures such as self-attention mechanisms. For example, in the self-attention mechanism, text and image tags can mutually calculate attention weights, allowing the model to understand the relationship between text and image. For instance, when the input text is "What is the name of the product in the picture?" and the input image is a screenshot of a product manual, the pre-trained language model can identify the text content in the image through the fusion processing of text and image tags, extract the key information of the product name, and ultimately generate the answer "The name of the product in the picture is XX". This multimodal fusion approach enables the pre-trained language model to utilize information from both text and images simultaneously, more comprehensively understanding the user's input intent and generating more accurate and meaningful answers. By inputting text and image tags into a pre-trained language model for fusion processing, the language processing capabilities and multimodal information fusion advantages of the pre-trained language model can be fully utilized to provide users with a more intelligent and efficient multimodal interactive experience. It is suitable for various scenarios that require processing by combining text and image information, such as intelligent customer service, image question answering, and document understanding.

[0076] The data processing method based on a pre-trained language model provided in this application determines the category of the input image and activates the corresponding visual feature extraction model according to the category. This dynamic selection mechanism enables the system to adopt the most suitable feature extraction strategy for different types of images, thereby significantly improving the accuracy and relevance of feature extraction. Next, the activated visual feature extraction model is used to extract visual features from the input image, and image tags are generated based on these features. These image tags, along with the text tags corresponding to the input text, are input into the pre-trained language model, enabling the pre-trained language model to more accurately understand and process multimodal data. Therefore, this application not only improves the processing performance of the pre-trained language model for different types of image data but also enhances its overall performance in multimodal data fusion scenarios, thus achieving more efficient and accurate multimodal data processing.

[0077] This application discloses a data processing method. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution. Specifically: see... Figure 2 The flowchart illustrates another data processing method according to an exemplary embodiment.

[0078] S201: Obtain the input text and input image.

[0079] S202: Generate text tags corresponding to the input text.

[0080] S203: Use an image classification model to determine the image category of the input image, and determine the activated visual feature extraction model based on the image category of the input image.

[0081] As a feasible implementation method, the activated visual feature extraction model is determined based on the image category of the input image, including: determining the probability of each category label output by the image classification model, and determining the visual feature extraction model corresponding to the category label with a probability greater than a preset value as the activated visual feature extraction model.

[0082] In practice, the image classification model is first invoked to perform forward reasoning on the input image to obtain the probability value corresponding to each preset category label (such as text, document, scene). Then, each probability is compared with a preset threshold (e.g., 0.5). For any category whose probability is greater than the threshold, its dedicated visual feature extraction model is marked as "activated" and enters the subsequent feature extraction process.

[0083] By using threshold filtering, the model selection process is fully automated, which can quickly eliminate low-confidence categories, reduce unnecessary computational overhead, and ensure that the activated model is highly matched with the image content, thereby improving the overall processing accuracy and efficiency.

[0084] As another feasible implementation, the activated visual feature extraction model is determined according to the image category of the input image, including: determining the probability of each category label output by the image classification model, and determining the visual feature extraction model corresponding to the category label with a probability greater than a preset value and the basic visual feature extraction model as the activated visual feature extraction model.

[0085] In practice, after obtaining the class probabilities output by the image classification model, the system not only activates the models corresponding to classes with probabilities greater than the threshold, but also forcibly activates a general basic visual feature extraction model; that is, regardless of the classification result, the basic model always participates in the subsequent process.

[0086] This implementation introduces a base model as a "safety net" path, which can provide robust visual features when classification is uncertain or the classification model is biased, thus enhancing the system's robustness. At the same time, the parallel operation of multiple models can also complement features of different granularities or domains, improving the richness and reliability of the final fused representation.

[0087] As a feasible implementation method, visual feature extraction is performed on the input image using a basic visual feature extraction model, including: dividing the input image into multiple non-overlapping image patches, performing linear embedding on each image patch to map each image patch to an embedding vector of a preset dimension; constructing learnable category labels and applying positional encoding to all labels; wherein, all labels include category labels and the embedding vector corresponding to each image patch; inputting the positionally encoded labels into a multi-layer transformer encoder; wherein, each layer transformer encoder includes a multi-head self-attention module, a first residual connection and layer normalization, a feedforward neural network, a second residual connection and layer normalization connected in sequence; and outputting the feature vector corresponding to the category label.

[0088] In its implementation, the basic visual feature extraction model first divides the input image into non-overlapping image blocks of fixed size, and then converts each block into an embedding vector of a preset dimension (e.g., 768-dimensional) through linear mapping. Subsequently, a learnable class token is introduced, which together with all the image block embeddings forms a token sequence, and positional encoding is applied to this sequence to preserve spatial information. Next, the sequence is fed into a multi-layer Transformer encoder, where each layer sequentially performs multi-head self-attention, the first residual connection and layer normalization, a feedforward neural network, and the second residual connection and layer normalization. Finally, the class token output by the encoder is the global visual feature vector of the entire image.

[0089] The basic visual feature extraction model extracts global and local features using a pure Transformer structure, with a clear computation path and high parallelism. The category label aggregates the information of the entire image, which can be directly used for downstream fusion tasks without additional pooling, thereby reducing the implementation complexity while ensuring feature expressiveness.

[0090] S204: If there are multiple activated visual extraction models, the visual features output by the multiple activated visual extraction models will be unified into target visual features with a preset scale and preset resolution.

[0091] S205: Determine the weights corresponding to the multiple activated visual extraction models, and weight the target visual features based on the weights corresponding to the multiple activated visual extraction models to generate image labels corresponding to the input image.

[0092] As a feasible implementation method, the weights corresponding to multiple activated visual extraction models are determined, including: determining the weights corresponding to the activated visual extraction models as the probabilities of the corresponding category labels output by the image classification model.

[0093] In practice, when the system detects that multiple visual feature extraction models are activated, it directly uses the probability of each category output by the image classification model as the weight of the corresponding model without additional calculation; these probability values ​​are then used to weighted fuse the visual features extracted by each model.

[0094] This implementation method uses the probability of category labels directly as weights, enabling dynamic adjustment of the contribution of different visual feature extraction models. Models corresponding to higher-probability categories will have a greater impact on the final feature fusion result, thus ensuring that the feature fusion result is closer to the actual category of the input image and improving the accuracy and adaptability of multimodal fusion.

[0095] As another feasible implementation, the weights corresponding to multiple activated visual extraction models are determined respectively, including: setting the weight corresponding to the basic visual feature extraction model to 1; and setting the weight corresponding to the activated visual extraction model to the probability of the corresponding category label output by the image classification model.

[0096] In practice, during the fusion phase, the system fixes the weight of the basic visual feature extraction model to 1, while the weights of the other activation models are still taken as the corresponding category probabilities output by the image classification model; all weights are normalized and then used for weighted summation to generate the final image label.

[0097] This implementation method considers the probability of category labels while retaining a weight of 1 for the basic visual feature extraction model. This ensures that the features of the basic model consistently have a stable impact on the final result, preventing a single category's excessively high probability from completely dominating the fusion result. This design can balance the features of specific category models and the basic model to a certain extent, improving the system's robustness and generalization ability. Especially when the input image category is unclear or the classification model has errors, the features of the basic model can provide more stable support. Simultaneously, the contributions of the remaining models are dynamically adjusted according to probability, balancing stability and adaptability.

[0098] This embodiment uses a weighted feature fusion method to fuse the outputs of multiple ViTs. The weighted average fusion can be expressed as: ,in, The visual features output by the basic visual feature extraction model. Extract the visual features output by the model for the other k-th activated visual features. Extract the weights corresponding to the model for the other k-th activated visual feature. , This represents the probability of the k-th class label output by the image classification model.

[0099] As a feasible implementation method, after determining the weights corresponding to the activated visual extraction model as the probabilities of the corresponding category labels output by the image classification model, the method further includes: receiving an adjustment command for the weights corresponding to the activated visual extraction model through an input interface, and responding to the adjustment command in order to adjust the weights corresponding to the activated visual extraction model.

[0100] In practical implementation, after the weights are initialized to class probabilities, an input interface (such as a graphical slider or command-line parameters) can be opened to receive user commands to adjust the weights of any activated model in real time. After parsing the commands, the weight values ​​are updated immediately and the fusion calculation is re-executed. This allows domain experts or end users to fine-tune the fusion ratio based on prior knowledge or actual results, significantly improving the system's controllability and final output quality in professional scenarios. At the same time, automatic initialization is retained as the default strategy, balancing generality and personalized needs.

[0101] S206: Input the text and image tags into the pre-trained language model so that the pre-trained language model can generate answers corresponding to the input text and input images.

[0102] The following describes an application embodiment provided by this application, a multimodal large model structure diagram based on an adaptive visual encoder, as shown below. Figure 3 As shown, the model includes components such as text and image input, a tokenizer, an adaptive visual feature extraction module, a pre-trained language model, and text output. This model can process various types of input data, including text and images, and generate corresponding text output.

[0103] A flowchart of a multimodal large model based on an adaptive visual encoder is shown below. Figure 4 As shown, the steps include: Step 1: Input text and images.

[0104] Step 2: Perform multi-label classification on the images: Use an image classification model to classify the input images to determine their categories.

[0105] Step 3: User selects image category: Users can manually select or confirm the image category results to ensure the accuracy of the classification.

[0106] Step 4: Image preprocessing: Based on the user-selected category, preprocess the image to prepare for subsequent feature extraction.

[0107] Step 5: Input the corresponding visual encoder: Based on the image classification results, select the corresponding Visual Transformer (ViT) model for feature extraction.

[0108] Step 6: Multi-Vision Encoder Feature Fusion: Fuse the features extracted from multiple ViT models to obtain a more comprehensive visual feature representation.

[0109] Step 7: Visual tokens and text tokens fusion: Input the extracted visual tokens and text tokens into the pre-trained language model to perform multimodal feature fusion.

[0110] Step 8: Input the pre-trained language model: The pre-trained language model processes the fused text and visual features.

[0111] Step 9: The pre-trained language model generates a text response.

[0112] This embodiment provides two interaction mechanisms, one of which is an automatic mode, such as... Figure 5 As shown, the ViT module is selected entirely based on the results of the image classification module; the second option is manual mode, as shown below. Figure 6 As shown, the user directly specifies the combination of ViT modules to use.

[0113] Therefore, compared to a unified architecture approach, this embodiment can dynamically select the most suitable processing path based on the characteristics of the input image, achieving significant accuracy improvements in document, text, and object segmentation tasks. Through resource sharing and dynamic scheduling, compared to methods that run multiple complete models in parallel, this embodiment effectively saves computational resources while maintaining equal or better processing quality. This is particularly important in scenarios with high real-time requirements, such as device information management. The introduced user intervention mechanism compensates for errors that may arise from purely automatic classification, making it particularly suitable for professional applications such as financial document analysis and device management, where domain knowledge can significantly improve processing accuracy. This embodiment can be applied to multiple fields such as finance, industry, and education, processing various types of documents and images. The modular design of this embodiment allows for the use of specific ViT modules or the entire system independently, adapting to different deployment scenarios. The cross-module attention fusion mechanism of this embodiment can better capture the semantic relationships between different representations.

[0114] The following describes a data processing apparatus provided in an embodiment of this application. The data processing apparatus described below can be referred to in conjunction with the data processing method described above. See also... Figure 7 The diagram illustrates a structural diagram of a data processing apparatus according to an exemplary embodiment.

[0115] The acquisition module 100 is used to acquire input text and input image.

[0116] The first generation module 200 is used to generate text tags corresponding to the input text.

[0117] The determination module 300 is used to determine the image category of the input image and determine the activated visual feature extraction model based on the image category of the input image.

[0118] The second generation module 400 is used to extract visual features from the input image using the activated visual feature extraction model, and to generate image labels corresponding to the input image based on the visual features output by the activated visual feature extraction model.

[0119] The input module 500 is used to input text and image tags into the pre-trained language model so that the pre-trained language model can generate answers corresponding to the input text and input images.

[0120] The data processing apparatus based on a pre-trained language model provided in this application determines the category of the input image and activates the corresponding visual feature extraction model according to the category. This dynamic selection mechanism enables the system to adopt the most suitable feature extraction strategy for different types of images, thereby significantly improving the accuracy and relevance of feature extraction. Next, the activated visual feature extraction model is used to extract visual features from the input image, and image tags are generated based on these features. These image tags, along with the text tags corresponding to the input text, are input into the pre-trained language model, enabling the pre-trained language model to more accurately understand and process multimodal data. Therefore, this application not only improves the processing performance of the pre-trained language model for different types of image data but also enhances its overall performance in multimodal data fusion scenarios, thus achieving more efficient and accurate multimodal data processing.

[0121] Based on the above embodiments, as a preferred implementation, the determining module 300 is specifically used to: determine the image category of the input image using an image classification model.

[0122] Based on the above embodiments, as a preferred implementation, the image classification model includes an input layer, a starting convolutional layer, multiple stage layers, a pooling layer, and a multi-label classification layer connected in sequence. The input layer is used to receive the input image. The starting convolutional layer includes a convolutional kernel of a preset size. The stage layers include multiple moving inverted bottleneck convolutional modules. The pooling layer is used to convert the feature map output by the last stage layer into a feature vector. The multi-label classification layer includes multiple parallel fully connected layers, which are used to output the probabilities of the corresponding different class labels.

[0123] Based on the above embodiments, as a preferred implementation, the second generation module 400 is specifically used for: if there are multiple active visual extraction models, unifying the visual features output by the multiple active visual extraction models into target visual features of a preset scale and preset resolution; determining the weights corresponding to the multiple active visual extraction models respectively, weighting the target visual features based on the weights corresponding to the multiple active visual extraction models, and generating image tags corresponding to the input image.

[0124] Based on the above embodiments, as a preferred implementation, the determining module 300 is specifically used to: determine the probability of each category label output by the image classification model, and determine the visual feature extraction model corresponding to the category label with a probability greater than a preset value as the activated visual feature extraction model; correspondingly, the second generating module 400 is specifically used to: determine the weight corresponding to the activated visual extraction model as the probability of the corresponding category label output by the image classification model.

[0125] Based on the above embodiments, as a preferred implementation, the determining module 300 is specifically used to: determine the probability of each category label output by the image classification model, and determine the visual feature extraction model corresponding to the category label with a probability greater than a preset value and the basic visual feature extraction model as the activated visual feature extraction model; correspondingly, the second generating module 400 is specifically used to: determine the weight corresponding to the basic visual feature extraction model as 1; and determine the weight corresponding to the activated visual extraction model as the probability of the corresponding category label output by the image classification model.

[0126] Based on the above embodiments, as a preferred implementation, the second generation module 400 is specifically used for: dividing the input image into multiple non-overlapping image blocks, performing linear embedding on each image block to map each image block to an embedding vector of a preset dimension; constructing learnable category labels and applying positional encoding to all labels; wherein, all labels include category labels and embedding vectors corresponding to each image block; inputting the positionally encoded labels into a multilayer transformer encoder; wherein, each layer transformer encoder includes a multi-head self-attention module, a first residual connection and layer normalization, a feedforward neural network, a second residual connection and layer normalization connected in sequence; and outputting feature vectors corresponding to the category labels.

[0127] Based on the above embodiments, as a preferred embodiment, it further includes: an adjustment module, configured to receive an adjustment command for the weights corresponding to the activated visual extraction model through an input interface, and respond to the adjustment command to adjust the weights corresponding to the activated visual extraction model.

[0128] Based on the above embodiments, as a preferred implementation, the determining module 300 is specifically used to: receive a model determining instruction through an input interface, and determine the activated visual feature extraction model based on the model determining instruction.

[0129] Based on the above embodiments, as a preferred implementation, the second generation module 400 is specifically used to: if there is only one active visual extraction model, then use the visual features output by the active visual extraction model as the image label corresponding to the input image.

[0130] Based on the above embodiments, as a preferred implementation, the image category includes any one or a combination of text, documents, and natural scenes.

[0131] Based on the above embodiments, as a preferred implementation, if the activated visual feature extraction model includes a visual feature extraction model corresponding to text, and the visual feature extraction model corresponding to text includes an optical character recognition model based on a transformer architecture, then the second generation module 400 is specifically used for: dividing the input image into multiple non-overlapping image blocks, performing linear embedding on each image block to map each image block to an embedding vector of a preset dimension, and applying position encoding to the embedding vector; inputting the position-encoded embedding vector into the multilayer transformer encoding layer to extract visual features, inputting the extracted visual features into the transformer decoder, and generating a text sequence using cross-attention and autoregressive methods; and performing activation function and beam search processing on the text sequence to obtain the text recognition result as the visual features output by the visual feature extraction model corresponding to text.

[0132] Based on the above embodiments, as a preferred implementation, if the activated visual feature extraction model includes a document-specific visual feature extraction model, and the document-specific visual feature extraction model includes a layout-aware multimodal document understanding model, then the second generation module 400 is specifically used for: performing text embedding, image embedding, and position embedding on the input image respectively to obtain text vectors, image vectors, and position vectors respectively; fusing the text vectors, image vectors, and position vectors using a multimodal transformer encoder to obtain multimodal features; performing a unified mask prediction task based on the multimodal features to obtain prediction results; wherein, the prediction results include text information, image information, and layout information; inputting the prediction results into a task-specific header, and using the document structure information or question-answering results output by the task-specific header as the visual features output by the document-specific visual feature extraction model.

[0133] Based on the above embodiments, as a preferred implementation, it further includes: a pre-training module for constructing a training dataset and determining the annotation results of training document images in the training dataset; wherein the annotation results include text bounding boxes and semantic category labels; performing at least one cross-modal pre-training task based on the training dataset to train a layout-aware multimodal document understanding model to understand the relationship between text information, image information and layout information; wherein the pre-training task includes any one or a combination of any of the following: cross-modal alignment matching task, unified mask prediction task, and document relationship reasoning task.

[0134] Based on the above embodiments, as a preferred implementation, if the activated visual feature extraction model includes a visual feature extraction model corresponding to a natural scene, and the visual feature extraction model corresponding to a natural scene includes an unsupervised visual transformer model based on unlabeled knowledge distillation, then the second generation module 400 is specifically used to: perform local cropping and global cropping on the input image through the data augmentation module to obtain a first augmented image and a second augmented image respectively; extract a first image patch label sequence of the first augmented image through the student visual transformer backbone network, and extract a second image patch label sequence of the second augmented image through the teacher visual transformer backbone network; calculate the self-distillation loss based on the first image patch label sequence and the second image patch label sequence; update the teacher network parameters based on the self-distillation loss through momentum, and output the first image patch label sequence as the visual feature output by the visual feature extraction model corresponding to the natural scene.

[0135] Based on the above embodiments, as a preferred implementation, the second generation module 400 is further configured to: input a multi-scale first image patch label sequence into a feature pyramid network for feature fusion to obtain a fused feature map; input the fused feature map into a shared prediction head network for feature encoding; input the encoded features output by the shared prediction head network into multiple parallel task branches to obtain instance segmentation results; wherein, the instance segmentation results include the target category, bounding box coordinates, and pixel-level mask; extract corresponding instance region features from the first image patch label sequence based on the instance segmentation results; and output the instance region features as visual features output by the visual feature extraction model corresponding to the natural scene.

[0136] Based on the above embodiments, as a preferred implementation, if the activated visual feature extraction models include a visual feature extraction model corresponding to text, a visual feature extraction model corresponding to document, and a visual feature extraction model corresponding to natural scene, then the second generation module 400 is specifically used to: map the visual features output by the visual feature extraction models corresponding to text, document, and natural scene to preset feature dimensions, respectively, to obtain text sequence features, first spatial features, and second spatial features; perform cross-modal attention calculation using the text sequence features as the query, the first spatial features as the key, and the second spatial features as the value, to obtain a fused multimodal feature representation; and generate an image tag corresponding to the input image based on the fused multimodal feature representation.

[0137] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0138] Embodiments of this application also provide an electronic device. Figure 8This is a structural diagram of an electronic device according to an exemplary embodiment, such as... Figure 8 As shown, the electronic device includes: a communication interface 1, capable of exchanging information with other devices such as network devices; and a processor 2, connected to the communication interface 1 to enable information exchange with other devices and, when running a computer program, to execute the data processing methods provided by one or more of the above-mentioned technical solutions. The computer program is stored in a memory 3.

[0139] Of course, in practical applications, the various components in an electronic device are coupled together through bus system 4. It can be understood that bus system 4 is used to achieve communication and connection between these components. In addition to the data bus, bus system 4 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 8 The general will label all buses as Bus System 4.

[0140] The memory 3 in this embodiment is used to store various types of data to support the operation of the electronic device. Examples of such data include any computer program used to operate on the electronic device.

[0141] It is understood that memory 3 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 3 described in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0142] The methods disclosed in the embodiments of this application can be applied to processor 2, or implemented by processor 2. Processor 2 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 2 or by instructions in the form of software. The processor 2 may be a general-purpose processor, DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 2 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory 3. Processor 2 reads the program in memory 3 and completes the steps of the aforementioned method in combination with its hardware.

[0143] When processor 2 executes the program, it implements the corresponding processes in the various methods of the embodiments of this application. For the sake of brevity, these will not be described in detail here.

[0144] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above data processing method embodiments when it is run.

[0145] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0146] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by processor 2, implements the steps in any of the above data processing method embodiments.

[0147] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by processor 2, implements the steps in any of the above data processing method embodiments.

[0148] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0149] The data processing system, method, apparatus, device, medium, and product provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A data processing method, characterized in that, include: Get the input text and input image; Generate text tags corresponding to the input text; The image category of the input image is determined, and the activated visual feature extraction model is determined based on the image category of the input image; wherein, the image category is the content type represented by the input image, and the image category includes any one or a combination of any of the following: text, document, and natural scene; Visual features are extracted from the input image using an activated visual feature extraction model, and image tags corresponding to the input image are generated based on the visual features output by the activated visual feature extraction model. The text and image tags are input into a pre-trained language model so that the pre-trained language model generates responses corresponding to the input text and the input image. Determining the image category of the input image includes: The image category of the input image is determined using an image classification model; The process of generating image labels corresponding to the input image based on the visual features output by the activated visual feature extraction model includes: If there are multiple activated visual extraction models, the visual features output by the multiple activated visual extraction models will be unified into target visual features with a preset scale and preset resolution. The weights corresponding to multiple activated visual extraction models are determined respectively, and the target visual features are weighted based on the weights corresponding to the multiple activated visual extraction models to generate image labels corresponding to the input image. The process of determining the activated visual feature extraction model based on the image category of the input image includes: Determine the probability of each category label output by the image classification model, and determine the visual feature extraction model corresponding to the category label with a probability greater than a preset value as the activated visual feature extraction model; Accordingly, the weights corresponding to the multiple activated visual extraction models are determined, including: The weights corresponding to the activated visual extraction model are determined as the probabilities of the corresponding category labels output by the image classification model.

2. The data processing method according to claim 1, characterized in that, The image classification model includes an input layer, a starting convolutional layer, multiple stage layers, a pooling layer, and a multi-label classification layer connected in sequence. The input layer is used to receive the input image. The starting convolutional layer includes a convolutional kernel of a preset size. The stage layers include multiple moving inverted bottleneck convolutional modules. The pooling layer is used to convert the feature map output by the last stage layer into a feature vector. The multi-label classification layer includes multiple parallel fully connected layers, which are used to output the probabilities of different corresponding class labels.

3. The data processing method according to claim 1, characterized in that, The method further includes determining the activated visual feature extraction model based on the image category of the input image, and also includes: The basic visual feature extraction model is identified as the activated visual feature extraction model; Accordingly, determining the weights corresponding to multiple activated visual extraction models also includes: The weight corresponding to the basic visual feature extraction model is set to 1.

4. The data processing method according to claim 3, characterized in that, Visual feature extraction of the input image is performed using the aforementioned basic visual feature extraction model, including: The input image is divided into multiple non-overlapping image blocks, and linear embedding is performed on each image block to map each image block into an embedding vector of a preset dimension; Construct learnable category labels and apply positional encoding to all labels; wherein, all labels include the category label and the embedding vector corresponding to each image patch; The position-encoded tag is input into the multilayer converter encoder; wherein, each layer of the converter encoder includes a multi-head self-attention module, a first residual connection and layer normalization, a feedforward neural network, and a second residual connection and layer normalization connected in sequence; Output the feature vector corresponding to the category label.

5. The data processing method according to claim 1 or 3, characterized in that, After determining the weights corresponding to the activated visual extraction model as the probabilities of the corresponding category labels output by the image classification model, the method further includes: The system receives an adjustment command for the weights corresponding to the activated visual extraction model via an input interface, and responds to the adjustment command to adjust the weights corresponding to the activated visual extraction model.

6. The data processing method according to claim 1, characterized in that, Determine the image category of the input image, and determine the activated visual feature extraction model based on the image category of the input image, including: The system receives model determination instructions through an input interface and determines the activated visual feature extraction model based on these instructions.

7. The data processing method according to claim 1, characterized in that, The visual features output by the activation-based visual feature extraction model generate image tags corresponding to the input image, including: If only one visual extraction model is active, the visual features output by the active visual extraction model will be used as the image tag corresponding to the input image.

8. The data processing method according to claim 1, characterized in that, If the activated visual feature extraction model includes a visual feature extraction model corresponding to the text, and the visual feature extraction model corresponding to the text includes an optical character recognition model based on a transformer architecture, then visual feature extraction of the input image is performed using the activated visual feature extraction model, including: The input image is divided into multiple non-overlapping image blocks, and linear embedding is performed on each image block to map each image block into an embedding vector of a preset dimension, and position encoding is applied to the embedding vector; The position-encoded embedding vector is input into the multi-layer transformer encoder layer to extract visual features. The extracted visual features are then input into the transformer decoder, and a text sequence is generated using cross-attention and autoregression. The text sequence is processed by activation function and beam search to obtain text recognition results, which are then used as the visual features output by the visual feature extraction model corresponding to the text.

9. The data processing method according to claim 1, characterized in that, If the activated visual feature extraction model includes a visual feature extraction model corresponding to the document, and the visual feature extraction model corresponding to the document includes a layout-aware multimodal document understanding model, then visual feature extraction is performed on the input image using the activated visual feature extraction model, including: Text embedding, image embedding, and position embedding are performed on the input image to obtain text vector, image vector, and position vector, respectively. Multimodal features are obtained by fusing the text vector, the image vector, and the position vector using a multimodal transformer encoder. A unified mask prediction task is performed based on the multimodal features to obtain prediction results; wherein, the prediction results include text information, image information, and layout information; The prediction result is input into a task-specific header, and the document structure information or question-and-answer results output by the task-specific header are used as the visual features output by the visual feature extraction model corresponding to the document.

10. The data processing method according to claim 9, characterized in that, The pre-training process of the layout-aware multimodal document understanding model includes: Construct a training dataset and determine the annotation results of the training document images in the training dataset; wherein, the annotation results include text bounding boxes and semantic category labels; At least one cross-modal pre-training task is performed based on the training dataset to train the layout-aware multimodal document understanding model to understand the relationships between text information, image information, and layout information; The pre-training tasks include any one or a combination of several of the following: cross-modal alignment matching task, unified mask prediction task, and document relation reasoning task.

11. The data processing method according to claim 1, characterized in that, If the activated visual feature extraction model includes a visual feature extraction model corresponding to a natural scene, and the visual feature extraction model corresponding to a natural scene includes an unsupervised visual transformer model based on unlabeled knowledge distillation, then visual feature extraction is performed on the input image using the activated visual feature extraction model, including: The input image is partially cropped and globally cropped using a data augmentation module to obtain a first enhanced image and a second enhanced image, respectively. The first image block label sequence of the first enhanced image is extracted through the student visual transformer backbone network, and the second image block label sequence of the second enhanced image is extracted through the teacher visual transformer backbone network. Calculate the self-distillation loss based on the first image block label sequence and the second image block label sequence; Based on the self-distillation loss, the teacher network parameters are updated using momentum, and the first image patch label sequence is output as the visual features output by the visual feature extraction model corresponding to the natural scene.

12. The data processing method according to claim 11, characterized in that, After extracting the first image patch label sequence of the first enhanced image through the student vision transformer backbone network, the method further includes: The first image patch label sequence at multiple scales is input into a feature pyramid network for feature fusion to obtain a fused feature map; The fused feature map is input into a shared prediction head network for feature encoding. The encoded features output by the shared prediction head network are then input into multiple parallel task branches to obtain instance segmentation results. The instance segmentation results include the target's category, bounding box coordinates, and pixel-level mask. Based on the instance segmentation result, the corresponding instance region features are extracted from the first image block label sequence; Accordingly, the first image patch label sequence is output as the visual features output by the visual feature extraction model corresponding to the natural scene, including: The features of the instance region are output as the visual features output by the visual feature extraction model corresponding to the natural scene.

13. The data processing method according to claim 1, characterized in that, If the activated visual feature extraction models include a visual feature extraction model corresponding to text, a visual feature extraction model corresponding to a document, and a visual feature extraction model corresponding to a natural scene, then image tags corresponding to the input image are generated based on the visual features output by the activated visual feature extraction models, including: The visual features output by the visual feature extraction models corresponding to text, documents, and natural scenes are mapped to preset feature dimensions to obtain text sequence features, first spatial features, and second spatial features, respectively. Using the text sequence features as the query, the first spatial feature as the key, and the second spatial feature as the value, cross-modal attention calculation is performed to obtain the fused multimodal feature representation; Based on the fused multimodal feature representation, an image tag corresponding to the input image is generated.

14. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to perform the steps of the data processing method as described in any one of claims 1 to 13 when executing the computer program.

Images