Visual encoding method and apparatus, and visual encoding model training method and apparatus

Abstract

The present application relates to the field of computer vision. Provided are a visual encoding method and apparatus, and a visual encoding model training method and apparatus, which are used for using the same visual encoding model to encode images of different resolutions, and are applied to encoding scenarios for images of more sizes. The visual encoding method comprises: first, acquiring an input image, wherein the input image may be a high-resolution image and may also be a low-resolution image; and then inputting the input image into a visual encoding model, so as to output visual encoding data, wherein the visual encoding model is used for dividing the input image into a plurality of image blocks according to positional embedding, extracting features from each image block, and outputting visual encoding data on the basis of the features of each image block and corresponding positional encoding, the positional embedding is obtained by means of adjusting initial positional embedding on the basis of the difference between the input image and a preset resolution, and the positional embedding may specifically comprise a matrix corresponding to the division of the input image

Description

A visual coding method, a visual coding model training method, and a device.

[0001] This application claims priority to Chinese Patent Application No. 202410806024.X, filed on June 20, 2024, entitled "A Visual Encoding Method, a Visual Encoding Model Training Method and Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer vision, and in particular to a visual coding method, a visual coding model training method, and an apparatus. Background Technology

[0003] Images are a common input data type in the field of artificial intelligence, especially in large-scale model scenarios. For example, multimodal large models utilize deep learning technology to fuse information from multiple data types such as text, images, and audio to achieve complex task processing. They have wide applications in natural language processing, speech recognition, sentiment analysis, autonomous driving, medical image analysis, entertainment, and social media, enhancing information integration and processing capabilities to bring users a more intelligent and richer interactive experience.

[0004] Visual encoding models are crucial components in large models, providing visual input. Typically, a pre-trained visual encoding model is set up within a large model to extract features from the input image, mapping these visual features to linguistic features. This allows the large model to perform downstream tasks based on the encoded information output by the visual model.

[0005] However, commonly used visual coding models typically only support one or more fixed image resolutions, or only support low-resolution image processing, thus limiting their applicability to various scenarios. Summary of the Invention

[0006] This application provides a visual coding method, a visual coding model training method, and an apparatus for encoding images of different resolutions using the same visual coding model, applicable to image coding scenarios of more sizes.

[0007] In view of this, in a first aspect, this application provides a visual encoding method, comprising: first, acquiring an input image, which may be a high-resolution image or a low-resolution image; then inputting the input image into a visual encoding model and outputting visual encoding data, wherein the visual encoding model is used to divide the input image into multiple image blocks according to position embedding, extract features from each image block, and output visual encoding data based on the features of each image block and the corresponding position encoding, wherein the position embedding includes adjusting the initial position embedding according to the difference between the input image and a preset resolution, wherein the position embedding may specifically include a matrix corresponding to the division of the input image, wherein each element in the matrix may correspond to a sampling region in the input image.

[0008] In this embodiment, the visual coding model can process images of different resolutions and can dynamically adjust the embedding position based on the resolution of the input image, thereby dividing the image of different resolutions into multiple image blocks. Then, feature extraction is performed on the image block as a unit, so it can be applied to various scenarios with different image resolutions and has excellent coding ability in images of different resolutions, with very strong generalization.

[0009] In one possible implementation, the aforementioned visual coding model is specifically used for: upsampling the initial position embedding when the resolution of the input image is higher than a preset resolution, to obtain a position embedding, which is equivalent to adapting the size of the position embedding to the resolution of the input image through upsampling; dividing the input image into multiple blocks according to the position embedding, and encoding the position of each block to obtain the position code corresponding to each block. Therefore, in this embodiment of the application, for high-resolution images, the initial position embedding can be upsampled to amplify the position embedding and adapt it to the image resolution, thus enabling the visual coding model to process high-resolution images.

[0010] In one possible implementation, the aforementioned visual coding model is further used to: when the resolution of the input image is lower than a preset resolution, downsample the initial position embedding to obtain a position embedding, which is equivalent to scaling down the initial position embedding for the low-resolution image, thereby obtaining a position embedding adapted to the low-resolution image; when the resolution of the input image is the preset resolution, use the initial position embedding as the position embedding. In this embodiment, the position embedding can be adaptively scaled down for low-resolution images, or the image can be divided into image blocks of a certain unit area to achieve processing of low-resolution images.

[0011] In one possible implementation, the aforementioned visual encoding model includes a visual encoding module and a representation layer. The visual encoding module is used to divide the input image into multiple image blocks according to positional embedding, extract features from each image block, and output visual encoded features based on the features of each image block and the corresponding positional encoding. The representation layer is used to obtain the representation vector corresponding to the visual encoded features and output visual encoded data. In this embodiment, the visual encoding model may also include a representation layer, which can be used to convert visual features into embedded representations, so that the downstream language model can perform downstream language processing tasks on the representations output by the representation layer.

[0012] In one possible implementation, the aforementioned visual coding model is obtained by distillation learning using a pre-trained coding model. In this application's embodiments, a visual coding model can be obtained by distillation learning using a pre-trained coding model, for example, by referring to the second aspect below or any optional implementation of the second aspect, thereby obtaining a visual coding model that performs exceptionally well at various resolutions.

[0013] Secondly, this application provides a visual coding model training method, comprising: first, acquiring a first training set and a second training set, wherein the image resolutions in the first training set and the second training set are different, and the image resolutions in the first training set include a preset resolution; then, using the first training set, the second training set, and a pre-trained coding model to train the visual coding model to obtain a trained visual coding model, wherein the pre-trained coding model and the first training set are used for distillation learning of the visual coding model, the image resolutions used when the pre-trained coding model is trained include the preset resolution, and the second training set can be used for supervised learning of the visual coding model, which is equivalent to setting different training branches using the first training set and the second training set, thereby adaptively training the visual coding model for images of different resolutions; wherein the visual coding model is used to divide the input image into multiple image blocks according to the position embedding, extract features from each image block, and output visual coding data based on the features of each image block and the corresponding position encoding, wherein the position embedding includes adjusting the initial position embedding according to the difference between the input image and the preset resolution.

[0014] In the embodiments of this application, during the training of the visual coding model, a training set including images of different resolutions can be used to set different training branches. For example, distillation learning can be performed using a pre-trained coding model for a preset resolution, thereby achieving more efficient model training. For other resolutions, supervised learning can be used to enable the visual coding model to perform better at other resolutions, so that the visual coding model can have the ability to process images of different resolutions.

[0015] In one possible implementation, one iteration of the aforementioned training process includes: using images from a first training set or a second training set as input to the visual coding model obtained in the previous iteration, and outputting first visual coding data; updating the visual coding model obtained in the previous iteration based on the first visual coding data, to obtain the visual coding model updated in the current iteration; wherein, when images from the first training set are used as input to the visual coding model obtained in the previous iteration, the images from the first training set are used as input to both the coding model and the visual coding model obtained in the previous iteration, and a distillation loss value is calculated based on the difference between the output of the coding model and the output of the visual coding model obtained in the previous iteration, and the distillation loss value is used to update the visual coding model obtained in the previous iteration. In this embodiment, for images at a preset resolution, the images can be used as input to the teacher model and the student model respectively, and distillation learning can be performed based on the outputs of the teacher model and the student model to update the parameters of the student model, thereby realizing knowledge distillation under images at a preset resolution and achieving more efficient training of the visual model.

[0016] In one possible implementation, the aforementioned process of updating the visual coding model obtained in the previous iteration based on the first visual coding data to obtain the visual coding model updated in the current iteration may include: inputting the first visual coding data into a language decoding model and outputting a decoding result; calculating a loss value based on the difference between the decoding result and the annotation corresponding to the image input into the visual coding model; and updating the visual coding model obtained in the previous iteration based on the loss value to obtain the visual coding model updated in the current iteration. In this embodiment, a language decoder can be used to decode the data output by the visual coding model, thereby representing the performance of the visual coding model's output in the downstream model. This facilitates comparison between the visual coding model's output and the corresponding annotation, thereby updating the visual coding model.

[0017] In one possible implementation, the aforementioned calculation of the distillation loss value based on the difference between the output of the encoding model and the output of the visual encoding model obtained in the previous iteration may include: using the output of the visual encoding model obtained in the previous iteration as the input of the visual decoder to obtain a visual feature vector; and calculating the distillation loss value based on the difference between the output of the encoding model and the visual feature vector. In this embodiment, the visual decoder can be used to decode the data output by the visual encoding model, thereby representing the performance of the visual encoding model's output in the downstream model, so as to facilitate comparison between the output of the visual encoder and the output of the pre-trained encoding model on the same dimension.

[0018] In one possible implementation, the aforementioned method may further include: acquiring an initial dataset, which includes multiple images and corresponding annotations for each image; the multiple images include natural images and document images; the natural images include images captured in a natural environment, and the document images are images containing document content; and scaling the images in the initial training set at least once to obtain a first training set and a second training set. In this embodiment, natural images and document images can be used for model training, thereby enabling the obtained visual encoding model to be applied in scenarios such as understanding natural images or parsing documents, thus improving the model's generalization ability.

[0019] In one possible implementation, the aforementioned method further includes: fine-tuning the updated visual coding model using a third training set to obtain a fine-tuned visual coding model. The third training set includes natural images and document images, wherein the fonts in the document images in the third training set are larger than the fonts in the document images in the first and second training sets, and the resolution of the natural images in the third training set is greater than the resolution of the natural images in the first and second training sets. In this embodiment, the model can be specifically fine-tuned for images with larger fonts or particularly high resolutions, thereby improving the robustness of the model.

[0020] Thirdly, this application provides a visual encoding device, comprising:

[0021] The acquisition module is used to acquire the input image;

[0022] The encoding module is used to input the input image into the visual encoding model and output visual encoding data. The visual encoding model is used to divide the input image into multiple image blocks according to the position embedding, extract features from each image block, and output visual encoding data based on the features of each image block and the corresponding position encoding. The position embedding is obtained by adjusting the initial position embedding according to the difference between the input image and the preset resolution.

[0023] The effects achieved by the third aspect or any optional implementation of the third aspect can be referred to the description of the first aspect or any optional implementation of the first aspect, and will not be repeated hereafter.

[0024] In one possible implementation, the aforementioned visual coding model is specifically used for: upsampling the initial position embedding to obtain a position embedding when the resolution of the input image is higher than a preset resolution; dividing the input image into multiple blocks according to the position embedding, and encoding the position of each block to obtain the position code corresponding to each block.

[0025] In one possible implementation, the aforementioned visual coding model is further used to: downsample the initial position embedding to obtain a position embedding when the resolution of the input image is lower than a preset resolution; and use the initial position embedding as the position embedding when the resolution of the input image is a preset resolution.

[0026] In one possible implementation, the aforementioned visual coding model includes a visual coding module and a representation layer. The visual coding module is used to divide the input image into multiple image blocks according to the position embedding, extract features from each image block, and output visual coding features based on the features of each image block and the corresponding position encoding. The representation layer is used to obtain the representation vector corresponding to the visual coding features and output visual coding data.

[0027] In one possible implementation, the aforementioned visual coding model is obtained by distillation learning using a pre-trained coding model.

[0028] Fourthly, this application provides a visual coding model training device, comprising:

[0029] The acquisition module is used to acquire a first training set and a second training set. The image resolutions in the first training set are different from those in the second training set. The image resolutions in the first training set include preset resolutions.

[0030] The training module is used to train the visual coding model using a first training set, a second training set, and a pre-trained coding model to obtain a trained visual coding model. The pre-trained coding model and the first training set are used to perform distillation learning on the visual coding model. The image resolution used when the pre-trained coding model is trained includes a preset resolution.

[0031] The visual encoding model is used to divide the input image into multiple image blocks according to the position embedding, extract features from each image block, and output visual encoding data based on the features of each image block and the corresponding position encoding. The position embedding is obtained by adjusting the initial position embedding according to the difference between the input image and the preset resolution.

[0032] The effects achieved by the fourth aspect or any optional implementation of the fourth aspect can be referred to the description of the second aspect or any optional implementation of the second aspect, and will not be repeated below.

[0033] In one possible implementation, one iteration of the aforementioned training process includes: using images from a first training set or a second training set as input to the visual coding model obtained in the previous iteration, and outputting first visual coding data; updating the visual coding model obtained in the previous iteration based on the first visual coding data, to obtain the visual coding model updated in the current iteration; wherein, when images from the first training set are used as input to the visual coding model obtained in the previous iteration, the images from the first training set are used as input to both the coding model and the visual coding model obtained in the previous iteration, and a distillation loss value is calculated based on the difference between the output of the coding model and the output of the visual coding model obtained in the previous iteration, and the distillation loss value is used to update the visual coding model obtained in the previous iteration.

[0034] In one possible implementation, the aforementioned training module is specifically used for: inputting the first visual coding data into the language decoding model and outputting the decoding result; calculating the loss value based on the difference between the decoding result and the annotation corresponding to the image input into the visual coding model; and updating the visual coding model obtained in the previous iteration based on the loss value to obtain the visual coding model updated in the current iteration.

[0035] In one possible implementation, the aforementioned training module is specifically used to: take the output of the visual encoding model obtained in the previous iteration as the input of the visual decoder to obtain a visual feature vector; and calculate the distillation loss value based on the difference between the output of the encoding model and the visual feature vector.

[0036] In one possible implementation, the aforementioned acquisition module is further configured to: acquire an initial dataset, which includes multiple images and corresponding annotations for each image, the multiple images including natural images and document images; and scale the images in the initial training set at least once to obtain a first training set and a second training set.

[0037] In one possible implementation, the aforementioned training module is further configured to: fine-tune the updated visual coding model using a third training set to obtain a fine-tuned visual coding model, wherein the third training set includes natural images and document images, and the fonts in the document images in the third training set are larger than the fonts in the document images in the first and second training sets, and the resolution of the natural images in the third training set is greater than the resolution of the natural images in the first and second training sets.

[0038] Fifthly, embodiments of this application provide a visual encoding apparatus, including a processor and a memory, wherein the processor and the memory are interconnected via a circuit, and the processor calls program code in the memory to execute processing-related functions in the difficult example recognition method shown in any of the first aspects above. Optionally, the difficult example recognition apparatus may be a chip.

[0039] Sixthly, embodiments of this application provide a visual coding model training apparatus, including a processor and a memory, wherein the processor and the memory are interconnected via circuits, and the processor calls program code in the memory to execute processing-related functions in the method shown in any of the second aspects above. Optionally, the visual coding model training apparatus may also be a chip.

[0040] In a seventh aspect, embodiments of this application provide a digital processing chip or chip, the chip including a processing unit and a communication interface, the processing unit obtaining program instructions through the communication interface, the program instructions being executed by the processing unit, the processing unit being used to perform processing-related functions as described in any of the first to second aspects above.

[0041] Eighthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the method described in any of the first or second aspects above.

[0042] Ninthly, embodiments of this application provide a computer program product comprising a computer program / instructions, which, when executed by a processor, causes the processor to perform the method in any of the embodiments of the first or second aspect described above. Attached Figure Description

[0043] Figure 1 is a schematic diagram of a system architecture provided in this application;

[0044] Figure 2 is a schematic diagram of another system architecture provided in this application;

[0045] Figure 3 is a flowchart illustrating a visual coding method provided in this application;

[0046] Figure 4 is a flowchart illustrating a visual coding model training method provided in this application;

[0047] Figure 5 is a flowchart illustrating another visual coding model training method provided in this application;

[0048] Figure 6 is a flowchart illustrating another visual coding model training method provided in this application;

[0049] Figure 7 is a schematic diagram of an application scenario of a visual coding method provided in this application;

[0050] Figure 8 is a schematic diagram of the structure of a visual coding model provided in this application;

[0051] Figure 9 is a schematic diagram of an application scenario for another visual coding method provided in this application;

[0052] Figure 10 is a schematic diagram of an application scenario of a visual coding model provided in this application;

[0053] Figure 11 is a schematic diagram of an application scenario for another visual coding model provided in this application;

[0054] Figure 12 is a schematic diagram of an application scenario for another visual coding model provided in this application;

[0055] Figure 13 is a schematic diagram of the structure of a visual coding device provided in this application;

[0056] Figure 14 is a schematic diagram of the structure of a visual coding model training device provided in this application;

[0057] Figure 15 is a schematic diagram of the structure of a computing device provided in this application. Detailed Implementation

[0058] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0059] The method provided in this application can be applied to artificial intelligence (AI) scenarios. For ease of understanding, some terms and concepts involved in this application are introduced below.

[0060] (1) Knowledge distillation

[0061] Knowledge distillation is a model compression method, a training method based on the "teacher-student network concept", which distills the knowledge contained in the pre-trained teacher model and extracts it into the student model.

[0062] Knowledge distillation transfers knowledge from one network to another, where the two networks can be isomorphic or heterogeneous. The process involves first training a teacher network (or teacher model), and then using the output of this teacher network to train a student network (or student model). During knowledge distillation, a pre-trained complex network can be used to train a simpler network, allowing the simpler network to possess the same or similar data processing capabilities as the complex network.

[0063] (2) Optical character recognition (OCR)

[0064] This typically refers to using electronic devices (such as scanners, cameras, or mobile phones) to scan physical documents (e.g., printed documents on paper or other materials), converting light signals into electrical signals to generate corresponding images, and then performing character recognition on the images to identify the content within them. For example, an electronic device can be used to scan a paper document, which may contain printed tables. This involves using optical methods to convert the printed characters into a black-and-white dot matrix image file, and then using recognition software to convert the text in the image into text format for further editing by word processing software.

[0065] (3) Language model (LM)

[0066] Language models play a crucial role in Natural Language Processing (NLP), where their task is to predict the probability of a sentence occurring in a language. For example, a language model is typically constructed as a probability distribution p(s) of a string s, where p(s) attempts to reflect the frequency of string s as a sentence. It can be applied to scenarios such as text recognition or machine translation. In the embodiments of this application, the NLP models mentioned below include language models.

[0067] (4) Large Language Model (LLM)

[0068] A large language model (LLM) refers to a language model containing hundreds of billions (or more) parameters trained on massive amounts of text data. It is a natural language processing model based on deep learning. These models can process large amounts of text data to learn the grammatical and semantic rules of natural language. LLMs can be applied to text generation, machine translation, question answering systems, text summarization, and sentiment analysis, offering advantages such as strong generative capabilities, high adaptability, accurate prediction, and strong scalability. For example, in movie recommendation scenarios, a large language model can generate descriptions of movie scenes, including genre, main actors, and plot, enabling the system to better recommend similar movies. Large language models can also generate recommendation reasons; for example, e-commerce websites can use large language models to generate reasons for recommending products, such as product quality, price, and features, allowing users to better understand the value of the product.

[0069] (5) Pre-trained language model (PLM)

[0070] A Natural Language Sequence Encoder (PLM) is a natural language sequence encoder that encodes each word in a natural language sequence into a vector representation for prediction tasks. PLM training consists of two phases: pre-training and fine-tuning. In the pre-training phase, the model is trained on large-scale unsupervised text to learn word representations. In the fine-tuning phase, the model is initialized using the parameters learned in the pre-training phase. For example, it can be trained on downstream tasks such as text classification or sequence labeling with fewer steps, successfully transferring the semantic information obtained in pre-training to downstream tasks.

[0071] (6) Transformer structure

[0072] Typically, PLMs can be built based on Transformer structures. For example, a PLM can be obtained by stacking multiple Transformer structures, and a Transformer structure can also be called a Transformer layer. Compared to other neural networks, such as recurrent neural networks or convolutional neural networks, Transformer layers rely on their attention mechanisms to capture long dependencies between words.

[0073] A transformer structure is a feature extraction network that includes an encoder and a decoder (classified as a convolutional neural network). Of course, in some cases, a transformer structure may not include an encoder but may include a decoder. The encoding model, encoder, or visual encoder mentioned in the methods provided in this application can be a model structure based on the Transformer structure.

[0074] Encoder: Learns features, such as pixel features, in the global receptive field through self-attention.

[0075] Decoder: Learns the features of the desired modules, such as the features of the output box, through self-attention and cross-attention.

[0076] (7) Self-attention

[0077] Self-attention refers to the efficient encoding of a sequence of data (such as the natural language corpus "Your phone is great.") into several multi-dimensional vectors for convenient numerical computation. These multi-dimensional vectors incorporate the similarity information between each element in the sequence; this similarity is called self-attention. A self-attention model can be understood as a mapping from a query to a series of value pairs (i.e., key-value pairs).

[0078] Specifically, attention mechanisms are typically applied between the encoder and decoder, such as between the input sentence and the generated sentence. Self-attention mechanisms in self-attention models occur within the encoding matrix or the output sequence, extracting connections between distant words within the same sentence, such as syntactic features (phrase structure). Self-attention mechanisms provide an effective modeling approach for capturing global contextual information through QKV (Queries, Keys, Values). Assuming the input is Q (query), and the context is stored as key-value pairs (K, V), then the attention mechanism is essentially a mapping function from the query to a series of key-value pairs (key, value). The essence of the attention function can be described as a mapping from a query to a series of (key-value) pairs. Attention essentially assigns a weight coefficient to each element in the sequence, which can also be understood as soft addressing. If each element in the sequence is stored in (K, V) form, then attention performs addressing by calculating the similarity between Q and K. The similarity calculated between Q and K reflects the importance of the extracted V values, i.e., the weights, and then a weighted sum is obtained to obtain the final feature value.

[0079] Attention calculation mainly consists of three steps. The first step is to calculate the similarity between the query and each key to obtain weights. Common similarity functions include dot product, concatenation, and perceptron. The second step typically uses a softmax function (which can normalize the weights, resulting in a probability distribution where the sum of all weight coefficients is 1, and also highlights the weights of important elements) to normalize these weights. Finally, the weights and their corresponding key values ​​are weighted and summed to obtain the final feature value. The specific calculation formula is as follows:

[0080] Where d is the dimension of matrix QK.

[0081] Furthermore, attention includes self-attention and cross-attention. Self-attention can be understood as a special type of attention where the inputs to the QKV features are consistent. Cross-attention, on the other hand, involves inconsistent inputs to the QKV features. Attention integrates the queried features as updated values ​​for the current features using the similarity between features (e.g., inner product) as weights. Self-attention is attention extracted based on the attention drawn from the feature map itself.

[0082] For convolutional networks, the kernel size limits the receptive field, often requiring multiple layers to focus on the entire feature map. Self-attention, on the other hand, offers the advantage of global focus; it can acquire global spatial information of the feature map through simple queries and assignments. A unique aspect of self-attention in query-key-value (QKV) models is that the inputs for each QKV value are consistent.

[0083] (8) Loss function

[0084] Also known as the cost function, it's a metric that compares the predicted output of a machine learning model with the true value of a sample (also called the supervised value). In other words, it measures the difference between the model's predicted output and the true value. This loss function typically includes loss functions such as mean squared error, cross-entropy, logarithmic loss, and exponential loss. For example, mean squared error can be used as the loss function, defined as... The specific loss function can be selected based on the actual application scenario.

[0085] (9) Gradient: The vector of the derivative of the loss function with respect to the parameters.

[0086] (10) Back propagation (BP): An algorithm for calculating the gradient of model parameters based on the loss function and updating the model parameters.

[0087] (11) markdown

[0088] It is a lightweight markup language that can be used to add formatting elements to plain text documents and is portable. It allows users to write documents in an easy-to-read and easy-to-write plain text format, and the resulting documents can be converted to HTML or other formats.

[0089] (12)Embedding

[0090] Embedding refers to the feature representation of a sample or word embedding representation.

[0091] (13)Embedding table

[0092] An embedding table can be understood as a collection of word embedding representations.

[0093] The method provided in this application can be executed on a server or on a terminal device. The terminal device can be a mobile phone with image processing capabilities, a tablet personal computer (TPC), a media player, a smart TV, a laptop computer (LC), a personal digital assistant (PDA), a personal computer (PC), a camera, a camcorder, a smartwatch, a wearable device (WD), or an autonomous vehicle, etc., and this application does not limit the specific device to this type.

[0094] Referring to Figure 1, this application embodiment provides a system architecture 100. This system architecture includes a database 230 and a client device 240. A data acquisition device 260 is used to collect data and store it in the database 230. A training device 220 generates a target model / rule 201 based on the data maintained in the database 230. The following will describe in more detail how the training device 220 obtains the target model / rule 201 based on the data. The target model / rule 201 is the large model mentioned in the following embodiments of this application, which can be used to process input data of different modalities and perform vision-based natural language processing (NLP) tasks.

[0095] The target model / rules obtained by training device 220 can be applied to different systems or devices. In Figure 1, execution device 210 is configured with transceiver 212, which can be a wireless transceiver, optical transceiver, or wired interface (such as an I / O interface), to interact with external devices. The "user" can input data to transceiver 212 through client device 240. For example, client device 240 can send a target task to execution device 210, request execution device to train a neural network, and send a database for training to execution device 210.

[0096] The execution device 210 can access data, code, etc., in the data storage system 250, and can also store data, instructions, etc., into the data storage system 250.

[0097] The computation module 211 processes the input data using the target model / rule 201. Specifically, the computation module 211 can be used to perform vision-based natural language processing (NLP) tasks, such as speech recognition, speech translation, and image caption generation, through the target model / rule 201.

[0098] For example, in one possible scenario, the target model / rule 201 can be a large model, and the visual coding model mentioned in the following embodiments of this application can be deployed in the large model. In other words, the visual coding model submitted in the following embodiments of this application can be a component in the large model, used to encode the image of the input large model and output visual coding data.

[0099] For example, in one possible scenario, a large model can be deployed in the execution device, and the target model / rule 201 trained by the training device can be a component of the large model, namely a visual coding model, used to encode the image input to the large model and output visual coding data.

[0100] Finally, transceiver 212 returns the constructed neural network to client device 240 for deployment in client device 240 or other devices.

[0101] In the scenario shown in Figure 1, the data input to the execution device 210 can be determined based on the user's input data. For example, the user can operate through the interface provided by the transceiver 212. Alternatively, the client device 240 can automatically input data to the transceiver 212 and obtain results. If the client device 240 needs user authorization to automatically input data, the user can set appropriate permissions in the client device 240. The user can view the results output by the execution device 210 on the client device 240; the specific presentation format can be display, sound, animation, etc. The client device 240 can also act as a data acquisition terminal, storing the acquired data associated with the target task into the database 230.

[0102] The training or update process mentioned in this application can be executed by the training device 220. It is understood that the training process of a neural network is learning how to transform the control space, more specifically, learning the weight matrix. The purpose of training the neural network is to make its output as close as possible to the expected value. Therefore, this can be achieved by comparing the current network's predicted value with the expected value, and then updating the weight vector of each layer of the neural network based on the difference between the two (of course, the weight vector can usually be initialized before the first update, i.e., pre-configured parameters for each layer in the deep neural network). For example, if the network's predicted value is too high, the values ​​of the weights in the weight matrix are adjusted to lower the predicted value. This adjustment continues until the neural network's output value is close to or equal to the expected value. Specifically, the difference between the neural network's predicted value and the expected value can be measured using a loss function or an objective function. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and the training of the neural network can be understood as a process of minimizing the loss as much as possible. The process of updating the weights of the starting network and training the serial network in the following embodiments of this application can be referred to in this process, and will not be repeated hereafter.

[0103] As shown in Figure 1, the target model / rule 201 is trained using the training device 220. In this embodiment, the target model / rule 201 can be a cross-modal model as described in this application. This cross-modal model can include, but is not limited to, deep convolutional neural networks (DCNN), recurrent neural networks (RNNS), and other networks. The neural networks mentioned in this application can include various types, such as deep neural networks (DNN), convolutional neural networks (CNN), recurrent neural networks (RNN), residual networks, and other neural networks.

[0104] During the training phase, database 230 can be used to store a sample set for training. Execution device 210 generates a target model / rule 201 for processing the samples and iteratively trains the target model / rule 201 using the sample set in the database to obtain a mature target model / rule 201, which is specifically represented as a neural network. The neural network obtained by execution device 210 can be applied to different systems or devices.

[0105] During the inference phase, the execution device 210 can access data, code, etc., from the data storage system 250, or it can store data, instructions, etc., in the data storage system 250. The data storage system 250 can be located within the execution device 210, or it can be an external memory relative to the execution device 210. The computing module 211 can process the samples acquired by the execution device 210 through a neural network to obtain prediction results. The specific form of the prediction results is related to the function of the neural network.

[0106] It should be noted that Figure 1 is merely an exemplary schematic diagram of a system architecture provided in this application embodiment, and the positional relationships between the devices, components, modules, etc. shown in the figure do not constitute any limitation. For example, in Figure 1, the data storage system 250 is an external memory relative to the execution device 210. In other scenarios, the data storage system 250 can also be placed within the execution device 210.

[0107] The target model / rule 201 trained by training device 220 can be applied to different systems or devices, such as mobile phones, tablets, laptops, augmented reality (AR) / virtual reality (VR), in-vehicle terminals, servers, or cloud devices.

[0108] The target model / rule 201 in this application embodiment can be the self-attention model in this application. Specifically, the self-attention model provided in this application embodiment can include CNN, deep convolutional neural networks (DCNN), recurrent neural networks (RNN), and other networks.

[0109] Referring to Figure 2, this application embodiment also provides a system architecture 200. The execution device 210 is implemented by one or more servers, optionally in conjunction with other computing devices, such as data storage, routers, load balancers, etc. The execution device 210 can be deployed on a single physical site or distributed across multiple physical sites. The execution device 210 can use data from the data storage system 250 or call program code from the data storage system 250 to implement the steps of the training method for computing devices corresponding to Figure 12 below.

[0110] Users can interact with execution device 210 by operating their respective user devices (such as local device 280 and local device 290, etc.). Each local device can represent any computing device, such as a personal computer, computer workstation, smartphone, tablet, smart camera, smart car or other type of cellular phone, media consumption device, wearable device, set-top box, game console, etc.

[0111] Each user's local device can interact with execution device 210 through a communication network using any communication mechanism / standard. The communication network can be a wide area network (WAN), a local area network (LAN), a point-to-point connection, or any combination thereof. Specifically, the communication network can include a wireless network, a wired network, or a combination of both. The wireless network includes, but is not limited to, any one or more combinations of: 5th-Generation (5G) systems, Long Term Evolution (LTE) systems, Global System for Mobile Communication (GSM) or Code Division Multiple Access (CDMA) networks, Wideband Code Division Multiple Access (WCDMA) networks, Wireless Fidelity (WiFi), Bluetooth, Zigbee, Radio Frequency Identification (RFID), Long Range (Lora) wireless communication, and Near Field Communication (NFC). The wired network can include fiber optic communication networks or networks composed of coaxial cables.

[0112] In another implementation, one or more aspects of the execution device 210 may be implemented by each local device. For example, local device 301 may provide local data or feedback calculation results to the execution device 210. This local device may also be referred to as a computing device.

[0113] It should be noted that all the functions of execution device 210 can also be implemented by a local device. For example, local device 280 implements the functions of execution device 210 and provides services to its own users, or provides services to the users of local device 302.

[0114] With the maturity of deep learning technology and the development of computing power, large-scale pre-training based on self-supervised learning (SSL) has been proven to significantly improve the representational ability of single-modal models such as text, speech, and images, thereby enhancing the performance of downstream tasks. In cross-modal models, with the emergence and popularity of contrastive learning / cross-attention Transformer techniques, pre-training for multimodal data has also begun to receive widespread attention. However, how to fully utilize resources from various modalities such as text, speech, and images to better learn cross-modal knowledge remains an ongoing research question.

[0115] Multimodal large models are artificial intelligence models that utilize deep learning technology to integrate information from multiple data types, such as text, images, and audio, to perform complex tasks. They are widely used in fields such as natural language processing, speech recognition, sentiment analysis, autonomous driving, medical image analysis, entertainment, and social media. By enhancing information integration and processing capabilities, they bring users a more intelligent and richer interactive experience. When users use multimodal large models, they input various data types (such as text, images, and audio), and the model processes them and outputs corresponding results, such as recognition results, sentiment states, or translated text.

[0116] Visual encoding models are key components of large multimodal models, also known as visual encoders, and are used to provide visual input, or visual tokens, to large language models. Typically, large multimodal models employ existing pre-trained visual encoders, which are then connected to the language model through intermediate layers such as linear convolution, attention mechanisms, or QFormer. This maps visual features to a shared multimodal representation space, converting visual signals into linguistic signals, and ultimately yielding the model's output.

[0117] Some commonly used large models, such as CLIP-ViT-L, OpenCLIP-ViT-H, and EVACLIP-ViT-G, employ similar training paradigms as CLIP. CLIP-ViT-L and OpenCLIP-ViT-H use image-text contrastive loss for training, demonstrating high performance and generalization ability when processing image and text data. These models essentially follow the same training paradigm as CLIP, using image-text contrastive loss. EVACLIP-ViT-G uses a mask-out completion scheme similar to BERT for training. However, while this type of large multimodal model, which directly uses a fixed pre-trained encoder, can cover most common coarse-grained visual perception tasks such as image captioning and image question answering, it performs poorly on tasks requiring fine-grained visual perception, such as text recognition and chart understanding. Because the frozen visual coding model (Vision Transformer, ViT) only performs well on data with resolutions similar to its pre-training data, it performs poorly on image inputs with large resolution differences. This is because the pre-trained positional coding cannot be adapted. If the positional coding is fine-tuned during the training phase, it needs to be interpolated up to the maximum resolution, which will greatly affect the model performance.

[0118] For example, one existing training scheme proposes a method to extend the fine-grained encoding capability based on an existing pre-trained visual encoder. This involves training a visual encoder to handle high-resolution input using fine-grained perceptual data, such as documents; using an existing pre-trained visual encoder to handle low-resolution images; passing all input images through the two corresponding visual encoders (high and low resolution), and then concatenating the output features to obtain the final visual encoding; after the encoding stage, during the pre-training stage of the multimodal large model, the visual encoding is input to the connection layer, and its output serves as the input to the large language model; the large language model outputs prediction results based on user commands; during the training stage, ground truth values ​​are used to supervise the prediction results, and backpropagation updates the entire model. However, this scheme requires adapting different visual encoders to different resolutions, resulting in wasted model capacity and increased inference latency. Furthermore, when adding new resolutions, a new visual encoding model needs to be introduced, requiring retraining the entire model, leading to very poor scalability.

[0119] For example, another existing approach dynamically extracts efficient and compact visual cues from a given image and descriptive text in a vision-language task, adapts them to a given pre-trained language model, and combines this with parameter-efficient transfer learning methods to achieve the conversion between unimodal and multimodal tasks. However, this approach cannot support inputs with large dynamic resolution spans and performs poorly on fine-grained vision tasks.

[0120] Based on the above description, existing large models often use pre-trained visual encoders with fixed resolution to encode multimodal data such as user-input images and videos. However, because they cannot handle high-resolution image inputs, they lack fine-grained perception capabilities when encoding dense visual inputs, resulting in severe information loss during the encoding process, which affects the prediction quality of multimodal large models.

[0121] Therefore, this application provides a multi-resolution visual coding method that can process image inputs of different resolutions through a resolution-adaptive visual coding module. This allows for flexible adaptation to input data with a wide resolution range from low to high, adaptively extracting corresponding visual information for coarse-grained or fine-grained perceptual tasks. Especially for high-resolution input images, it can retain more detail information, enabling more accurate completion of user-specified tasks, particularly in situations requiring the extraction of effective information from densely texted documents to fulfill user instructions.

[0122] The method and process provided in this application are described below.

[0123] Referring to Figure 3, a flowchart of a visual coding method provided in this application is shown below.

[0124] 301. Obtain the input image.

[0125] The input image can be a high-resolution image or a low-resolution image. That is, the method provided in this application does not limit the resolution of the input image, thus adapting to scenarios with different resolutions.

[0126] 302. Input the input image into the visual coding model and output the visual coding data.

[0127] After obtaining the input image, it can be fed into the visual coding model, which can encode the input image and output the encoded data, i.e., the visually encoded data.

[0128] Specifically, the visual encoding model is used to divide the input image into multiple image blocks according to the position embedding, extract features from each image block, and output visual encoding data based on the features of each image block and the corresponding position encoding. The position embedding is obtained by adjusting the initial position embedding according to the difference between the input image and the preset resolution. The position embedding can be represented as a matrix, where each element in the matrix corresponds to an image block or the position of the image block.

[0129] Furthermore, the visual encoder may adopt a transformer-based structure, a masked autoencoder (MAE), or other models that can be used to extract features from images, etc., and this application does not limit it in this regard.

[0130] Optionally, the visual coding model includes a visual coding module and a representation layer. The visual coding module is used to divide the input image into multiple image blocks according to the position embedding, extract features from each image block, and output visual coding features based on the features of each image block and the corresponding position encoding. The representation layer is used to obtain the representation vector corresponding to the visual coding features and output visual coding data.

[0131] Therefore, the visual coding model provided in this application can be used to adaptively adjust the position embedding based on the resolution of the input image, thereby adapting to scenarios with multiple image resolutions and having stronger generalization ability.

[0132] In one possible implementation, the visual coding model is specifically used for: when the resolution of the input image is higher than a preset resolution, upsampling the initial position embedding, such as through interpolation or padding in other ways, to obtain the position embedding, i.e., the partitioning matrix corresponding to the image segmentation; then, dividing the input image into multiple blocks based on the position embedding, and encoding the position of each block to obtain the position code corresponding to each block. Therefore, in this embodiment, for high-resolution images, the initial position embedding can be upsampled to obtain a position embedding containing more positions, thereby dividing the input image into more blocks to adapt to image coding in high-resolution image scenarios.

[0133] In one possible implementation, the visual coding model is further configured to: downsample the initial position embedding to obtain a position embedding when the resolution of the input image is lower than a preset resolution; and use the initial position embedding as the position embedding when the resolution of the input image is the preset resolution. Therefore, in this embodiment, when the resolution of the input image is lower than the preset resolution, the initial position embedding can be downsampled, thereby obtaining a position embedding that matches the size of the input image. This ensures that the unit volume of each image block is kept within a certain size, allowing subsequent processing to be performed based on image blocks of unit volume, thus achieving image processing for different resolutions.

[0134] In addition, optionally, in order to keep the output length of the visual coding model within a certain range, a representation layer can be set in the visual coding model. This representation layer can be used to obtain the representation vector corresponding to the visual coding data. For example, it can query a pre-set representation vocabulary to determine the representation vector corresponding to the visual coding data.

[0135] Furthermore, the aforementioned visual coding model can be specifically trained using the visual coding model training method provided in this application. The process of the visual coding model training method provided in this application will be described below.

[0136] Referring to Figure 4, a flowchart of a visual coding model training method provided in this application is shown below.

[0137] 401. Obtain the first training set and the second training set.

[0138] The image resolution in the first training set is different from that in the second training set, and the image resolution in the first training set includes a preset resolution.

[0139] Specifically, the first and second training sets can each include various types of images, such as images of natural environments or images containing documents. Typically, different types of images may be applied to different scenarios. For ease of distinction, images of natural environments will be referred to as natural images, and images containing documents will be referred to as document images.

[0140] 402. Use the first training set, the second training set, and the pre-trained encoding model to train the visual encoding model to obtain the trained visual encoding model.

[0141] The training of the visual coding model can be divided into different branches. For example, the pre-trained coding model and the first training set are used to perform distillation learning on the visual coding model. The image resolution used when the coding model is trained includes a preset resolution. The second training set can be used for supervised learning or unsupervised learning of the visual coding model.

[0142] The processing flow of this visual coding model can be referred to the description in Figure 3 above, and will not be repeated here.

[0143] For a given resolution, the pre-trained encoding model typically performs better at that resolution. Therefore, the pre-trained encoding model can be directly used for distillation to learn the image processing capabilities at the given resolution more efficiently. For images other than the given resolution, a second training set can be used for learning to improve the visual encoding model's processing capabilities at resolutions other than the given resolution, thus making the final visual encoding model perform better at various image resolutions.

[0144] Therefore, in the embodiments of this application, a pre-trained encoding model can be used as a teacher model to distill the visual encoding model, so that the visual encoding model can inherit the image processing capabilities of the pre-trained encoding model at a preset resolution, and at the same time learn images at other resolutions besides the preset resolution. Thus, the resulting visual encoding model can learn better processing capabilities at each resolution, enabling the visual encoding model to process images at various resolutions simultaneously and have stronger generalization ability.

[0145] Typically, training a visual coding model can involve multiple iterations. Taking any one of these iterations as an example, one iteration in the training process can include:

[0146] Images from either the first or second training set are used as input to the visual coding model obtained in the previous iteration, and the first visual coding data is output. Then, the visual coding model obtained in the previous iteration is updated based on the first visual coding data to obtain the visual coding model updated in the current iteration. Specifically, when images from the first training set are used as input to the visual coding model obtained in the previous iteration, the images from the first training set are used as input to both the coding model and the visual coding model obtained in the previous iteration. The distillation loss value is calculated based on the difference between the output of the coding model and the output of the visual coding model obtained in the previous iteration, and the distillation loss value is used to update the visual coding model obtained in the previous iteration.

[0147] In this embodiment, images from the first training set and the second training set can be used as inputs to train the visual coding model. When an image from the first training set is used as input to the visual coding model, it is also used as input to the pre-trained coding model. This allows the output of the pre-trained coding model to be used for distillation learning, enabling the visual coding model to inherit the image processing capabilities of the pre-trained visual coding model at a preset resolution, thereby improving the learning efficiency of the visual coding model.

[0148] Optionally, the aforementioned process of updating the visual coding model obtained in the previous iteration based on the first visual coding data to obtain the visual coding model updated in the current iteration may specifically include: inputting the first visual coding data into a language decoding model and outputting a decoding result, wherein the language decoding model may be a pre-trained language model used to perform downstream tasks; subsequently calculating a loss value based on the difference between the decoding result and the annotation corresponding to the image input to the visual coding model; and updating the visual coding model obtained in the previous iteration based on the loss value to obtain the visual coding model updated in the current iteration. In this embodiment, a pre-trained model used to perform downstream tasks can be used to process the visual coding data output by the visual coding model, and a loss value can be calculated based on the difference between the processing result and the annotation of the image. This allows the performance of the visual coding model's output in downstream tasks to be used to update the visual coding model, thereby improving the performance of the visual coding model's output data in downstream tasks.

[0149] Optionally, the aforementioned calculation of the distillation loss value based on the difference between the output of the encoding model and the output of the visual encoding model obtained in the previous iteration may include: using the output of the visual encoding model obtained in the previous iteration as the input of the visual decoder to obtain a visual feature vector; the visual decoder can be used to decode the encoded data; and the distillation loss value is calculated based on the difference between the output of the encoding model and the visual feature vector. In this embodiment, when calculating the distillation loss value, the visual decoder can be used to process the visual encoded data, thereby aligning the decoded output with the output of the pre-trained encoding model in various dimensions, such as length or number of dimensions, to facilitate the calculation of the loss value.

[0150] Optionally, the aforementioned method may further include: obtaining an initial dataset, which includes multiple images and corresponding annotations for each image, wherein the multiple images include natural images and document images; and scaling the images in the initial training set at least once to obtain a first training set and a second training set. In this embodiment, scaling the initial images can yield images of different resolutions, allowing the visual coding model to be trained at different resolutions, thus enabling the visual coding model to perform better on images of various resolutions.

[0151] Optionally, the aforementioned method may further include: fine-tuning the updated visual coding model using a third training set to obtain a fine-tuned visual coding model. The third training set includes natural images and document images, wherein the fonts in the document images in the third training set are larger than the fonts in the document images in the first and second training sets, and the resolution of the natural images in the third training set is greater than the resolution of the natural images in the first and second training sets. In this embodiment, for images with large fonts or very high resolution images, a separate training set can be set to fine-tune the visual coding model, thereby enabling the visual coding model to encode images with large fonts or very high resolution images, improving the robustness of the visual coding model.

[0152] The method provided in this application has been described above. The following section describes the method flow provided in this application in conjunction with specific application scenarios.

[0153] The method provided in this application can be divided into a training phase and an inference phase. The training phase is the phase of training the visual encoding model, and the inference phase is the phase of using the visual encoding model to perform inference tasks. The training phase and the inference phase are described below.

[0154] I. Model Training

[0155] For example, the architecture of the training phase in the method provided in this application can be as shown in Figure 5.

[0156] The following section describes the different steps of the training phase, based on the architecture shown in Figure 5 above.

[0157] 1. Model Initialization

[0158] During the model training phase, distillation can be used to train the visual encoding model. Therefore, in the initialization phase, a student model and a teacher model can be set up separately. The student model is the visual encoder, or visual encoding model, and the teacher model is the pre-trained encoding model.

[0159] For visual encoders, embodiments of this application can employ a vision transformer (ViT) architecture as the backbone network. For example, let fθ(·) be a visual encoder with parameter θ, for an input image I∈R H×W×3 Here, H and W represent the height and width of the image, respectively. The ViT model first divides each image into fixed-size p×p pixel blocks, then encodes these blocks into latent features of dimension d. Subsequently, these features, along with their corresponding positional codes, are input into multi-layered stacked Transformer modules, which interact with each other through an attention mechanism. Finally, the model outputs visual feature labels X∈R. N×d,like: Where N = N s +N c This represents the total number of visual markers, including spatial marker N. s =H p ×W p The number of [CLS] tags, Nc.

[0160] This application refers to the visual encoder as UNIT (UNifying image and text recognition), which may include multiple modules, such as a resolution-adaptive visual encoding module and a task-adaptive visual information extraction module.

[0161] The visual encoding module enables multi-resolution adaptive positional encoding and position-constrained semantic feature reconstruction, outputting visual features that incorporate positional encoding. For multi-resolution adaptive positional encoding, given multi-resolution input, the original positional encoding is first upsampled to the maximum resolution, and then uniformly sampled according to the resolution scaling. For position-constrained semantic feature reconstruction, a vision decoder is included to independently (token-wise) reconstruct each visual feature, thus preserving the model's original natural image encoding capabilities.

[0162] In the task-adaptive visual information extraction module, an input embedding layer is set up during the training phase: it is used to extract the required visual features for coarse-grained or fine-grained perception tasks respectively. These visual features are used as inputs into the language model to perform downstream tasks. A lightweight language decoder is also set up to decode the fine-grained features encoded by the model into detailed text information, such as a document-level optical character recognition (OCR) model, to assist in training the visual information extraction module.

[0163] During the training phase, a visual decoder and a language decoder were introduced to assist in training. The specific model structure of the additionally configured model during the training phase is described below.

[0164] First, the model input can include multi-scale input, meaning the model input can include images of various resolutions, thus adapting to various image resolution scenarios.

[0165] In this embodiment, multi-resolution adaptive position coding is introduced. By upsampling or downsampling the initial position embedding, the resulting position embedding matches the scale of the input image.

[0166] Unlike image recognition, which focuses on global feature extraction and classification, text recognition requires sophisticated local feature extraction and sequence prediction. ViT excels in global feature extraction but may perform poorly in text recognition. This is because ViT uses the learned positional embeddings for each input image patch, which in turn forces the model to always run at a constant resolution, typically chosen as 224*224, 256*256, or 336*336. Visual features encoded at these resolutions may lose crucial information, making it difficult to recognize characters in dense text in PDF documents, websites, or digital books. Simply cropping a high-resolution document image to a low-resolution input often disrupts the character order and may impair the recognition of letters at the cut boundaries. Therefore, to prevent these problems, embodiments of this application introduce multi-resolution adaptive positional encoding, which matches the number of positions corresponding to the maximum input size of the high-resolution image by interpolating or downsampling the original positional embedding. For images of different resolutions, the image after setting the positional embedding is randomly and uniformly sampled, outputting multiple blocks to meet the number of input blocks of the original model.

[0167] Secondly, for the language decoder, this application embodiment introduces a lightweight transformer structure as the language decoder to more efficiently predict the performance of the data output by the visual encoder in downstream tasks.

[0168] Traditional visual encoding schemes typically employ CLIP-style contrastive learning to align images with their linguistic descriptions. However, this approach has significant drawbacks in various scenarios. First, contrastive learning relies on a pre-trained CLIP text encoder to generate language embeddings. The maximum sequence length is 77 tags, which is insufficient for text recognition annotation, especially in dense documents. Second, this training paradigm is inadequate for the visual encoder to accurately capture each word, leading to unwanted information loss during encoding. Considering these factors, this application introduces a lightweight transformer decoder, such as the OPT-125M, to efficiently predict the text sequences presented in the input document. This application employs an input embedding layer to project visual features from the visual encoder into the embedding space of the language decoder. The input embedding layer is initialized with a two-layer Q-Former and has K learnable query tags.

[0169] In addition, to ensure that the visual features output by the visual encoder can be effectively projected into the language feature space, an image description task is introduced, which, together with the document OCR task, optimizes the language decoder to ensure that the learned visual features can be effectively projected into the language feature space, thereby enhancing the model's image understanding ability.

[0170] For visual decoders:

[0171] Fine-tuning the visual encoder solely on a document dataset can reduce its ability to encode the original images. Therefore, to mitigate this issue, this embodiment introduces a token-wise feature reconstruction task based on training on a natural image dataset, ensuring that the newly learned visual encoder retains its ability to encode visual concepts. For example, a visual decoder fπ(·) can be introduced, consisting of two fully connected layers with a GeLU activation function in the middle layer. The decoder processes each visual feature vector independently, more effectively preserving the inherent positional information of each feature. This embodiment utilizes a pre-trained encoding model as a teacher model, providing the original features for each natural image and offering supervisory signals for visual feature reconstruction.

[0172] 2. Pre-training phase

[0173] In this embodiment, multi-scale training is employed to effectively capture fine-grained information within densely packed documents. Images are processed at the original resolution of the visual encoder, while documents are processed at a higher resolution, such as 4x4 times the original image resolution. Furthermore, different loss functions are used for the input images and documents at different scales. During this training phase, image and text recognition are trained at a fixed scale: low resolution for images and high resolution for documents.

[0174] This application exemplarily refers to the training phase as Intra-scale Pretraining.

[0175] First, images of different resolutions are acquired for training, namely the first training set and the second training set mentioned above.

[0176] Let D be a carefully selected dataset containing 5 million samples, where here This represents a dataset of natural images with coarse captions (typically fewer than 30 words). These natural images are resized to their original scale r x 1, primarily derived from a concept caption dataset containing 3 million samples. Simultaneously, these natural images are resized to their original scale r x 4, also primarily derived from a concept caption dataset containing 3 million samples. This represents a document dataset with dense OCR data (each document image contains over 500 words). The document images are scaled to four times their original size (r), derived from a synthetic English PDF document dataset containing 2 million samples. The aforementioned images, scaled once or multiple times, can be used to create natural and document images at different resolutions.

[0177] For the input instructions to the language decoder, i.e., the language model, the instruction format can be pre-defined. It typically follows the format used in LVLM, where image tags serve as a prefix followed by the user instruction text. Specifically, two special tags can be used. The quotation marks "" and "" indicate the start and end positions of the image markers, respectively, followed by the user-inputted task instructions. For natural images, the prompt is set to "Add a caption to this image:", and the model outputs a language sequence summarizing the image's content. For document images, the prompt is set to "Read the text in this image:", and the model outputs sentences from the document line by line in reading order.

[0178] Referring to Figure 5 above, let's take the teacher model as an example, which is pre-trained using low-resolution images. During the training phase, if the input image is a low-resolution image, it is used as the input to both the teacher model and the visual encoding module. The visual features (vision tokens) output by the visual encoding module are used as the input to the visual decoder, which outputs student tokens. The teacher model outputs teacher tokens.

[0179] Teacher tokens are represented as We use a weighted sum of cosine distance Lcos and smoothed L1 loss Ll1 to force the newly learned student tokens to align with the teacher tokens. This method ensures consistency in vector direction and magnitude between the new and old features, as shown below:

[0180] Where C represents a subset of features randomly drawn from X′ to mitigate overfitting and enhance robustness during training. μ represents the loss weight scalar.

[0181] Meanwhile, visual tags generated by the visual encoder First, the input is fed into the input embedding layer, and then into the language decoder. In this embodiment, this process is denoted as f. φ (), where φ represents the parameters of the input embedding layer and the language decoder: {z1,…,z N}=f φ ({x1,…,x N},{q1,…,q K Then use the last hidden state at decoder time step t). Predict the language sequence y′={y′1,y′2,…,y′} of text tags using an autoregressive language decoder. T The cross-entropy loss used to train the autoregressive transformer decoder is defined as follows:

[0182] Where y = {y1, y2, ..., y} T} represents the target sequence, and the probability P() comes from the output of the softmax layer in the decoder.

[0183] The cross-entropy loss L between the language prediction output and the true value is used. lan and feature reconstruction loss L vis Optimization is performed to force the newly learned visual features to be similar to the original features. During training, the parameters of the visual encoding model are updated in reverse by minimizing the total loss function. For example, the updated parameters can be expressed as:

[0184] If the input image is a high-resolution image, it does not need to be used as input to the teacher model; that is, only L needs to be calculated. lan (y,y′), using cross-entropy loss to update model parameters, as represented as

[0185] Therefore, in this embodiment, for low-resolution images, a distillation method is used. A pre-selected, high-performing model is used as the teacher model, and the visual encoding model is used as the student model for distillation learning, allowing the visual encoding model to inherit the image encoding capabilities of the teacher model. For high-resolution images, a language decoder can be combined to assist training, enabling the visual features output by the visual encoding model to be mapped into the language feature space, thus allowing application in the downstream language model. This approach is applicable to scenarios with various input image resolutions and exhibits very strong generalization ability.

[0186] 3. Fine-tuning stage

[0187] After the internal scale pre-training phase, UNIT is able to handle image and text recognition at commonly used resolutions, namely low-resolution images and high-resolution documents.

[0188] In this embodiment, to improve the performance of the visual coding model when processing large-font text or large-size images and to enhance the model's robustness, a fine-tuning stage is added to fine-tune the visual coding model, thereby obtaining a more robust visual coding model. This process can be referred to as inter-scale fine-tuning in this embodiment.

[0189] First, construct another scale-swapping dataset. It contains 1 million samples for cross-scale fine-tuning, i.e., the third training set mentioned above. Dataset It includes natural images with detailed descriptions (each description contains more than 100 words). These images have been scaled to 4 times their original scale r and are primarily derived from the ShareGPT4V dataset, totaling 1 million samples. Additionally, this application embodiment creates a dataset... Contains 1 million document images, with an average font size of [missing information]. The document images are scaled to 4 times the original size. The dataset synthesized in this application embodiment comes from digital books, advertisements, and websites, primarily consisting of cropped pages containing short text and large font sizes. The same instruction prompts as during the training phase can be used.

[0190] Because teacher models typically perform better at low resolutions, when processing higher-resolution natural images, the number of visual tokens increases compared to lower resolutions, leading to gradient overflow during training, primarily due to token-wise feature reconstruction loss. To address this issue, this application's implementation randomly samples a set of visual tokens equal in number to the low-resolution input and discards other visual tokens to avoid gradient overflow and improve the feasibility of the solution.

[0191] To enhance the scale-robust recognition capability of images and text, some data from the aforementioned training phase can be retained, and then scale-transformed parameters can be introduced. During training, the model parameters are updated by minimizing the total loss function.

[0192] In addition, by combining the aforementioned training and fine-tuning phases, the UNIT can be further trained in conjunction with specific application scenarios, thereby enabling the visual features output by the UNIT to perform better in downstream tasks.

[0193] For example, as shown in Figure 6, during the training phase, the UNIT is trained using natural and document images at different resolutions, followed by external scale fine-tuning. The trained UNIT is then deployed in a larger model. Here, we take the construction of LVLMs based on the Vicuna-7B LLM as an example. The input embedding layer and LLM in the UNIT are fine-tuned using LoRA (low-rank matrix). When fine-tuning in a larger model, the dataset from the aforementioned training phase can be used, or it can be combined with the dataset used in the fine-tuning phase, thereby improving the UNIT's image encoding capabilities in real-world application scenarios.

[0194] Therefore, in this embodiment, a training phase and a fine-tuning phase are respectively set to improve the image processing capabilities of the UNIT. In the training phase, the UNIT is trained to have better information extraction capabilities at all scales. In the fine-tuning phase, images from specific scenarios are used to specifically train the model, thereby improving its robustness and resulting in a visual encoding model with better overall performance. Furthermore, the visual encoding model is deployed in a larger model and fine-tuned in conjunction with actual inference tasks, thereby further improving the performance of the visual encoding model in real-world application scenarios.

[0195] II. Reasoning Stage

[0196] After training the aforementioned model, a visual coding model that performs better at various image resolutions can be output. This visual coding model can then be deployed in application scenarios to perform inference tasks, such as processing image understanding or document analysis based on user input instructions.

[0197] Based on the architecture shown in Figure 5 above, the UNIT can be deployed in a device that performs inference tasks, specifically including the visual encoding module and the visual information extraction module of the UNIT.

[0198] For example, consider applying the visual encoding model output from the training phase to a large model.

[0199] Referring to Figure 7, this application provides a schematic diagram of an application scenario.

[0200] The visual encoder can be deployed within a multimodal LLMS, which can also house encoders of other modalities, such as the text sequence encoder shown in Figure 7. The input of the large multimodal model is connected to the visual encoder, which is then fed into the large model through intermediate connection layers, such as linear convolution, attention mechanisms, or QFormer. This allows visual features to be mapped to a shared multimodal representation space, converting visual signals into language signals, and obtaining the model output through a large language model.

[0201] The structure of the visual coding model provided in this application can be shown in Figure 8.

[0202] The visual coding model can specifically include a resolution-adaptive visual coding module and a task-adaptive visual information extraction module, which will be described below.

[0203] The resolution-adaptive visual encoding module (also referred to as the visual encoding module) takes an image as input. This embodiment does not limit the image resolution and can be used to extract features from images of any resolution, outputting visually encoded features, such as vision tokens. It efficiently encodes input images of different resolutions, enhancing the fine-grained information representation capability of feature encoding while maintaining global semantic understanding capabilities. By using the resolution-adaptive visual encoding module to process image inputs of different resolutions, it flexibly adapts to input data with a large range of resolutions from low to high.

[0204] Specifically, the visual encoding module can determine the location embedding based on the resolution of the input image, divide the input image into multiple blocks according to the location embedding, and then extract features from each block to output visually encoded features. The visual encoding module can pre-set an initial location embedding, which can be a matrix set during distillation learning in the training phase, representing the matrix used to partition the image. For example, as shown in Figure 9, the location embedding can be understood as a matrix used for image partitioning, with each element corresponding to a block. When the input image resolution is high, the initial location embedding can be interpolated to increase the number of matrix elements in the location embedding, resulting in more blocks during image partitioning. This ensures that the size of each block remains within the block size range that the model can handle during training, adapting to the processing of high-resolution images. When the input image resolution is low, the initial location embedding can be downsampled to reduce the number of matrix elements in the location embedding, resulting in fewer blocks during image partitioning. This ensures that the size of each block remains within the block size range that the model can handle during training, adapting to the processing of low-resolution images. For each image block, feature extraction can be performed using convolution operations or attention mechanisms to output visually encoded features.

[0205] The task-adaptive visual information extraction module efficiently extracts information from visual tokens for different tasks, compresses token length to improve computational efficiency, and, in conjunction with LLM decoding, produces natural language output that meets the requirements, i.e., the converted representation vector. Through this module, visual information is adaptively extracted for both coarse-grained and fine-grained perceptual tasks, maximizing the preservation of detail in high-resolution images. This enables more precise completion of user-specified tasks, especially for extracting effective information from densely texted documents to fulfill user instructions.

[0206] This can be understood as follows: the resolution-adaptive visual encoding module efficiently encodes input images at different resolutions while maintaining global semantic understanding and fine-grained spatial understanding capabilities; the task-adaptive visual information extraction module efficiently extracts information from visual tokens for different tasks, compresses token length, thereby improving computational efficiency, and works with a large language model to decode and obtain natural language output that meets the requirements.

[0207] Optionally, the visual encoding model obtained by the method provided in this application can also be applied to various scenarios, such as:

[0208] Intelligent Image Search and Annotation: The Visual Language Multimodal Large Model can provide more accurate and intelligent image search and annotation services by understanding image content and related text descriptions. This is significant for e-commerce platforms, social media applications, and content creation platforms, helping users find content they are interested in more easily.

[0209] Personalized content recommendation: Multimodal big data models that combine image and text information can better understand users' interests and preferences, thereby providing personalized content recommendation services. Whether in video streaming, news reading, or social networking platforms, this personalized recommendation can improve user retention and engagement.

[0210] Virtual Shopping Assistant: In the field of e-commerce, visual language multimodal big data models can be used to develop virtual shopping assistants, helping users find products that meet their needs based on images and text descriptions, and providing relevant suggestions and recommendations, thereby improving the user's shopping experience and purchase conversion rate.

[0211] Intelligent Customer Service and Question Answering Systems: This multimodal big model can be used to develop intelligent customer service and question answering systems, which can more accurately understand the questions raised by users and provide personalized answers and solutions based on relevant image and text information, thereby improving customer service efficiency and user satisfaction.

[0212] Intelligent Assistance Tools: In fields such as manufacturing, healthcare, and scientific research, visual language multimodal large models can serve as intelligent assistance tools to help people understand and analyze complex image and text data more quickly, thereby improving work efficiency and decision-making accuracy.

[0213] To facilitate the description of the effects achieved by the method provided in this application, the following detailed description is based on data results collected in a practical application scenario.

[0214] Based on the dimension of input data, scenes can be divided into scenes based on natural images and scenes based on document images.

[0215] For example, application scenarios based on document images can include:

[0216] Document Image OCR Task: The model of this application embodiment was evaluated on three publicly available OCR datasets: FUNSD (50 images), SROIE (347 images), and CORD (100 images), and F1 scores are reported. Due to the lack of large blocks of dense text in these datasets, two additional OCR evaluation datasets were created using English PDF files from arXiv. The first is SYN-L-val (200 images), consisting of PDF files with small fonts at a resolution of approximately 1k. The second is SYN-S-val (200 images), containing images cropped from these PDF files and resized to a lower resolution (e.g., 224) to produce larger fonts.

[0217] Document Image to Markdown Source Code Task: This task requires extracting structured text from document images, capturing the text's style and structural elements, and outputting it in Markdown format. During the cross-scale fine-tuning stage, this embodiment incorporates 1MB of Markdown data, enhancing the Markdown conversion capabilities of the model provided. Due to the lack of a publicly available Markdown conversion validation set MD-val, a dataset containing 82 images with Markdown format annotations can be constructed as the training set.

[0218] Combining the two scenarios described above, the UNIT output by the method provided in this application is compared with two ViT-based OCR models (Donut, Nougat) and two open-source visual encoders that support high-resolution input (Vary, RADIO). For example, the open-source weights of Donut-Base and Nougat are used for evaluation here. For both Vary and RADIO models, they are retrained on the dataset for fair comparison. The models are evaluated on public OCR datasets such as FUNSD, SROIE, CORD, and the synthetic datasets SYN-L-val and MD-val. As shown in Table 1 below, the model (method) is configured with different backbone networks, parameter counts (#params), and input resolutions. The method provided in this application consistently outperforms other methods in OCR and Markdown conversion tasks, except for Nougat, which is fully trained on Markdown data. Therefore, the method provided in this application has superior capabilities in encoding detailed sequential and spatial information in documents and predicts accurate OCR text extraction results. Based on its basic text recognition capabilities, the method of this application embodiment can be further adapted to highly specialized tasks, such as text recognition with localization (including natural images and plain text), Markdown conversion, and Chinese OCR, through fine-tuning with a small amount of data.

[0219] Table 1

[0220] For example, as shown in Figure 10, the document content in the document image can be converted into editable text, thus realizing the image-to-text conversion.

[0221] For example, in the case of document analysis, as shown in Figure 11, the user can input instructions, and the large model can combine the user's input instructions with the document content in the input image to output the corresponding analysis results.

[0222] For example, scenes based on natural images can include:

[0223] Zero-shot classification (ZS cls.): Calculate the feature similarity between the [CLS] tags of the visual encoder and the text features extracted by the corresponding CLIP text encoder. Evaluation is performed on the ImageNet-1K dataset, and top-1 accuracy is reported.

[0224] k-NN classification (kNN cls.): First, the [CLS] labels of the visual encoders for the ImageNet-1K training images are computed. Then, for each validation image, the label is selected using the weighted sum of the k nearest neighbor training vectors.

[0225] Semantic segmentation (Segm.): A linear head is added to the visual encoder, and training is performed with the encoder frozen. The AdamW optimizer is used with a learning rate of 1e-3. The segmentation mIoU (percentage) is computed on the ADE20k dataset following the MMSeg framework. A 512x512 input size is used for both training and evaluation.

[0226] Combining the aforementioned various natural image application scenarios, this paper compares the results with existing efficient visual encoders on ImageNet-1k zero-shot and kNN classification and semantic segmentation benchmarks. The comparison results are shown in Table 2 below. The introduced models (methods) can include EfficientViT-L1, Swinv2-s, ConvNext-B, MViTV2-B, NFNet-F3, MaxViT-B, OpenCLIP-H, RADIO-L, E-RADIO-L, RADIO-H, and UNIT provided in this application embodiment. All models share the same data, i.e., teacher knowledge, which is typically derived from DINOv2 and OpenCLIP-H. The compared models directly use these two models for feature reconstruction training. As can be seen from the results in Table 2, the UNIT method provided in this application embodiment performs best among all these models, demonstrating its ability to effectively preserve basic image coding capabilities.

[0227] Table 2

[0228] For example, a scenario for image analysis using a large model can be shown in Figure 12. Users can input commands for natural images, such as "What objects are in the image?", "What people are in the image?", "What animals are in the image?", or "What is in the image?". The large model performs language understanding on the image based on the user's input commands and outputs the corresponding analysis results.

[0229] Furthermore, in multimodal large-scale model applications, the visual encoder plays a crucial role in large visual language models (LVLMs), providing rich and detailed visual representations to achieve effective cross-modal understanding and reasoning. LVLMs can be built based on the Vicuna-7B LLM, using the methods provided in the embodiments of this application or other visual encoder scenarios. Typically, the processing of large models follows the processing procedure of LVLMs, such as LLaVA-1.5, and the visual encoder is frozen during processing, i.e., the parameters of the visual encoder are preserved. The input embedding layer and LLM are fine-tuned using LoRA. The multimodal pre-training process uses 4 million image-title data (randomly sampled from CC3M and LAION-COCO) and 200,000 document OCR data (randomly sampled from the training set). In the supervised fine-tuning (SFT) stage, the training sets of LLaVA-80k or LLaVA-CC665k, as well as DocVQA and ChartQA, are used as fine-tuning datasets. For natural images, use an input image resolution of 224x224; for documents, use a resolution of 896x896.

[0230] In this scenario, the analysis of the output results is shown in Table 3 below. The UNIT method provided in this application significantly outperforms the teacher model RADIO-H and other comparative models on document analysis tasks (including ChartQA, DocQA, and InfoVQA), while maintaining comparable performance on image understanding tasks. In document analysis tasks such as ChartQA and DocQA, UNIT can extract text from documents, charts, or websites, demonstrating its potential in practical document analysis applications.

[0231] Table 3

[0232] Typically, in general scenarios involving large multimodal models, user-instructed tasks often involve both coarse-grained and fine-grained perception. For example, an input image may contain both visual objects and text, and the user instruction requires the model to perceive and understand the content in the image, ultimately outputting results according to the task requirements. These scenarios all rely on a unified visual encoding model capable of both coarse-grained and fine-grained perception. This application provides a unified encoding scheme for multi-resolution, multi-task processing, capable of simultaneously handling multi-task training of multi-resolution inputs, thereby achieving unified visual encoding of both coarse-grained and fine-grained visual content. It can generate unified visual codes for input images that possess both coarse-grained and fine-grained perception capabilities, and accurately predict results based on user input instructions using a lightweight decoder.

[0233] The foregoing has described the method flow provided in the embodiments of this application. The following describes the structure of the apparatus for executing the method flow provided in this application.

[0234] Referring to Figure 13, a schematic diagram of the structure of a visual coding device provided in this application includes:

[0235] The acquisition module 1301 is used to acquire the input image;

[0236] The encoding module 1302 is used to input the input image into the visual encoding model and output visual encoding data. The visual encoding model is used to divide the input image into multiple image blocks according to the position embedding, extract features from each image block, and output visual encoding data based on the features of each image block and the corresponding position encoding. The position embedding is obtained by adjusting the initial position embedding according to the difference between the input image and the preset resolution.

[0237] In one possible implementation, the aforementioned visual coding model is specifically used for: upsampling the initial position embedding to obtain a position embedding when the resolution of the input image is higher than a preset resolution; dividing the input image into multiple blocks according to the position embedding, and encoding the position of each block to obtain the position code corresponding to each block.

[0238] In one possible implementation, the aforementioned visual coding model is further used to: downsample the initial position embedding to obtain a position embedding when the resolution of the input image is lower than a preset resolution; and use the initial position embedding as the position embedding when the resolution of the input image is a preset resolution.

[0239] In one possible implementation, the aforementioned visual coding model includes a visual coding module and a representation layer. The visual coding module is used to divide the input image into multiple image blocks according to the position embedding, extract features from each image block, and output visual coding features based on the features of each image block and the corresponding position encoding. The representation layer is used to obtain the representation vector corresponding to the visual coding features and output visual coding data.

[0240] In one possible implementation, the aforementioned visual coding model is obtained by distillation learning using a pre-trained coding model.

[0241] Referring to Figure 14, this application provides a schematic diagram of the structure of a visual coding model training device, including:

[0242] The acquisition module 1401 is used to acquire a first training set and a second training set. The image resolutions in the first training set and the second training set are different. The image resolutions in the first training set include a preset resolution.

[0243] Training module 1402 is used to train a visual coding model using a first training set, a second training set, and a pre-trained coding model to obtain a trained visual coding model. The pre-trained coding model and the first training set are used to perform distillation learning on the visual coding model. The image resolution used when the pre-trained coding model is trained includes a preset resolution.

[0244] The visual encoding model is used to divide the input image into multiple image blocks according to the position embedding, extract features from each image block, and output visual encoding data based on the features of each image block and the corresponding position encoding. The position embedding is obtained by adjusting the initial position embedding according to the difference between the input image and the preset resolution.

[0245] The effects achieved by the fourth aspect or any optional implementation of the fourth aspect can be referred to the description of the second aspect or any optional implementation of the second aspect, and will not be repeated below.

[0246] In one possible implementation, one iteration of the aforementioned training process includes: using images from a first training set or a second training set as input to the visual coding model obtained in the previous iteration, and outputting first visual coding data; updating the visual coding model obtained in the previous iteration based on the first visual coding data, to obtain the visual coding model updated in the current iteration; wherein, when images from the first training set are used as input to the visual coding model obtained in the previous iteration, the images from the first training set are used as input to both the coding model and the visual coding model obtained in the previous iteration, and a distillation loss value is calculated based on the difference between the output of the coding model and the output of the visual coding model obtained in the previous iteration, and the distillation loss value is used to update the visual coding model obtained in the previous iteration.

[0247] In one possible implementation, the aforementioned training module 1402 is specifically used for: inputting the first visual coding data into the language decoding model and outputting the decoding result; calculating the loss value based on the difference between the decoding result and the annotation corresponding to the image input into the visual coding model; and updating the visual coding model obtained in the previous iteration based on the loss value to obtain the visual coding model updated in the current iteration.

[0248] In one possible implementation, the aforementioned training module 1402 is specifically used to: take the output of the visual encoding model obtained in the previous iteration as the input of the visual decoder to obtain a visual feature vector; and calculate the distillation loss value based on the difference between the output of the encoding model and the visual feature vector.

[0249] In one possible implementation, the aforementioned acquisition module 1402 is further configured to: acquire an initial dataset, which includes multiple images and corresponding annotations for each image, the multiple images including natural images and document images; and scale the images in the initial training set at least once to obtain a first training set and a second training set.

[0250] In one possible implementation, the aforementioned training module 1402 is further configured to: fine-tune the updated visual coding model using a third training set to obtain a fine-tuned visual coding model, wherein the third training set includes natural images and document images, and the fonts in the document images in the third training set are larger than the fonts in the document images in the first and second training sets, and the resolution of the natural images in the third training set is greater than the resolution of the natural images in the first and second training sets.

[0251] Figure 15 shows a schematic diagram of the hardware structure of a computing device 150 provided in an embodiment of this application. This computing device 150 can be used to implement the steps of the visual coding model or visual coding model training device described in Figures 3 to 12.

[0252] The computing device 150 shown in Figure 15 may include a processor 1501, a memory 1502, a communication interface 1503, and a bus 1504. The processor 1501, the memory 1502, and the communication interface 1503 can be connected to each other via the bus 1504.

[0253] Processor 1501 is the control center of computing device 150. It can be a general-purpose central processing unit (CPU) or other general-purpose processors. The general-purpose processor can be a microprocessor or any conventional processor, such as a GPU or NPU, and can be adapted to the actual application scenario.

[0254] As an example, processor 1501 may include one or more CPUs, and may also include other processors, such as the CPU, NPU or GPU shown in Figure 15.

[0255] The memory 1502 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0256] In one possible implementation, the memory 1502 may exist independently of the processor 1501. The memory 1502 can be connected to the processor 1501 via a bus 1504 and is used to store data, instructions, or program code. When the processor 1501 calls and executes the instructions or program code stored in the memory 1502, it can implement the methods provided in the embodiments of this application, such as the methods shown in Figures 3 to 12.

[0257] In another possible implementation, the memory 1502 can also be integrated with the processor 1501.

[0258] Communication interface 1503 is used for connecting computing device 150 to other devices via a communication network, which may be Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. Communication interface 1503 may include a receiving unit for receiving data and a transmitting unit for transmitting data.

[0259] Bus 1504 can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus. This bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in Figure 15, but this does not indicate that there is only one bus or one type of bus.

[0260] It should be noted that the structure shown in FIG15 does not constitute a limitation on the computing device 150. In addition to the components shown in FIG15, the computing device 150 may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0261] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0262] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0263] This application also provides a computer-readable storage medium storing a program for training a model or performing inference tasks, which, when run on a computer, causes the computer to perform all or part of the steps in the methods described in the embodiments shown in Figures 3 to 12 above.

[0264] This application also provides a digital processing chip. This digital processing chip integrates circuitry for implementing the aforementioned processor or processor functions, and one or more interfaces. When the digital processing chip integrates a memory, it can perform the method steps of any one or more of the foregoing embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory via a communication interface. The digital processing chip implements the method steps of any one or more of the foregoing embodiments based on the program code stored in the external memory.

[0265] This application also provides a computer program product comprising one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0266] The data comparison device provided in this application embodiment can be a chip, which includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip in the server to execute the method described in the embodiments shown in Figures 3-12 above. Optionally, the storage unit is a storage unit within the chip, such as a register or cache. The storage unit can also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, such as random access memory (RAM).

[0267] Specifically, the aforementioned processing unit or processor can be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0268] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0269] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0270] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0271] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0272] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. The term "and / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Additionally, the character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps in this application does not imply that the steps in the method flow must be executed in the time / logical order indicated by the naming or numbering. The execution order of the named or numbered process steps can be changed according to the technical purpose to be achieved, as long as the same or similar technical effect can be achieved. The division of modules in this application is a logical division. In actual applications, there may be other division methods. For example, multiple modules may be combined into or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the modules shown or discussed may be through some ports, and the indirect coupling or communication connection between modules may be electrical or other similar forms, which are not limited in this application. Furthermore, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in multiple circuit modules. Some or all of the modules can be selected to achieve the purpose of the solution in this application according to actual needs.

Claims

1. A visual encoding method, characterized in that, include: Obtain the input image; The input image is input to a visual coding model, and visual coding data is output. The visual coding model is used to divide the input image into multiple image blocks according to the position embedding, extract features from each image block, and output the visual coding data based on the features of each image block and the corresponding position code. The position embedding includes adjusting the initial position embedding according to the difference between the input image and the preset resolution.

2. The method according to claim 1, characterized in that, The visual coding model is specifically used for: If the resolution of the input image is higher than the preset resolution, the initial position embedding is upsampled to obtain the position embedding; The input image is divided into multiple blocks based on the location embedding, and the position of each block is encoded to obtain the position code corresponding to each block.

3. The method according to claim 2, characterized in that, The visual coding model is also used for: If the resolution of the input image is lower than the preset resolution, the initial position embedding is downsampled to obtain the position embedding. When the resolution of the input image is the preset resolution, the initial position embedding is used as the position embedding.

4. The method according to any one of claims 1-3, characterized in that, The visual coding model includes a visual coding module and a representation layer. The visual coding module is used to divide the input image into multiple image blocks according to the position embedding, extract features from each image block, and output visual coding features based on the features of each image block and the corresponding position encoding. The representation layer is used to obtain the representation vector corresponding to the visual coding features and output visual coding data.

5. The method according to any one of claims 1-4, characterized in that, The visual coding model is obtained by distillation learning using a pre-trained coding model.

6. A method for training a visual coding model, characterized in that, include: Obtain a first training set and a second training set. The image resolution in the first training set is different from that in the second training set. The image resolution in the first training set is a preset resolution. The visual coding model is trained using the first training set, the second training set, and the pre-trained coding model to obtain the trained visual coding model. The pre-trained coding model and the first training set are used to perform distillation learning on the visual coding model. The image resolution used when the pre-trained coding model is trained is the preset resolution. The visual encoding model is used to divide the input image into multiple image blocks according to the position embedding, extract features from each image block, and output visual encoding data based on the features of each image block and the corresponding position encoding. The position embedding includes adjusting the initial position embedding according to the difference between the input image and the preset resolution.

7. The method according to claim 6, characterized in that, One of the iterations in the training process includes: The images in the first training set or the second training set are used as input to the visual coding model obtained in the previous iteration, and the first visual coding data is output. Update the visual coding model obtained in the previous iteration based on the first visual coding data to obtain the visual coding model updated in the current iteration. In this case, where the images in the first training set are used as input to the visual coding model obtained in the previous iteration, The images in the first training set are used as inputs to the encoding model and the visual encoding model obtained in the previous iteration, respectively. The distillation loss value is calculated based on the difference between the output of the encoding model and the output of the visual encoding model obtained in the previous iteration. The distillation loss value is used to update the visual encoding model obtained in the previous iteration.

8. The method according to claim 7, characterized in that, The step of updating the visual coding model obtained in the previous iteration based on the first visual coding data to obtain the visual coding model updated in the current iteration includes: The first visual encoded data is input into the language decoding model, and the decoding result is output. The loss value is calculated based on the difference between the decoding result and the annotation corresponding to the image input to the visual coding model; The visual coding model obtained in the previous iteration is updated based on the loss value to obtain the visual coding model updated in the current iteration.

9. The method according to claim 7 or 8, characterized in that, The step of calculating the distillation loss value based on the difference between the output of the coding model and the output of the visual coding model obtained in the previous iteration includes: The output of the visual coding model obtained in the previous iteration is used as the input of the visual decoder to obtain the visual feature vector; The distillation loss value is calculated based on the difference between the output of the encoding model and the visual feature vector.

10. The method according to any one of claims 6-9, characterized in that, The method further includes: Obtain an initial dataset, which includes multiple images and corresponding annotations for each image, including natural images and document images; The images in the initial training set are scaled at least once to obtain the first training set and the second training set.

11. The method according to claim 10, characterized in that, The method further includes: The updated visual coding model is fine-tuned using a third training set to obtain a fine-tuned visual coding model. The third training set includes natural images and document images, and the fonts in the document images in the third training set are larger than the fonts in the document images in the first and second training sets. The resolution of the natural images in the third training set is greater than the resolution of the natural images in the first and second training sets.

12. A visual encoding device, characterized in that, include: The acquisition module is used to acquire the input image; An encoding module is used to input the input image into a visual encoding model and output visual encoding data. The visual encoding model is used to divide the input image into multiple image blocks according to the position embedding, extract features from each image block, and output the visual encoding data based on the features of each image block and the corresponding position encoding. The position embedding includes adjusting the initial position embedding according to the difference between the input image and a preset resolution.

13. The apparatus according to claim 12, characterized in that, The visual coding model is specifically used for: If the resolution of the input image is higher than the preset resolution, the initial position embedding is upsampled to obtain the position embedding; The input image is divided into multiple blocks based on the location embedding, and the position of each block is encoded to obtain the position code corresponding to each block.

14. The apparatus according to claim 13, characterized in that, The visual coding model is also used for: If the resolution of the input image is lower than the preset resolution, the initial position embedding is downsampled to obtain the position embedding. When the resolution of the input image is the preset resolution, the initial position embedding is used as the position embedding.

15. The apparatus according to any one of claims 12-14, characterized in that, The visual coding model includes a visual coding module and a representation layer. The visual coding module is used to divide the input image into multiple image blocks according to the position embedding, extract features from each image block, and output visual coding features based on the features of each image block and the corresponding position encoding. The representation layer is used to obtain the representation vector corresponding to the visual coding features and output visual coding data.

16. The apparatus according to any one of claims 12-15, characterized in that, The visual coding model is obtained by distillation learning using a pre-trained coding model.

17. A visual coding model training device, characterized in that, include: The acquisition module is used to acquire a first training set and a second training set, wherein the image resolution in the first training set is different from the image resolution in the second training set, and the image resolution in the first training set is a preset resolution. The training module is used to train the visual coding model using the first training set, the second training set, and the pre-trained coding model to obtain the trained visual coding model. The pre-trained coding model and the first training set are used to perform distillation learning on the visual coding model. The image resolution used when the pre-trained coding model is trained is the preset resolution. The visual encoding model is used to divide the input image into multiple image blocks according to the position embedding, extract features from each image block, and output visual encoding data based on the features of each image block and the corresponding position encoding. The position embedding includes adjusting the initial position embedding according to the difference between the input image and the preset resolution.

18. The apparatus according to claim 17, characterized in that, One of the iterations in the training process includes: The images in the first training set or the second training set are used as input to the visual coding model obtained in the previous iteration, and the first visual coding data is output. Update the visual coding model obtained in the previous iteration based on the first visual coding data to obtain the visual coding model updated in the current iteration. In this case, where the images in the first training set are used as input to the visual coding model obtained in the previous iteration, The images in the first training set are used as inputs to the encoding model and the visual encoding model obtained in the previous iteration, respectively. The distillation loss value is calculated based on the difference between the output of the encoding model and the output of the visual encoding model obtained in the previous iteration. The distillation loss value is used to update the visual encoding model obtained in the previous iteration.

19. The apparatus according to claim 18, characterized in that, The training module is specifically used for: The first visual encoded data is input into the language decoding model, and the decoding result is output. The loss value is calculated based on the difference between the decoding result and the annotation corresponding to the image input to the visual coding model; The visual coding model obtained in the previous iteration is updated based on the loss value to obtain the visual coding model updated in the current iteration.

20. The apparatus according to claim 18 or 19, characterized in that, The training module is specifically used for: The output of the visual coding model obtained in the previous iteration is used as the input of the visual decoder to obtain the visual feature vector; The distillation loss value is calculated based on the difference between the output of the encoding model and the visual feature vector.

21. The apparatus according to any one of claims 17-20, characterized in that, The acquisition module is also used for: Obtain an initial dataset, which includes multiple images and corresponding annotations for each image, including natural images and document images; The images in the initial training set are scaled at least once to obtain the first training set and the second training set.

22. The apparatus according to claim 21, characterized in that, The training module is also used for: The updated visual coding model is fine-tuned using a third training set to obtain a fine-tuned visual coding model. The third training set includes natural images and document images, and the fonts in the document images in the third training set are larger than the fonts in the document images in the first and second training sets. The resolution of the natural images in the third training set is greater than the resolution of the natural images in the first and second training sets.

23. A visual encoding device, characterized in that, The method comprises: a processor coupled to a memory, the memory storing a program, wherein when the program instructions stored in the memory are executed by the processor, the method of any one of claims 1-5 is implemented.

24. A visual coding model training device, characterized in that, The method comprises: a processor coupled to a memory, the memory storing a program, wherein when the program instructions stored in the memory are executed by the processor, the method of any one of claims 6-11 is implemented.

25. A computer-readable storage medium comprising a program, when executed by a processing unit, performing the steps of the method as claimed in any one of claims 1-5 or 6-11.

26. A computer program product, characterized in that, The computer program product includes software code for performing the steps of the method as described in any one of claims 1-5 or 6-11.

Images