Image processing method and apparatus, electronic device, storage medium, and program product

Abstract

The present disclosure provides an image processing method and device, electronic equipment, storage medium and program product, which generates a global view image and a plurality of local patch images by dynamically determining a target preset resolution according to the original resolution of an input image and a preset fixed inference resolution. Thus, a single fixed resolution visual model can simultaneously obtain the overall semantics and high-resolution local details of an image without recompiling a large resolution model. Furthermore, the global position code corresponding to the original image space position of each local visual token is calculated, so that the subgraph visual tokens after segmentation maintain the spatial continuous layout of the original image in the language model input sequence, avoiding the position relationship fragmentation caused by independent input of subgraphs. The global visual features, local visual features and global position codes are jointly constructed to form an input sequence for inference, realizing efficient and accurate inference on images of any resolution while maintaining the output effect under high-resolution input.

Description

Technical Field

[0001] This disclosure relates to image reasoning technology, and in particular to an image processing method and apparatus, electronic devices, storage media, and program products. Background Technology

[0002] Multimodal models, used in document understanding, image question answering, and complex scene analysis, often need to utilize both global semantic information and local detail information of images simultaneously. For high-resolution images, reasoning based solely on a single low-resolution input can easily lead to the loss of crucial details such as text, tables, and local targets.

[0003] Current NPU (Neural Processing Unit) inference typically relies on pre-compiled models with a fixed input resolution. Directly compiling high-resolution models leads to problems such as complex model compilation, high resource consumption, and high deployment costs. If images of different sizes are uniformly scaled and filled to a fixed resolution, high-resolution images suffer significant loss of detail, while low-resolution images generate a lot of invalid padding.

[0004] Therefore, a high-resolution image processing method suitable for fixed-resolution NPU inference environments is needed to solve the above problems, namely: to achieve efficient inference on images of arbitrary resolution without recompiling the high-resolution model, and to maintain the output effect under high-resolution input. Summary of the Invention

[0005] In view of the above-mentioned problems in related technologies, this disclosure provides an image processing method and apparatus, an electronic device, a storage medium, and a program product.

[0006] In a first aspect, this disclosure provides an image processing method applied to a multimodal model, the multimodal model having a preset fixed inference resolution, the method comprising:

[0007] In response to receiving an input image, a target preset resolution is determined based on the original resolution of the input image and a preset fixed inference resolution, so as to generate a global view image and multiple local block images that match the target preset resolution for the input image.

[0008] The global view image is visually encoded to obtain the corresponding global visual features;

[0009] Visual encoding is performed on the multiple local image blocks to obtain the corresponding local visual features;

[0010] Based on the spatial position of each of the local block images in the input image, determine the global position encoding of each visual token in the corresponding local visual features;

[0011] Based on the global visual features, each of the local visual features, and the global position encoding of each visual token, the input sequence of the multimodal model is constructed and inference is performed to output the image inference result.

[0012] As an optional embodiment, in response to receiving an input image, determining a target preset resolution based on the original resolution of the input image and a preset fixed inference resolution, and generating a global view image and multiple local block images matching the target preset resolution for the input image, includes:

[0013] Obtain the original resolution of the input image, wherein the original resolution includes the length of the input image, and the length of the input image is the larger of the width and height of the input image;

[0014] Based on the original resolution of the input image and the preset fixed inference resolution, a target preset resolution is determined. The target preset resolution is an integer multiple of the preset fixed inference resolution and does not exceed the long side dimension of the original resolution.

[0015] The input image is scaled and filled to the preset fixed inference resolution to generate the global view image;

[0016] Based on the target preset resolution and the preset fixed inference resolution, the input image is segmented to obtain multiple non-overlapping local block images.

[0017] As an optional embodiment, scaling and filling the input image to the preset fixed inference resolution to generate the global view image includes:

[0018] The input image is scaled proportionally to obtain a first intermediate image, wherein the long side of the first intermediate image is equal to the long side of the preset fixed inference resolution.

[0019] The blank areas of the intermediate image are filled so that the resolution of the filled image reaches the preset fixed inference resolution, thereby obtaining the global view image. The blank areas are the edge regions formed between the short side size of the intermediate image and the short side size of the preset fixed inference resolution.

[0020] As an optional embodiment, the step of segmenting the input image based on the target preset resolution and the preset fixed inference resolution to obtain multiple non-overlapping local block images includes:

[0021] The input image is scaled proportionally to obtain a second intermediate image, wherein the long side of the second intermediate image is equal to the long side of the target preset resolution;

[0022] The number of rows and columns to be segmented are determined based on the size of the second intermediate image and the size of the preset fixed inference resolution.

[0023] According to the number of rows and columns to be divided, the second intermediate image is evenly divided into multiple non-overlapping image blocks;

[0024] The size of each image block is adjusted to the preset fixed inference resolution to obtain the multiple local block images.

[0025] As an optional embodiment, determining the global position encoding of each visual token in the corresponding local visual features based on the spatial position of each of the local block images in the input image includes:

[0026] Determine the absolute position coordinates of each of the local image blocks, wherein the absolute position coordinates are the position coordinates of each local image block in the global coordinate space after the input image is scaled proportionally to the target preset resolution;

[0027] Based on the absolute position coordinates of each of the local segment images and the preset patch size, a global height index and a global width index are calculated for each visual token in the local segment image as the global position code of the visual token. The patch size is the width and height of the pixel block used when the local segment image is divided into non-overlapping visual tokens.

[0028] As an optional embodiment, determining the absolute position coordinates of each of the said partial image blocks includes:

[0029] Obtain the grid position index of each of the local block images. The grid position index is the row number and column number used when the input image is scaled proportionally to the target preset resolution and then uniformly divided according to the size of the preset fixed inference resolution.

[0030] Based on the grid position index and the size of the preset fixed inference resolution, the horizontal and vertical coordinates of the preset reference point of the local block image in the global coordinate space are calculated as the absolute position coordinates of the local block image.

[0031] As an optional embodiment, the step of calculating a global height index and a global width index for each visual token in the local segment image based on the absolute position coordinates of each of the local segment images and a preset patch size, as the global position encoding of the visual token, includes:

[0032] According to the preset patch size, each local block image is divided into multiple non-overlapping visual tokens, and the local row number and local column number of each visual token within the local block image are determined.

[0033] Based on the absolute position coordinates of the local block image, the preset patch size, the local row number, and the local column number, the global height index and global width index of the visual token in the global coordinate space are calculated as the global position encoding of the visual token.

[0034] As an optional embodiment, the step of calculating the global height index and global width index of the visual token in the global coordinate space based on the absolute position coordinates of the local patch image, the preset patch size, the local row number, and the local column number, to serve as the global position encoding of the visual token, includes:

[0035] Divide the ordinate of the absolute position coordinates of the local block image by the preset patch size and round down to obtain the first rounded result;

[0036] Add the first rounded result to the local row number to obtain the global height index of the visual token in the global coordinate space;

[0037] Divide the x-coordinate of the absolute position coordinates of the local block image by the preset patch size and round down to obtain the second rounding result;

[0038] Add the second rounding result to the local column number to obtain the global width index of the visual token in the global coordinate space;

[0039] The global height index and global width index of the visual token in the global coordinate space are determined as the global position code of the visual token.

[0040] As an optional embodiment, the step of constructing the input sequence of the multimodal model based on the global visual features, each of the local visual features, and the global position encoding of each visual token, and performing inference to output the image inference result includes:

[0041] According to the preset token arrangement order, determine the global visual token corresponding to the global visual feature, the visual token in each of the local visual features, and the token interval occupied in the input sequence of the multimodal model.

[0042] The global visual features and each of the local visual features are written into the corresponding token intervals to obtain the input sequence of filled feature vectors;

[0043] Based on the global position encoding of each visual token, the position encoding of the corresponding token position in the input sequence of the filled feature vector is determined, and an input sequence with position encoding is obtained.

[0044] The input sequence with positional encoding is submitted to the multimodal model, and the positional encoding is used as positional information for pre-filling and decoding to output image inference results.

[0045] As an optional embodiment, the multimodal model includes a visual model and a language model, and the method includes:

[0046] The visual model performs visual encoding on a preset number of local block images from the plurality of local block images to obtain corresponding local visual features. Based on the spatial position of each local block image in the input image, the global position encoding of each visual token in the corresponding local visual features is determined, so as to transmit the local visual features corresponding to the preset number of local block images and the global position encoding of each visual token to the language model.

[0047] The language model receives the local visual features of the preset number of local block images and the global position code of each visual token, and performs pre-filling calculation on the corresponding token interval according to the global position code to obtain the corresponding pre-filled block.

[0048] Repeat the above process until all the local block images have been processed, and obtain all the pre-filled blocks corresponding to all local block images;

[0049] The language model is used to decode the complete input sequence consisting of all pre-filled blocks, and the image reasoning result is output.

[0050] A second aspect of this disclosure provides an image processing apparatus applied to a multimodal model, the multimodal model having a preset fixed inference resolution, the apparatus comprising:

[0051] A preset resolution determination module is used to respond to receiving an input image and determine a target preset resolution based on the original resolution of the input image and a preset fixed inference resolution, so as to generate a global view image and multiple local block images that match the target preset resolution for the input image.

[0052] The first encoding module is used to perform visual encoding on the global view image to obtain the corresponding global visual features;

[0053] The second encoding module is used to perform visual encoding on the multiple local block images respectively to obtain the corresponding local visual features;

[0054] The third encoding module is used to determine the global position encoding of each visual token in the corresponding local visual features based on the spatial position of each of the local block images in the input image.

[0055] The inference module is used to construct the input sequence of the multimodal model and perform inference based on the global visual features, each of the local visual features and the global position encoding of each visual token, and output the image inference result.

[0056] As an optional embodiment, the preset resolution determination module includes:

[0057] An acquisition unit is configured to acquire the original resolution of the input image, wherein the original resolution includes the length of the input image, and the length of the input image is the larger of the width and height of the input image;

[0058] The determining unit is configured to determine a target preset resolution based on the original resolution of the input image and the preset fixed inference resolution, wherein the target preset resolution is an integer multiple of the preset fixed inference resolution and does not exceed the long side dimension of the original resolution;

[0059] A global view generation unit is used to scale and fill the input image to the preset fixed inference resolution to generate the global view image;

[0060] The local segmented image generation unit is used to segment the input image based on the target preset resolution and the preset fixed inference resolution to obtain multiple non-overlapping local segmented images.

[0061] As an optional embodiment, the global view generation unit includes:

[0062] The first scaling subunit is used to scale the input image proportionally to obtain a first intermediate image, wherein the long side dimension of the first intermediate image is equal to the long side dimension in the preset fixed inference resolution.

[0063] A filling subunit is used to fill the blank area of ​​the intermediate image so that the resolution of the filled image reaches the preset fixed inference resolution, thereby obtaining the global view image. The blank area is the edge area formed between the short side dimension of the intermediate image and the short side dimension of the preset fixed inference resolution.

[0064] As an optional embodiment, the local block image generation unit includes:

[0065] The second scaling subunit is used to scale the input image proportionally to obtain a second intermediate image, wherein the long side of the second intermediate image is equal to the long side of the target preset resolution.

[0066] The segmentation parameter determination subunit is used to determine the number of segmentation rows and columns based on the size of the second intermediate image and the size of the preset fixed inference resolution.

[0067] The image segmentation unit is used to uniformly divide the second intermediate image into multiple non-overlapping image blocks according to the number of segmentation rows and the number of segmentation columns;

[0068] The resolution adjustment subunit is used to adjust the size of each image block to the preset fixed inference resolution to obtain the multiple local block images.

[0069] As an optional embodiment, the third encoding module includes:

[0070] The position coordinate determination unit is used to determine the absolute position coordinates of each of the local block images, wherein the absolute position coordinates are the position coordinates of each local block image in the global coordinate space after the input image is scaled proportionally to the target preset resolution;

[0071] The position encoding unit is used to calculate a global height index and a global width index for each visual token in the local segment image based on the absolute position coordinates of each of the local segment images and a preset patch size, as the global position encoding of the visual token, wherein the patch size is the width and height of the pixel block used when the local segment image is divided into non-overlapping visual tokens.

[0072] As an optional embodiment, the position coordinate determination unit includes:

[0073] The acquisition subunit is used to acquire the grid position index of each of the local block images. The grid position index is the row number and column number used when the input image is scaled proportionally to the target preset resolution and then uniformly divided according to the size of the preset fixed inference resolution.

[0074] The calculation subunit is used to calculate the horizontal and vertical coordinates of a preset reference point of the local block image in the global coordinate space based on the grid position index and the size of the preset fixed inference resolution, as the absolute position coordinates of the local block image.

[0075] As an optional embodiment, the position encoding unit includes:

[0076] The row and column sequence determination subunit is used to divide each local block image into multiple non-overlapping visual tokens according to the preset patch size, and to determine the local row number and local column number of each visual token within the local block image.

[0077] The position encoding determination subunit is used to calculate the global height index and global width index of the visual token in the global coordinate space based on the absolute position coordinates of the local block image, the preset patch size, the local row number and the local column number, so as to serve as the global position encoding of the visual token.

[0078] As an optional embodiment, the position encoding determination subunit is further configured to:

[0079] Divide the ordinate of the absolute position coordinates of the local block image by the preset patch size and round down to obtain the first rounded result;

[0080] Add the first rounded result to the local row number to obtain the global height index of the visual token in the global coordinate space;

[0081] Divide the x-coordinate of the absolute position coordinates of the local block image by the preset patch size and round down to obtain the second rounding result;

[0082] Add the second rounding result to the local column number to obtain the global width index of the visual token in the global coordinate space;

[0083] The global height index and global width index of the visual token in the global coordinate space are determined as the global position code of the visual token.

[0084] As an optional embodiment, the inference module includes:

[0085] The token interval determination unit is used to determine, according to a preset token arrangement order, the global visual token corresponding to the global visual feature and each visual token in each of the local visual features, the token interval occupied in the input sequence of the multimodal model.

[0086] A visual feature writing unit is used to write the global visual features and each of the local visual features into the corresponding token intervals to obtain an input sequence of filled feature vectors.

[0087] A sequence position encoding unit is used to determine the position encoding of the corresponding token position in the input sequence of the filled feature vector based on the global position encoding of each visual token, so as to obtain an input sequence with position encoding.

[0088] The inference unit is used to submit the input sequence with position encoding to the multimodal model, pre-fill and decode the position encoding as position information, and output the image inference result.

[0089] As an optional embodiment, the multimodal model includes a visual model and a language model, and the device further includes:

[0090] The second encoding module performs visual encoding on a preset number of local block images from the plurality of local block images by calling the visual model, thereby obtaining the corresponding local visual features;

[0091] The third encoding module, by calling the visual model, determines the global position encoding of each visual token in the corresponding local visual features based on the spatial position of each of the local block images in the input image, so as to transmit the local visual features corresponding to the preset number of local block images and the global position encoding of each visual token to the language model.

[0092] The inference module receives the local visual features of the preset number of local block images and the global position code of each visual token by calling the language model, so as to pre-fill the corresponding token interval according to the global position code and obtain the corresponding pre-filled block.

[0093] The second encoding module, the third encoding module, and the inference module repeat the above process until all the multiple local block images are processed, and all pre-filled blocks corresponding to all local block images are obtained;

[0094] The inference module also decodes the complete input sequence consisting of all pre-filled blocks by calling the language model, and outputs the image inference result.

[0095] A third aspect of this disclosure provides an electronic device, including:

[0096] Memory, used to store computer program products;

[0097] A processor for executing a computer program product stored in memory, wherein when the computer program product is executed, it implements the method of any of the first aspects described above.

[0098] A fourth aspect of this disclosure provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the method of any one of the first aspects described above.

[0099] The fifth aspect of this disclosure provides a computer program product including computer program instructions that, when executed by a processor, implement any of the methods described in the first aspect above.

[0100] In summary, by dynamically determining the target preset resolution based on the original resolution of the input image and the preset fixed inference resolution, dynamically adapting to the input size, and generating a global view image and multiple local block images respectively, a single fixed-resolution visual model can simultaneously obtain the overall semantics (global view) and high-resolution local details (local blocks) of the input image without recompiling a high-resolution model. Furthermore, by calculating the global positional encoding corresponding to the spatial location of each local visual token in the original image, it is ensured that the visual tokens of each sub-image maintain the spatial continuity of the original image in the language model input sequence, avoiding the fragmentation of positional relationships caused by simply treating sub-images as independent multi-image inputs. Finally, the global visual features, local visual features, and their global positional encoding are used to construct the input sequence and perform inference, outputting the image inference result. This achieves efficient and accurate inference on images of arbitrary resolution without increasing deployment costs, while maintaining the output effect under high-resolution input.

[0101] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0102] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0103] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0104] Figure 1 This is a schematic flowchart of an image processing method according to an exemplary embodiment of the present disclosure.

[0105] Figure 2 This is a schematic flowchart of an image processing method as another exemplary embodiment of the present disclosure.

[0106] Figure 3 This is a schematic diagram of an input image, a global view image, and a partial block image, which are further exemplary embodiments of this disclosure.

[0107] Figure 4 This is a schematic flowchart of an image processing method for another exemplary embodiment of the present disclosure.

[0108] Figure 5 This is a schematic diagram illustrating the alignment of absolute position and position encoding of a figure, which is another exemplary embodiment of this disclosure.

[0109] Figure 6 This is a schematic flowchart of an image processing method for another exemplary embodiment of the present disclosure.

[0110] Figure 7 This is a schematic flowchart of an image processing method for another exemplary embodiment of the present disclosure.

[0111] Figure 8 This is a schematic diagram of asynchronous processing as another exemplary embodiment of the present disclosure.

[0112] Figure 9 This is a schematic diagram of the overall architecture of an image processing method as another exemplary embodiment of the present disclosure.

[0113] Figure 10 This is a structural block diagram of an image processing apparatus according to an exemplary embodiment of the present disclosure.

[0114] Figure 11 This is a structural block diagram of an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0115] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0116] The following description of at least one exemplary embodiment is merely illustrative and is not intended to limit the scope of this disclosure or its application or use.

[0117] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0118] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0119] This disclosure can be applied to electronic devices such as computer systems / servers, which can operate with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with electronic devices such as computer systems / servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0120] Electronic devices such as computer systems / servers can be described in the general context of computer system executable instructions (such as program modules) executed by the computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are performed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0121] In order to accurately describe the technical content in this disclosure, and to accurately understand the embodiments of this disclosure, the terms used in the embodiments of this disclosure are explained or defined as follows:

[0122] NPU (Neural Processing Unit): A hardware processor specifically designed to accelerate neural network computations, commonly used to speed up inference in deep learning models, characterized by high computing power and low power consumption. In this embodiment, both the visual model and the language model can perform inference on the NPU.

[0123] Multimodal model: A deep learning model that can process both image and text modal information simultaneously, including at least a visual model for extracting image features and a language model for processing sequences and generating text.

[0124] Visual model: A neural network model used to extract image features. It can adopt the ViT (Vision Transformer) architecture, which segments the input image into a sequence of patches and then extracts features through a self-attention mechanism, outputting a feature matrix of visual tokens.

[0125] Language model: A deep learning model used to process sequences and generate text, usually a large language model (LLM), which can understand natural language and perform reasoning and generation based on input sequences.

[0126] LLM (Large Language Model): A language model with a large number of parameters, capable of understanding and generating natural language. The language model in this embodiment can be used to process mixed sequences of text and image tokens, ultimately outputting image reasoning results.

[0127] ViT (Vision Transformer): An image feature extraction model based on a self-attention mechanism, which segments the image into fixed-size patches, linearly projects each patch into a token sequence, and then extracts features through a Transformer encoder. The visual model in this embodiment can adopt the ViT architecture.

[0128] Preset fixed inference resolution: This is the only input image resolution supported by the visual model in the multimodal model, that is, the fixed input size determined by the visual model at compile time, such as 448×448 pixels.

[0129] Target preset resolution: This is a resolution dynamically determined based on the original resolution of the input image and the preset fixed inference resolution. This resolution is an integer multiple of the preset fixed inference resolution and does not exceed the long side size of the original resolution of the input image. It is used to control the granularity of segmentation of local image blocks.

[0130] Global view image: refers to the image obtained by scaling the input image as a whole and filling it to a preset fixed inference resolution, which is used to provide global semantic features of the input image.

[0131] Local block image: refers to the sub-image obtained by scaling the input image proportionally to the target preset resolution, then uniformly dividing it according to the size of the preset fixed inference resolution, and scaling each segmented image block to the preset fixed inference resolution. It is used to provide high-resolution local detail features of the input image.

[0132] Resizing: An operation that changes the size of an image, including proportional scaling (maintaining aspect ratio) and non-proportional scaling (changing aspect ratio). In this embodiment of the disclosure, proportional scaling can be used to maintain the original aspect ratio of the image and avoid content distortion.

[0133] Padding: The operation of adding pixel values ​​(typically 0 or black) around the edges of an image to expand the image size to the target resolution. In embodiments of this disclosure, the generation of the global view image includes two steps: scaling and padding, to adapt to a preset fixed inference resolution.

[0134] Visual encoding refers to the process of inputting an image into a visual model and extracting the image feature matrix through the model's forward computation. This feature matrix consists of feature vectors of multiple visual tokens.

[0135] Visual token: refers to a basic unit in the feature matrix output by the visual model. Each visual token corresponds to a local image region (i.e., a patch) in the input image, and its form is a feature vector.

[0136] Token: The basic unit in a sequence. A text token represents a word or subword, while an image token (visual token) represents the feature vector corresponding to a patch in an image. In this embodiment, the input sequence of the language model is composed of both text tokens and visual tokens.

[0137] Embedding maps discrete tokens (such as text words or image patches) to continuous vector representations that contain semantic information. The input embedding matrix of a language model is a stack of token embedding vectors.

[0138] In Vision Transformer, the input image is divided into fixed-size blocks, each called a patch and corresponding to a visual token. The patch size (e.g., 32×32 pixels) determines the number of tokens in the visual model's output feature matrix.

[0139] Patch size: refers to the width and height of the pixel blocks used to segment an image into non-overlapping visual tokens, usually expressed in pixels, such as 32×32 pixels. This size determines the number of tokens in the output feature matrix of the visual model, which is the quotient of the image resolution divided by the patch size.

[0140] Global position encoding: refers to the position identifier in the global coordinate space assigned to each visual token, including the global height index and the global width index, which is used to maintain the continuous consistency between each visual token and the original image space layout in the input sequence of the language model.

[0141] Global coordinate space: refers to the two-dimensional coordinate space corresponding to the input image after it has been scaled proportionally to the target preset resolution, established with the long and short sides of the scaled image as coordinate axes. In the embodiments of this disclosure, the absolute position coordinates of local block images and the global position encoding of visual tokens are both based on this space.

[0142] Absolute position coordinates: refers to the position coordinates of a local block image in the global coordinate space, such as the horizontal and vertical coordinates of a preset reference point (such as the upper left corner) on the local block image in the global coordinate space.

[0143] Grid position index: refers to the row and column numbers used when a local block image is uniformly divided according to the preset inference resolution after the input image is scaled proportionally to the target preset resolution.

[0144] Prefill: refers to the computational process by which the language model establishes the context state of the constructed input sequence (including text tokens and image tokens) all at once before generating the output through token-by-token decoding. This process usually has higher parallel computational efficiency than token-by-token decoding.

[0145] Decoding refers to the process by which the language model generates an output sequence token by token after pre-filling. The generation of each token depends on all previously generated tokens and the context state established during the pre-filling phase.

[0146] RoPE (Rotary Position Embedding) is a position encoding method commonly used in Transformer models. It adds relative position information to each position in a sequence using a rotation matrix, enabling the model to perceive the order and distance between tokens. In this embodiment, RoPE can be used as the position encoding scheme based on the overall token length.

[0147] The technical solutions involved in the embodiments of this disclosure will be described in detail below.

[0148] Figure 1 This is a schematic flowchart illustrating an exemplary embodiment of the image processing method disclosed herein. Figure 1 As shown, an image processing method according to an embodiment of this disclosure can be applied to a multimodal model, wherein the multimodal model has a preset fixed inference resolution, and the method may include the following steps:

[0149] Step 110: In response to receiving the input image, determine the target preset resolution based on the original resolution of the input image and the preset fixed inference resolution, so as to generate a global view image and multiple local block images that match the target preset resolution for the input image.

[0150] In this embodiment of the disclosure, an input image to be processed is received. The input image can be a single image or multiple images (e.g., multiple images in a multi-image question-and-answer scenario). The format of the input image includes, but is not limited to, JPEG, PNG, BMP, etc., and can also be pre-processed tensor data. The image can come from local storage, network transmission, or a real-time acquisition device (such as a camera, scanner, etc.). The resolution of the input image is arbitrary, for example, it can be high resolution (e.g., 1920×1080) or low resolution (e.g., 224×224).

[0151] After obtaining the input image, its original resolution is acquired, including the image's width and height, and the length of its longer side is determined. The preset fixed inference resolution is a fixed input size (e.g., 448×448 pixels) supported by the pre-compiled visual model in the multimodal model. In this embodiment, the target preset resolution can be determined through methods including, but not limited to, the following:

[0152] (1) Based on rounding: Let the longer side be L, the fixed inference resolution be F, calculate n = round(L / F), and the target preset resolution = n×F;

[0153] (2) Based on the rounding up method: Let the long side be L, the fixed inference resolution be F, n = ceil(L / F), the target preset resolution = n×F, to ensure that the scaled image completely covers the high-resolution details;

[0154] (3) Based on the method of rounding down: Let the long side be L, the fixed inference resolution be F, n = floor(L / F), the target preset resolution = n×F, to ensure that it does not exceed the original resolution and avoid blurring caused by enlarging the image.

[0155] In a preferred embodiment of this disclosure, a rounding method can be used to preserve details while controlling the number of segments. Specific values ​​can be selected based on actual deployment requirements. For example, when the fixed inference resolution is 448×448, for a 1920×1080 image, L=1920, n=round(1920 / 448)=4, the target preset resolution=4×448=1792, meaning the long side is aligned to 1792 pixels, and the short side is scaled proportionally to 1008 pixels. The image is then segmented into 4×3=12 local image blocks.

[0156] After determining the target preset resolution, two processing chains can be constructed simultaneously for the same input image: a global chain and a local chain. Specifically:

[0157] Generation of the global view image: The original input image is scaled proportionally so that its long side is equal to the long side of the preset fixed inference resolution. Then, blank areas (i.e., uncovered areas resulting from maintaining aspect ratio) are further padded with pixels (e.g., filled with 0s or black) so that the final image resolution is exactly equal to the preset fixed inference resolution. This ensures that the global view image retains the global semantic information and overall layout of the original image, providing macroscopic context for the language model.

[0158] Generation of local image blocks: The original input image is scaled proportionally so that its long side is equal to the long side of the target preset resolution. Based on the size of the preset fixed inference resolution, the scaled image is further evenly divided into multiple non-overlapping image blocks, each with a size equal to the preset fixed inference resolution (because the target preset resolution is an integer multiple of the fixed inference resolution). If the block size deviates slightly due to integer division issues, each block is fine-tuned and scaled to the precise preset fixed inference resolution. These local image blocks cover different local regions of the original image, preserving high-resolution details to provide fine-grained local features for the language model.

[0159] This embodiment dynamically determines the target preset resolution based on the ratio of the original resolution of the input image to the preset fixed inference resolution. This ensures that high-resolution images do not lose details due to direct scaling, and low-resolution images do not generate a large amount of invalid computation due to forced padding. Secondly, based on this target preset resolution, a global view image and multiple local block images are simultaneously generated: the global view preserves the global semantics and overall layout of the image through overall scaling and padding, while the local blocks preserve the local details of the high-resolution areas through proportional enlargement and uniform segmentation. The combination of these two methods enables visual models that originally only supported a single fixed resolution (e.g., 448×448) to effectively process images of any size, from low to ultra-high resolution, without recompiling a large-resolution model. This avoids the high deployment costs and resource consumption associated with compiling large-resolution models and solves the problems of detail loss or redundant computation caused by uniform scaling and padding, laying a high-quality, high-efficiency multimodal input foundation for subsequent visual feature extraction and language model inference.

[0160] Step 120: Visually encode the global view image to obtain the corresponding global visual features.

[0161] After generating the global view image, it can be input into a multimodal model, such as the visual model of the multimodal model. The corresponding global visual features are extracted through the forward computation of the multimodal model. The multimodal model can include a visual model, which is a pre-compiled neural network model that only supports a preset fixed inference resolution (e.g., 448×448 pixels). Since the global view image has been scaled and padded to this preset fixed inference resolution, it can be directly encoded using, for example, the visual model within the multimodal model.

[0162] In this disclosure, no specific architecture of the visual model in the multimodal model is limited, as long as it can receive images at a fixed resolution and output a feature matrix. This disclosure provides implementation schemes for visual encoding, including but not limited to the following:

[0163] (1) Encoding based on Vision Transformer (ViT)

[0164] The global view image can be segmented into fixed-size patches (e.g., 16×16 pixels or 32×32 pixels). Each patch is linearly projected as a token, and after adding positional encoding, it is fed into a Transformer encoder to output a global visual feature matrix. The dimension of this feature matrix is ​​(number of patches × feature dimension), and each token corresponds to a local region in the original image. This scheme utilizes a self-attention mechanism to capture global dependencies, making it suitable for tasks requiring overall semantic understanding.

[0165] (2) Encoding based on Convolutional Neural Network (CNN)

[0166] This approach uses deep convolutional networks (such as ResNet and EfficientNet) to perform multi-layer convolution and pooling operations on the global view image, and finally obtains the global visual feature vector through global average pooling or flattening. This method is computationally efficient and suitable for scenarios with high real-time requirements.

[0167] (3) Coding based on hybrid architecture

[0168] By combining CNNs with Transformers (such as ConvNeXt and Swin Transformer), low-level features are first extracted using a small number of convolutional layers, and then long-distance dependencies are modeled through Transformer layers, taking into account both local details and global semantics.

[0169] In a preferred embodiment of this disclosure, the visual model can employ a ViT architecture with a patch size of 32×32 pixels. For a global view image of 448×448, it can be divided into 14×14=196 patches, and the output global visual feature matrix has a size of 196×D, where D is the feature dimension (e.g., 768 or 1024). In subsequent steps, this global visual feature matrix will be provided as part of the input sequence to, for example, a language model.

[0170] This embodiment reduces deployment costs and computational resource requirements because the global view image is already adapted to a preset fixed inference resolution, eliminating the need to compile or train a separate high-resolution model for the global link. The global view image contains the overall layout and macroscopic semantics of the input image. The global visual features obtained after visual encoding can provide scene-level context for the language model, compensating for potential global perception gaps caused by local blocks. Both the global visual features and subsequent local block features originate from the same visual model, possessing the same feature dimensions and token structure, facilitating seamless splicing and fusion within the language model. Furthermore, visual encoding is not limited to a specific architecture and can be flexibly selected based on actual hardware (NPU) characteristics and task requirements, exhibiting good portability and scalability.

[0171] Step 130: Visually encode the multiple local image blocks to obtain the corresponding local visual features.

[0172] After generating multiple local patch images, each local patch image is sequentially (or in batches) input into the visual model of the multimodal model (which can be the same model as the visual model in step 120, i.e., the same pre-compiled fixed-resolution visual model, or it can be a different model, but both can be pre-compiled fixed-resolution visual models). The local visual features corresponding to each local patch image are extracted through the forward computation of the visual model. Since each local patch image has been scaled to a preset fixed inference resolution (e.g., 448×448 pixels), it can be directly fed into the visual model for encoding. This process can be executed serially, or in parallel or asynchronously depending on the hardware capabilities.

[0173] In this embodiment of the disclosure, the specific architecture of the visual model is not limited, and the visual encoding of local block images includes, but is not limited to, the following methods:

[0174] (1) Encoding based on Vision Transformer (ViT)

[0175] Each local image block is segmented into fixed-size patches (e.g., 32×32 pixels). Each patch is linearly projected as a token, and after adding positional encoding, it is fed into a Transformer encoder to output a local visual feature matrix. For a 448×448 image, a patch size of 32×32 yields 14×14=196 tokens, resulting in a feature matrix size of 196×D. This approach captures the local dependencies within each sub-image, making it suitable for preserving detailed features.

[0176] (2) Encoding based on Convolutional Neural Network (CNN)

[0177] Lightweight CNNs (such as MobileNet and ShuffleNet) are used to extract features from each local image block, outputting feature maps that are then flattened or globally pooled into feature vectors. This approach is computationally fast and suitable for scenarios with high real-time requirements.

[0178] (3) The same visual model with shared weights

[0179] All local image blocks share the same visual model (the same model used in the global view), meaning the model weights are unique. This greatly saves storage space, and since each sub-image has the same resolution, recompilation is unnecessary.

[0180] In a preferred embodiment of this disclosure, the visual model employs the ViT architecture, with a patch size of 32×32 pixels. Each local block image (448×448) is visually encoded to output 196 visual tokens, each token corresponding to a feature vector. For the K segmented local block images (e.g., 12), a total of K×196 local visual tokens are obtained.

[0181] In this embodiment, the local segmented images are derived from the input image scaled to a target preset resolution. Each sub-image is fed into a low-resolution model with its original high-resolution details, avoiding the loss of detail caused by direct scaling. For example, small text or table borders in the original image can be clearly preserved in the sub-image. All local segmented images are scaled to a preset fixed inference resolution and share the same pre-compiled visual model with the global view, eliminating the need to compile a large model separately for high resolution, thus reducing NPU deployment costs and storage overhead. The local visual feature matrix output by each local segmented image has the same token structure and feature dimensions as the global visual feature matrix, facilitating direct concatenation or positional writing in the language model input sequence.

[0182] Step 140: Determine the global position encoding of each visual token in the corresponding local visual features based on the spatial position of each local block image in the input image.

[0183] Each local visual feature comprises multiple visual tokens, each corresponding to a patch in the local patch image. To maintain the continuity and consistency of the spatial layout of each visual token with the original image (input image), a global positional code is computed for each visual token based on its spatial location within the input image. This global positional code allows visual tokens originally from different sub-images (local patch images) to be rearranged according to their coordinates in the actual image, thus avoiding spatial relationship distortion caused by treating sub-images as independent multi-image inputs.

[0184] One implementation approach is to scale the input image proportionally to the target preset resolution, establishing a global coordinate space. The global position code for each visual token is determined based on the absolute position coordinates (e.g., top-left corner coordinates) of each local image block within this global coordinate space and the patch size of the visual model. Another implementation approach, when the target preset resolution is an integer multiple of the preset fixed inference resolution, allows the segmented local images to form a regular grid in the global coordinate space. The global position index can be directly obtained using a linear offset formula based on the row and column indices of the local image blocks within the grid and the row and column indices of the tokens within each sub-image, thus yielding the global position code for each visual token. This approach eliminates the need for floating-point operations, resulting in high computational efficiency and making it particularly suitable for hardware accelerators such as NPUs. Yet another implementation approach is to map the sub-image positions in the original coordinate system of the input image to coordinates at the target preset resolution, and then combine this with the patch size to obtain the index, thereby obtaining the global position code for each visual token. This approach is suitable for complex scenarios requiring handling image distortion, overlapping segmentation, or non-uniform segmentation.

[0185] This embodiment enables global positional encoding to correctly perceive the geometric relationships (such as left-right adjacency, top-bottom alignment) between tokens in different subgraphs, thereby accurately understanding the overall layout of the image, such as table structure, document paragraph order, or target associations across subgraphs. It addresses the problem in related technologies where the lack of positional encoding leads to each local block image being treated as an independent image, preventing the model from establishing spatial relationships across subgraphs and resulting in incorrect layout understanding.

[0186] Step 150: Based on global visual features, various local visual features and the global position encoding of each visual token, construct the input sequence of the multimodal model and perform inference, and output the image inference result.

[0187] In this step, after obtaining the global visual features, each local visual feature, and the global positional encoding of each visual token, these features are combined with the text input (such as user questions, system prompts, etc.) in a preset order to form a complete input sequence, and each token is assigned a corresponding positional encoding. Finally, the sequence is submitted to the multimodal model for reasoning, and the reasoning results corresponding to the image are output (such as answer text, category labels, image descriptions, etc.).

[0188] The input sequence includes, but is not limited to, a sequence such as "text prefix → global visual token → local visual tokens → user question text". The global visual token originates from global visual features, while the local visual tokens are arranged sequentially according to their spatial order in the original image (from left to right, from top to bottom). The global positional encoding of each visual token is written into the language model's positional encoding cache, enabling the language model to perceive the absolute spatial position of the token. This approach is simple and direct; placing global features first helps the model quickly establish the overall scene context. Alternatively, local visual tokens can be arranged before global visual tokens, suitable for tasks requiring priority on details (such as fine-grained object detection), with global visual features placed later as supplementary information. Another option is to alternate between global and local visual tokens according to certain rules; for example, when processing very large images, a global token can be inserted after every few local tokens to balance the acquisition of overall and local information.

[0189] After constructing the input sequence, it can be submitted to the multimodal model for inference. In this embodiment, the inference implementation includes, but is not limited to:

[0190] One approach is to first prefill the complete input sequence to establish the context state of all tokens at once, and then proceed to the token-by-token decoding stage, generating one output token at a time until an end marker is produced or the maximum length is reached. This scheme is simple to implement and suitable for most scenarios.

[0191] As an alternative implementation, it can be combined with the previous asynchronous encoding of local block images. As visual features become increasingly ready, the multimodal model performs pre-padding on the blocks. That is, after visual encoding of a local block image or a batch of local block images is completed, pre-padding calculation is immediately performed on the token interval for that portion, while the visual model continues encoding subsequent local block images. Decoding and output are then performed only after all pre-padding blocks are completed. This scheme effectively hides the visual encoding overhead and reduces the latency of generating the first token, making it suitable for real-time interactive scenarios with high-resolution images.

[0192] As another implementation method, if the task only requires a simple judgment (such as "whether it contains a cat") during the decoding process, it can be terminated early after generating a small number of tokens to avoid unnecessary calculations and improve efficiency.

[0193] In general, in the embodiments of this disclosure, the multimodal model (e.g., the language model) can adopt an autoregressive generation approach, with its inference divided into two stages: prefill and decoding. In the prefill stage, the language model reads in the constructed complete input sequence (including text prefixes, global visual tokens, each local visual token and its corresponding global positional encoding, user question text, etc.) all at once, and computes the key-value cache and context state of all tokens in parallel, thereby establishing an understanding of the entire input. This stage is computationally intensive but is executed only once. In the decoding stage, the language model generates output tokens (e.g., each character or sub-word of the answer text) one by one in an autoregressive manner. Each time a new token is generated, the context is updated using the cache increment computed in the prefill stage, until an end marker is generated or the maximum length is reached. Since the global positional encoding provides the absolute spatial coordinates in the original graph for each visual token, the language model can model the tokens of different subgraphs according to their actual layout relationship during the prefill stage, thereby accurately answering spatially related questions during decoding (e.g., "What is the value in the second row and third column of the table?"). The entire inference process fully utilizes the NPU's parallel acceleration capability for matrix operations, achieving efficient inference while ensuring accuracy.

[0194] Through the above embodiments, global visual tokens, local visual tokens, and text tokens are organized in the same sequence. The multimodal model can simultaneously utilize the overall semantics of the image, local details, and user questions to perform cross-modal reasoning and output accurate results. Each visual token is assigned a global positional code reflecting its original image location, enabling the language model to understand geometric relationships across subgraphs (e.g., cells in the same row of a table, or blocks of text in consecutive paragraphs), avoiding spatial distortion caused by segmentation.

[0195] Figure 2 This is a schematic flowchart illustrating an image processing method as an exemplary embodiment of this disclosure. Figure 2 As shown in the figure, an image processing method according to an embodiment of this disclosure may further include the following steps:

[0196] Step 111: Obtain the original resolution of the input image, wherein the original resolution includes the length of the input image, which is the larger of the width and height of the input image.

[0197] After receiving the input image, its original resolution can be obtained, namely the image's width and height. To adaptively determine the target preset resolution later, the longer side dimension can be extracted from the width and height, defined as the larger of the two. Let the width of the input image be W and the height be H, then the formula for calculating the longer side L is, for example:

[0198] , formula (1)

[0199] The length of the longer side determines the overall scale of the image, which is a key basis for subsequent judgments on whether segmentation is necessary and for selecting the target preset resolution.

[0200] Specifically, the metadata of the input image can be read to obtain the pixel width W and pixel height H. The image format can be common formats such as JPEG, PNG, and BMP, or it can be decoded tensor data. Then, the sizes of W and H are compared, and the larger value is taken as the longer side L. For example, if the image resolution is 1920×1080, then L=1920; if the image resolution is 720×1280, then L=1280. The longer side dimension L is recorded, while the original width and height are retained to maintain the aspect ratio during subsequent proportional scaling. In this embodiment, this step can be performed by the CPU, reading the header information of the image file or the shape of the image tensor in memory. The calculation process is simple and has low computational overhead.

[0201] This embodiment determines the target preset resolution using only the long side dimension, eliminating the need to consider both width and height simultaneously, thus reducing the complexity of the decision-making logic. Most visual models (such as ViT) use square inputs (e.g., 448×448), and image scaling typically maintains the aspect ratio. Therefore, aligning with the long side maximizes the preservation of image information and avoids excessive stretching or compression along the short side. Regardless of whether the input image is horizontal or vertical, a uniform scale is achieved through the long side, ensuring consistent rules for calculating the number of rows and columns in the subsequent mesh segmentation, eliminating the need to design separate logic for different directions.

[0202] Step 112: Determine the target preset resolution based on the original resolution of the input image and the preset fixed inference resolution. The target preset resolution is an integer multiple of the preset fixed inference resolution and does not exceed the long side dimension of the original resolution.

[0203] After obtaining the long side dimension L of the input image, a positive integer multiple n is calculated based on the ratio of L to the preset fixed inference resolution F of the multimodal model (e.g., 448 pixels), and the target preset resolution R = n × F is determined. This target preset resolution must satisfy at least two conditions: first, it must be an integer multiple of the preset fixed inference resolution F; second, it must not exceed the long side dimension L of the original image. In this way, the subsequently segmented local image blocks can be perfectly adapted to the fixed resolution model, while avoiding blurring caused by image enlargement.

[0204] In this embodiment of the disclosure, the positive integer multiple n can be determined in the following manner:

[0205] One implementation method is to use rounding to the nearest integer. Specifically, the calculation... This means rounding L / F to the nearest integer, and the result must be at least 1. If n×F>L, then take... This ensures that the image does not exceed the original longer side. This method achieves a balance between detail preservation and computational cost. For example, given an input image of 3840×2160, F=448, L=3840, L / F≈8.57, rounding gives n=9, 9×448=4032>3840, so we take... The target preset resolution is R = 8 × 448 = 3584.

[0206] As another implementation method, we can use the floor function, specifically: And similarly check if it exceeds L; if it does, then take... This method tends to retain more detail but may produce more sub-images. For example, given an input image of 800×600, F=448, L=800, L / F≈1.79, rounding gives n=2, 2×448=896>800, exceeding L, so we take... The target preset resolution is R = 1 × 448 = 448.

[0207] As another implementation method, we can use the floor function, specifically: This method guarantees that n×F≤L always holds true, but may lose edge details.

[0208] Through the above embodiments, the target preset resolution is limited to an integer multiple of the preset fixed inference resolution, ensuring that each subsequently segmented local image can be precisely scaled or filled to the fixed resolution without recompiling the model or dealing with edge redundancy caused by non-integer multiples. Furthermore, limiting the target preset resolution to no more than the long side of the original resolution avoids blurring or artifacts caused by image enlargement, ensuring that the local image is based on the original high-resolution information. By rounding, the target preset resolution is neither too large (avoiding excessive computation due to too many subgraphs) nor too small (avoiding loss of detail), achieving a good trade-off between inference performance and speed. Regardless of whether the input image is ultra-high resolution or low resolution, a suitable target preset resolution can be automatically determined based on its long side, enabling adaptive dual-link (global view image and multiple local image segments) processing.

[0209] Step 113: Scale the input image as a whole and fill it to a preset fixed inference resolution to generate a global view image.

[0210] After the target preset resolution is determined, this step generates a global view image for the same input image. The global view image is used to provide the multimodal model with the overall semantics and macroscopic layout of the input image. The generation process includes, for example, scaling the input image proportionally so that its long side is equal to the long side of the preset fixed inference resolution to obtain an intermediate image; then filling the blank areas of the intermediate image so that the resolution of the final image is exactly equal to the preset fixed inference resolution.

[0211] Furthermore, the generation of the global view image provided in this embodiment may include the following steps: scaling the input image proportionally to obtain a first intermediate image, wherein the long side dimension of the first intermediate image is equal to the long side dimension in the preset fixed inference resolution; filling the blank area of ​​the intermediate image so that the resolution of the filled image reaches the preset fixed inference resolution to obtain a global view image, wherein the blank area is the edge area formed between the short side dimension of the intermediate image and the short side dimension of the preset fixed inference resolution, so that the resolution of the final image is exactly equal to the preset fixed inference resolution.

[0212] Specifically, for example, suppose the preset fixed inference resolution is F×F (e.g., 448×448, i.e., the long side and the short side are equal), the original width of the input image is W, the height is H, and the original long side is... Original short side .

[0213] First, scale proportionally:

[0214] Calculate the scaling factor r = F / L (because the long side is aligned to F), and multiply the width and height of the input image by r respectively to obtain the width W of the first intermediate image.m =W×r and height H m =H×r. Since L is aligned to F, the longer side of the middle image is equal to F, and the shorter side is S. m =S×r is less than or equal to F. This operation preserves the aspect ratio of the original image and avoids content distortion.

[0215] Then, fill in the blank areas:

[0216] The short side S of the middle image m Typically less than F, therefore, pixel values ​​(e.g., 0 or black) need to be added on both sides (or one side, depending on the fill strategy) of the shorter side to ensure that the width and height after filling both reach F. The fill amount is calculated as padding = (FS). m If the value is not divisible by 2, then one extra pixel is added to one side. The filled image is the global view image, and its resolution is exactly F×F.

[0217] The blank area refers to the edge region formed between the intermediate image and the short side dimension of the preset fixed inference resolution along the short side direction. For example, the height of the intermediate image is H. m If the short side dimension of the preset fixed resolution is F, then the blank area is located at the top and bottom of the image (if the original image is horizontal) or to the left and right (if the original image is vertical), and its total thickness is (FH). m ).

[0218] The following examples demonstrate the generation of global view images based on different input images (horizontal, vertical, etc.):

[0219] As an example, the input image is a landscape image, 1920×1080, with a preset fixed inference resolution F=448. The longer side L=1920, and the scaling ratio r=448 / 1920≈0.2333. The width W of the first intermediate image... m =1920 × 0.2333 = 448, height H m =1080×0.2333≈252. The short side gap is 448-252=196, and 98 pixels are filled at the top and bottom. The final global view image is 448×448, with the middle image located in the center and the top and bottom areas filled with 0 values.

[0220] As another example, the input image is a portrait image of 1080×1920, with the longer side L=1920 and a scaling ratio r=448 / 1920≈0.2333. The width W of the first intermediate image. m =1080×0.2333≈252, height H m =1920×0.2333=448. The short side gap is 448-252=196, and 98 pixels are filled on each side.

[0221] As another example, the input image is a square image, 448×448, with a scaling ratio r=1. The middle image size is already 448×448, with no blank areas, so no padding is needed.

[0222] In this embodiment, the global view image is unified to a preset fixed inference resolution, allowing direct input into the visual model of a pre-compiled multimodal model without additional compilation or adjustment of the model structure. Overall scaling maintains the general structure and layout of the image, and the filling operation does not affect the main content areas, enabling the model to capture scene-level context. Proportional scaling avoids image content distortion, ensuring that features such as object shape and text orientation are not distorted, which is beneficial for accurate model understanding. Furthermore, scaling and filling computations are minimal and can be completed quickly on the CPU without becoming a bottleneck for inference, thus accelerating inference output efficiency. Additionally, through a long-side alignment strategy, a fixed-size global view can be generated regardless of the original image orientation, simplifying subsequent processing logic.

[0223] Step 114: Based on the target preset resolution and the preset fixed inference resolution, the input image is segmented to obtain multiple non-overlapping local block images.

[0224] After determining the target preset resolution R (R=n×F, where F is the preset fixed inference resolution and n is a positive integer), in this step, the same input image is segmented to generate multiple local block images. These local block images cover different local regions of the input image and do not overlap with each other. Each block is finally adjusted to an F×F resolution so that it can be input into the same fixed resolution visual model.

[0225] Further, the input image is scaled proportionally to obtain a second intermediate image, the long side of which is equal to the long side of the target preset resolution. Based on the size of the second intermediate image and the size of the preset fixed inference resolution, the number of rows and columns to be segmented are determined. According to the number of rows and columns to be segmented, the second intermediate image is evenly divided into multiple non-overlapping image blocks. The size of each image block is adjusted to the preset fixed inference resolution to obtain multiple local block images.

[0226] For example, in this step, the input image is segmented based on the target preset resolution R and the preset fixed inference resolution F to obtain multiple non-overlapping local block images. Specifically, this may include:

[0227] First, the input image is scaled proportionally so that the long side of the scaled image (called the second intermediate image) is equal to the long side of the target preset resolution R.

[0228] Let the original image have a width of W and a height of H, with the longer side... The scaling factor r = R / L. Therefore, the width of the second intermediate image... ,high ,and In this way, the original aspect ratio can be maintained.

[0229] Then, determine the number of rows and columns to be segmented. Specifically, this can be based on the size of the second intermediate image. Calculate the number of columns C and the number of rows R to be segmented, based on the preset fixed inference resolution size F×F. c :

[0230] , formula (2)

[0231] Since the target preset resolution R is an integer multiple of F, the long side direction is usually divisible, while the short side direction may not be divisible. Using rounding up can ensure that the entire image is completely covered without missing any edges.

[0232] Secondly, the image is uniformly divided into non-overlapping blocks. This is done according to the calculated number of rows R. c And the number of columns C, the second intermediate image is evenly divided into R. c ×C rectangular blocks. The width of each block is... The height is .because It may not be an integer multiple of F, and the size of the edge blocks may be slightly larger or smaller than F, but the blocks do not overlap strictly and, when stitched together, exactly cover the entire second intermediate image.

[0233] Specifically, image segmentation based on the number of rows and columns can be achieved in ways including but not limited to the following:

[0234] (1) Mesh-based segmentation

[0235] First, a virtual segmentation grid is created on the second intermediate image: the horizontal (row direction) is equally divided into R... c Segments, each segment is [height] The column is divided into C segments, each segment having a width of [missing information]. These dividing lines form a The grid is such that each grid cell corresponds to an image block. Then, for the r-th row and c-th column of the grid ( The top-left corner coordinates of its block are The block size is By traversing all grid cells, all non-overlapping local image blocks can be extracted. For example, the second intermediate image is 1792×1008, R... c=3, C=4. Therefore, the grid has 3 rows and 4 columns, with each block having a width of 1792 / 4=448 and a height of 1008 / 3=336. Twelve 448×336 local block images are extracted sequentially.

[0236] (2) A fixed-size region can be cropped from the image each time by looping through the row and column indices:

[0237] For the i-th row (i=0, 1, 2, ..., R) c -1), starting ordinate ;

[0238] For the j-th column (j=0, 1, 2, ..., C-1), the starting x-coordinate ;

[0239] Using (x, y) as the top left corner, cut out a section with a width of Height is The region (the last block may contain remaining pixels). This method uses integer arithmetic to avoid floating-point errors.

[0240] In some embodiments, an image segmentation network (such as U-Net) can be used to automatically learn the segmentation boundaries to achieve the division of local block images.

[0241] Finally, each local block image is scaled to precisely achieve an F×F resolution. The scaling algorithm can employ bilinear interpolation, nearest neighbor interpolation, or other common methods. Each adjusted block becomes a local block image. All local block images are non-overlapping, and each sub-image can be directly input into a visual model with a preset fixed inference resolution.

[0242] For example, given an input image of 1920×1080, a preset fixed inference resolution F=448, and a target preset resolution R=1792 (n=4), then the scaling ratio r=1792 / 1920≈0.9333, the second intermediate image size W′=1792, H′=1008, and C= =4,R c = =3, divided into 3×4=12 blocks, each block size is 448×336 (because 1792 / 4=448, 1008 / 3=336). Finally, each 448×336 block is scaled down to 448×448, resulting in 12 local block images.

[0243] In this way, an input image of arbitrary resolution is adaptively divided into multiple fixed-size sub-images, which preserves high-resolution details and ensures that each sub-image can reuse the same fixed-resolution visual model, avoiding the overhead of compiling a large model.

[0244] The implementation process of the global view image and the local block image is briefly described below with reference to the accompanying drawings. Figure 3 This is a schematic diagram of an input image, a global view image, and a partial block image, as shown in another exemplary embodiment of this disclosure. Figure 3 As shown, the left side displays an input image (original image) with a relatively high resolution (e.g., 1920×1080). Based on the ratio of the input image's longer side to a preset fixed inference resolution (e.g., 448×448), a target preset resolution (e.g., 1792 pixels) is adaptively selected. Based on this target preset resolution, two processing chains are generated simultaneously:

[0245] Global View Image (global image): The original image (input image) is scaled proportionally and filled to a preset fixed inference resolution to obtain a global view image with a size of 448×448. This global view image retains the overall layout and macroscopic semantics of the original image, such as the approximate distribution of objects and scene structure. It is represented by the box "global image" in the image and labeled "scaled to a fixed resolution".

[0246] Local sub-images: The original image (input image) is scaled proportionally to the target preset resolution (e.g., 1792×1008), and then the scaled image is evenly divided into multiple non-overlapping rectangular blocks according to the preset fixed inference resolution size (448×448). Each block is then scaled separately to a resolution of 448×448, forming multiple local sub-images. Figure 3 The example shows the child Figure 1 To the Son Figure 6 (The actual number depends on the size of the original image, for example, it may be 12.) Each sub-image contains high-resolution local details of the original image, such as text, table cells, small objects, etc. They are indicated in the image by a "local sub-image" box and labeled "scaled or filled to a fixed resolution respectively".

[0247] Furthermore, Figure 3 The arrows in the diagram indicate that the global view image and all local block images are fed into the visual model in parallel for visual encoding. Since all images are processed to a uniform fixed resolution (448×448), they can share the same pre-compiled fixed-resolution visual model without the need to recompile a high-resolution model.

[0248] Figure 4 This is a schematic flowchart illustrating an image processing method as an exemplary embodiment of this disclosure. Figure 4 As shown, an image processing method according to an embodiment of this disclosure may further include the following steps:

[0249] Step 141: Determine the absolute position coordinates of each local block image, wherein the absolute position coordinates are the position coordinates of each local block image in the global coordinate space after the input image is scaled proportionally to the target preset resolution.

[0250] In this step, the absolute position coordinates of each generated local image block in the global coordinate space are further determined. The global coordinate space is a two-dimensional coordinate space corresponding to the input image after it has been scaled proportionally to the target preset resolution R (established with the long and short sides of the scaled image as axes). The absolute position coordinates are used to calculate the global position code for each visual token and are crucial for maintaining spatial continuity between sub-images. In this step, the absolute position coordinates are defined as the x and y coordinates of a preset reference point (e.g., the top left corner, the center point, etc.) on each local image block in the global coordinate space.

[0251] As one implementation method, specifically, the grid position index of each local block image can be obtained. The grid position index is the row number and column number used when the input image is scaled proportionally to the target preset resolution and then uniformly divided according to the size of the preset fixed inference resolution. Then, based on the grid position index and the size of the preset fixed inference resolution, the horizontal and vertical coordinates of the preset reference point of the local block image in the global coordinate space are calculated as the absolute position coordinates of the local block image.

[0252] For example, since the local image block is obtained by scaling the input image proportionally to the target preset resolution R, and then uniformly dividing it into segments of a preset fixed inference resolution F×F, each local image block has a unique row and column number in the segmentation grid. These row and column numbers are called grid position indices. Let the number of rows in the segmentation grid be R. c (i.e., the number of rows to be divided), the number of columns is C (i.e., the number of columns to be divided), and the grid position index is denoted as (r, c), where r = 0, 1, ..., R. c -1; c = 0, 1, ..., C-1. These indices can be obtained directly from the segmentation process (e.g., recording the current row and column while iteratively generating subgraphs). Then, based on the grid position index (r, c) and the size F of the preset fixed inference resolution, the coordinates of the preset reference point (e.g., the top left corner) of the local block image in the global coordinate space are calculated. Assuming the preset reference point is the top left corner of the subgraph, then: x-coordinate (Because each sub-image occupies F pixels in the width direction, and the width direction may be rounded down during segmentation, but the long side direction is exactly c×F when divided by an integer; if the width direction results in a non-integer multiple due to rounding up, it can be corrected to) ,in The width of the second intermediate image. In the preferred integer multiples of this disclosure, ,therefore When the target preset resolution R is an integer multiple of F, the longer side is always divisible, while the shorter side may not be divisible. Therefore, the calculation formula can be, for example:

[0253] , formula (3)

[0254] in, and These are the actual width and height of each local block image (which may be non-integer, but the actual pixel coordinates are rounded down). In practice, an integer index multiplied by F can be used as an approximation (because the precision of subsequent positional encoding only needs to reach the order of the patch size, and small errors can be ignored).

[0255] For example, continuing from the previous example, the input image is 1920×1080, F=448, the target preset resolution is R=1792, and the grid is divided into 3 rows and 4 columns (R). c =3, C=4). The actual size of each local block image is 448×336 (width 1792 / 4=448, height 1008 / 3=336). For the sub-image located in the 2nd row and 3rd column (r=1, c=2), its absolute position coordinates of the top left corner are: x=2×448=896 (if calculated according to the block width, it is also 2×448=896), y=1×336=336 (the y-coordinate uses the actual block height of 336, not F, because the shorter side direction is not divisible). Therefore, the absolute position coordinates of this local block image are (896, 336). Similarly, if the reference point is changed to the center of the sub-image, half the block width and height can be added.

[0256] As an alternative implementation, the position of a sub-image (local block image) in the original coordinate system of the input image (e.g., the pixel region of the sub-image in the original image) can be mapped to the global coordinate space (i.e., the space scaled to the target preset resolution) using the same scaling ratio, thereby obtaining the absolute position coordinates of the sub-image in that space. This method does not rely on the row and column indices of the segmentation grid, but directly uses the pixel coordinates of the sub-image in the original image for calculation. Specifically, the pixel region corresponding to each local block image in the original input image can be determined. Since the local block images are obtained by first scaling the original image to the target preset resolution and then segmenting it, we can first deduce it in reverse: let the coordinates of the upper left corner of a certain sub-image in the original image be... Width is The height is For uniform segmentation, these values ​​can be directly calculated from the segmentation parameters. Calculate the scaling factor from the input image to the global coordinate space: The global coordinate space is the space formed by scaling the original image (input image) proportionally to the target preset resolution R. Therefore, the scaling factor s = R / L, where L is the longer side dimension of the original image. Multiply the top-left corner coordinates of the sub-image in the original image by this scaling factor to obtain its absolute position coordinates in the global coordinate space.

[0257] , formula (4)

[0258] If the origin of the original coordinate system is located at the top left corner of the image, then direct mapping is sufficient.

[0259] For example, the original image is 1920×1080, the target preset resolution R=1792, and the scaling ratio s=1792 / 1920≈0.9333. Assuming a sub-image (local block image) corresponds to the region with top-left corner coordinates (800, 500) in the original image (input image), its absolute position coordinates in the global coordinate space are:

[0260] x = 800 × 0.9333 ≈ 746.67, y = 500 × 0.9333 ≈ 466.67. These coordinates can be rounded down or to the nearest integer in pixels as needed.

[0261] This embodiment does not require the target preset resolution to be an integer multiple of the fixed inference resolution and is applicable to any segmentation method (including overlapping segmentation, non-uniform segmentation, etc.).

[0262] Through the above embodiments, the positions of different sub-images (local block images) are unified into a global coordinate space, enabling subsequent token-level position encoding to directly reflect the actual layout of the original image. Using grid position indices and known segmentation parameters (number of rows, number of columns, block size), absolute position coordinates can be obtained with a small number of multiplication and addition operations, eliminating the need for complex image processing or floating-point calculations, making it suitable for NPU environments. The absolute position coordinates, combined with the patch size in subsequent steps, can accurately calculate the global height and width indices of each visual token, ensuring spatial continuity.

[0263] Step 142: Based on the absolute position coordinates of each local block image and the preset patch size, calculate the global height index and global width index for each visual token in the local block image as the global position code of the visual token. The patch size is the width and height of the pixel block used when dividing the local block image into non-overlapping visual tokens.

[0264] In this step, a global positional encoding is calculated for each visual token within each local segment image. The global positional encoding includes a global height index and a global width index, used to map the local token positions within the local segment image to a global coordinate space. This allows, for example, a language model to perceive the spatial order and geometric relationships between tokens in different local segments. A preset patch size s (e.g., 32 pixels) determines the granularity at which the local segment image is segmented into non-overlapping visual tokens. For a fixed inference resolution F×F (e.g., 448×448), if the patch size is s×s, the number of visual tokens generated per local segment image is (F / s)×(F / s), i.e., 14×14=196 (when s=32).

[0265] As one implementation method, specifically, according to the preset patch size, each local block image is divided into multiple non-overlapping visual tokens, and the local row number and local column number of each visual token within the local block image are determined. Then, based on the absolute position coordinates of the local block image, the preset patch size, the local row number and local column number, the global height index and global width index of the visual token in the global coordinate space are calculated as the global position code of the visual token. Furthermore, the vertical coordinate of the absolute position coordinates of the local block image can be divided by the preset patch size and rounded down to obtain the first rounding result. Then, the first rounding result is added to the local row number to obtain the global height index of the visual token in the global coordinate space. The horizontal coordinate of the absolute position coordinates of the local block image can be divided by the preset patch size and rounded down to obtain the second rounding result. Then, the second rounding result is added to the local column number to obtain the global width index of the visual token in the global coordinate space. Finally, the global height index and global width index of the visual token in the global coordinate space are determined as the global position code of the visual token.

[0266] For example, each local block image is uniformly divided into multiple non-overlapping visual tokens according to a preset patch size s. Since the resolution of the local block image is F×F, and F is an integer multiple of s, the division results in a (F / s)×(F / s) grid. For the visual token in the m-th row and n-th column (m, n starts from 0) within this sub-image, its local row number is m, and its local column number is n. These local coordinates can be obtained directly from the loop traversal. Let the absolute position coordinates of this local block image be (x, y) (e.g., the coordinates of the top left corner in the global coordinate space), the patch size be s, the local row number be m, and the local column number be n. First, calculate the physical position of the visual token in the global coordinate space:

[0267] , formula (5)

[0268] Then, the global height index h and global width index w are obtained by dividing the physical location by the patch size and rounding down:

[0269] , formula (6)

[0270] The above formula is equivalent to:

[0271] , formula (7)

[0272] because It may not be an integer, but It is an integer (because m is an integer). The same applies to x. Further, the calculation process is as follows:

[0273] Divide the y-coordinate by s and round down to obtain the first rounded result. ,Then,

[0274] Will Add the local row number m to obtain the global height index. Similarly, dividing the x-coordinate by s and rounding down yields the second rounded result. Then Add the local column index n to obtain the global width index. Ultimately, (h,w) is the global positional encoding of the visual token.

[0275] In other embodiments, if the physical offset of each subgraph in the global coordinate space is known, it can be calculated directly using physical coordinates without explicit use. and This is suitable for scenarios requiring high-precision physical coordinates, and will not be elaborated further here.

[0276] Figure 5 This is a schematic diagram illustrating the alignment of absolute position and position encoding of a graph, which is another exemplary embodiment of this disclosure. Figure 5As shown, the left side displays three partial image blocks (sub-image A, sub-image B, and sub-image C). Each sub-image is divided into multiple visual tokens (represented by a grid in the image) according to a preset patch size (e.g., 32×32 pixels). These tokens have local row and column indices within the sub-image (e.g., the local coordinates of a token in sub-image A are (m, n)). Each sub-image is independently fed into the visual model, but the positional encoding of its output tokens has been remapped back to a unified global coordinate space (as noted in the text, "instead of treating the sub-images as completely independent multi-image inputs, they are mapped back to the original image coordinates"). This allows the language model to process the segmented sub-image sequence as if it were a complete image. Since the global positional encoding of each token is calculated based on the actual position of the sub-image in the original image, tokens from different sub-images are arranged continuously in the global coordinate space. For example, the tokens on the right edge of sub-image A are adjacent to the tokens on the left edge of sub-image B in the global index, thus preserving the spatial layout of the original image. The image is labeled "continuous positional encoding" and explains that "the relative layout of each sub-image in the original image is preserved."

[0277] Specifically, based on the absolute position coordinates of each subgraph determined in step 141 (i.e., the preset reference point of each subgraph in the global coordinate space, such as the coordinates of the top left corner), and combined with the local row and column numbers of each token in step 142, the global height index and global width index of each token in the global coordinate space are calculated using formulas. Taking "Subgraph A token sequence" as an example, the global position code of each token is marked as [x0, y0] ... [x1, y1], where [x, y] represents the global coordinates of the token (such as the physical coordinates of the top left corner or a discrete index). Similarly, the tokens of subgraphs B and C are also mapped to the corresponding positions in the global coordinate space: [x2, y2] ... [x3, y3] and [x4, y4] ... [x5, y5], respectively.

[0278] Through the above embodiments, a global positional code reflecting the true geometric position of the visual token in each local image block is generated, ensuring that all tokens are arranged continuously in the input sequence of the language model according to the spatial layout of the original image. This avoids the problem of fragmented positional relationships caused by treating subgraphs as independent multi-graph inputs. As a result, the multimodal model can accurately perceive the geometric structure across subgraphs (such as left-right adjacency and top-bottom alignment), improving the inference accuracy in tasks such as table analysis, document understanding, and cross-regional target recognition. At the same time, it eliminates the need to compile high-resolution models, achieving both high-resolution detail preservation and overall semantic understanding using only a single fixed-resolution visual model, thus realizing efficient and accurate image processing at any resolution.

[0279] Figure 6This is a schematic flowchart illustrating an image processing method as an exemplary embodiment of this disclosure. Figure 6 As shown, an image processing method according to an embodiment of this disclosure may further include the following steps:

[0280] Step 151: According to the preset token arrangement order, determine the token interval occupied by the global visual token corresponding to the global visual feature and each visual token in each local visual feature in the input sequence of the multimodal model.

[0281] Before constructing the input sequence for a multimodal model, each visual token (including global and local visual tokens) needs to be assigned a specific positional range within the input sequence according to a predefined token arrangement order. This assignment determines the order in which the multimodal model (e.g., the language model) reads visual features, directly affecting how the model understands the image content. The predefined arrangement order should ensure that global semantic information enters the model first, while local details are arranged in spatial order to maintain consistency with the original image layout.

[0282] In this embodiment, the predefined token arrangement order of the input sequence (predefined token arrangement order) can be, for example, a text prefix (optional), global visual tokens, local visual tokens (in spatial order), and user question text. The global visual tokens originate from the global view image and are used to provide the language model with overall scene semantics. The local visual tokens are arranged sequentially according to the spatial position of the subgraphs in the original image, typically using a row-first, column-later order (from left to right, from top to bottom), ensuring that tokens from adjacent subgraphs remain adjacent in the sequence, thus facilitating the model's capture of spatial dependencies across subgraphs. The text prefix may include, for example, system instructions or task prompts (e.g., "Please describe the image content" or "Please answer the following questions"), while the user question text (e.g., "What is the total number in the third row of the table?") is placed at the end of the sequence so that the model generates an answer after integrating all visual information.

[0283] For example, a specific example is provided: the original image is divided into 4 sub-images of 2×2 pixels each, each sub-image generating M=196 visual tokens (448×448, patch 32), and the global view image generates N=196 visual tokens. Assume the text prefix has 4 tokens (e.g., system prompts), and the user question text has 10 tokens. The order and range of tokens in the input sequence are as follows:

[0284] Text prefix token: position [0, 3];

[0285] Global visual token: position [4, 199] (because 4 + 196 - 1 = 199).

[0286] Local visual tokens (in spatial order: subgraphs (1,1), (1,2), (2,1), (2,2)): subgraph (1,1): position, [200, 395] (200+196-1=395); subgraph (1,2): position [396, 591]; subgraph (2,1): position [592, 787]; subgraph (2,2): position [788, 983];

[0287] User question text: Location [984, 993].

[0288] In some embodiments, if a more compact sequence is required, the text prefix can be omitted, in which case the visual tokens of the global view image start from 0.

[0289] Furthermore, based on the above arrangement, the starting position and interval length of each visual token in the input sequence are statically calculated by accumulating the token counts of each part. First, the number of tokens for the text prefix (if it exists) can be determined as the starting offset of the global visual tokens; the number of global visual tokens is determined by the square of the ratio of the preset fixed inference resolution to the patch size (e.g., 448 / 32=14, so the number of tokens is 14). 2 =196). Then, following the spatial row and column order of the subgraphs, consecutive, equal-length token intervals are assigned to each local block image, with the number of tokens generated for each subgraph being the same as for the global interval. Finally, the user question text follows all the visual tokens. This interval assignment process relies only on known numerical parameters, eliminating the need for dynamic runtime calculations, and directly generates the absolute index position of each token, providing a deterministic basis for the precise writing of subsequent feature vectors.

[0290] Assume the total length of the input sequence is L, where the length of the text portion (prefix + user question) is L. The total length of the visual portion is The token range can be determined by accumulating offsets:

[0291] Global visual token range: from the first Starting from position , with a length of . = (Number of tokens in the text prefix). For example, the text prefix has If there are 1 token, then the global range is 1. .

[0292] Local visual token intervals: allocated sequentially according to the spatial order of the subgraphs. For the i-th subgraph (i=0,1,…,K-1), its starting position is... The interval length is .

[0293] User question text area: located after all visual tokens, starting at position [position missing]. .

[0294] In practice, an array or list can be used to record the absolute index position of each visual token in the input sequence, which facilitates subsequent scatter write operations.

[0295] Through the above embodiments, this disclosure assigns a specific input sequence interval to each visual token according to a preset token arrangement order (text prefix, global visual token, spatially arranged local visual tokens, and user question text). This order allows the language model to first acquire the global scene context and then gradually incorporate high-resolution local details. Simultaneously, since the local tokens are strictly arranged according to the original image's row and column order, tokens from adjacent subgraphs are naturally adjacent in the sequence, facilitating the model's use of self-attention mechanisms to capture spatial dependencies across subgraphs. Furthermore, the interval allocation is entirely based on static calculation of the known number of tokens, requiring no dynamic memory allocation, which aligns with the inference characteristics of the NPU and provides an accurate indexing basis for subsequent feature vector scattering, effectively improving the efficiency and reliability of sequence construction.

[0296] Step 152: Write the global visual features and each local visual feature into the corresponding token interval to obtain the input sequence of filled feature vectors.

[0297] After determining the token interval of each visual token in the input sequence in step 151, in this step, the global visual features and each local visual feature are written into the corresponding token interval in the input sequence according to a preset arrangement order, thereby obtaining an input sequence with filled feature vectors. This writing operation can adopt a scatter (distributed writing) method, that is, the feature vector is directly assigned to the specified position in the sequence according to the pre-calculated index position, instead of simply concatenating it to the end of the sequence.

[0298] As one implementation method, this step can use a scatter operation to write the visual features of each image region (global view or local block image) into the corresponding position of the language model input embedding matrix according to the pre-allocated token interval.

[0299] Specifically, the positions of text tokens are pre-assigned in the input embedding matrix of multimodal models (such as language models) (e.g., the embedding vectors corresponding to text prefixes and user question texts are filled by word embedding layers). For the global view image, each row in its visual feature matrix corresponds to a feature vector of a global visual token. Based on the determined starting position of the global token interval, these feature vectors are sequentially copied to the row positions corresponding to that interval in the input embedding matrix.

[0300] For each local image block, each row of its visual feature matrix (corresponding to the feature vector of the local visual token) is copied to the corresponding row position in the input embedding matrix according to the token interval assigned to that sub-image. The token intervals do not overlap, so the writing operation can be performed in any order (e.g., write the global interval first, then write the local intervals sequentially according to the sub-image space). After all visual features are written, the visual token positions in the input embedding matrix are filled with feature vectors, while the text token positions retain their original embeddings, thus obtaining a complete input sequence with filled feature vectors.

[0301] For example, assume the text prefix occupies 4 tokens, the global visual tokens occupy 196, the 4 local subgraphs each occupy 196, and the user question occupies 10. The global visual feature matrix V... g The size is 196×D, and it is written into the input sequence at position [4, 199]. Local subgraphs are written in the order (1, 1), (1, 2), (2, 1), (2, 2) respectively:

[0302] The feature matrix of subgraph (1,1) is written at position [200, 395]; subgraph (1,2) is written at position [396, 591]; subgraph (2,1) is written at position [592, 787]; subgraph (2,2) is written at position [788, 983].

[0303] After writing, positions 0-3 in the input sequence retain the embedding vector of the original text prefix, while positions 984-993 retain the embedding vector of the user question. All feature vectors of the visual tokens are now correctly filled, resulting in a complete input sequence with filled feature vectors.

[0304] Through the above embodiments, accurate fusion of visual and textual features in the language model input sequence is achieved. The scatter writing method ensures that each visual token falls accurately into the designated interval according to the preset spatial order, avoiding order disorder caused by simple splicing. Simultaneously, since the interval positions are statically determined in step 151, the writing operation can directly use parallel assignment without conditional judgment or dynamic memory allocation, fully utilizing the efficient memory access capabilities of the NPU. Furthermore, this method supports any number of local block images, exhibiting good scalability and providing a structurally clear and positionally complete input sequence for subsequent language model pre-filling and decoding.

[0305] Step 153: Based on the global positional encoding of each visual token, determine the positional encoding of the corresponding token position in the input sequence with filled feature vectors, and obtain the input sequence with positional encoding.

[0306] After obtaining the input sequence with filled feature vectors, position encoding needs to be set for each token in the sequence (including text tokens and visual tokens) so that the multimodal model can perceive the order relationship between tokens. For visual tokens, the position encoding is determined based on the global height index and global width index (i.e., the global position encoding of each visual token) calculated in the aforementioned embodiments (e.g., the embodiments related to step 142); for text tokens, the position encoding is assigned according to their natural order in the sequence. Finally, a complete input sequence with position encoding is obtained, which is then submitted to the multimodal model for pre-filling and decoding.

[0307] Specifically, after obtaining the input sequence with filled feature vectors, the first step is to establish the correspondence between the position of each visual token in the input sequence and the global positional encoding calculated in the aforementioned embodiment. Since step 151 has assigned a unique token interval and sequence index to each visual token, and the aforementioned embodiment has generated a global height index and a global width index (as well as the image sequence number in multi-image scenarios) for each token, the two can be mapped one-to-one using an index array or loop iteration. Based on this, the actual positional encoding vector is generated according to the positional encoding mechanism adopted by the multimodal model (such as the language model): For learnable positional encoding, the corresponding positional vector is queried from the two-dimensional or three-dimensional position embedding table using the global height index and width index (or plus the image sequence number) as keys, and added to the token's feature vector; for trigonometric function positional encoding, the sine and cosine values ​​are calculated based on the height index and width index respectively and combined into a positional vector; for rotational positional encoding (RoPE), the global positional encoding can be mapped to a one-dimensional index (e.g., Next, input the RoPE function, or directly use the absolute position index in the sequence. The positional encoding of the text tokens is generated using the default method of the multimodal model, following their natural order in the sequence (incrementing from 0). Finally, the generated positional encoding vector is added to the feature vector of the corresponding token (or fused using other methods) to obtain the final representation of each token, thus forming a complete input sequence with positional encoding.

[0308] For example, assume that global visual tokens occupy positions 4-199 in the input sequence, and the global positional encoding (h, w) of each token is known (e.g., position 4 corresponds to (0, 0), position 5 corresponds to (0, 1), ..., position 199 corresponds to (13, 13)). The language model uses learnable two-dimensional positional encodings and maintains a 14×14 positional embedding table. For the token at position 4, the embedding vector corresponding to position (0, 0) is looked up and added to the feature vector of the global visual token. For local visual tokens, such as positions 200-395 of subgraph (1, 1), whose global positional encoding covers (0, 0) to (13, 13), the corresponding vectors are also looked up from the positional embedding table and added. For text tokens (positions 0-3 and 984-993), the vectors are looked up from the text positional encoding table according to the absolute sequence index (0, 1, 2, ...) and added.

[0309] Through the above embodiments, the global positional encoding is precisely mapped to the specific token positions in the input sequence, enabling the multimodal model to model the geometric relationships across subgraphs using the spatial coordinate information of the visual tokens. Since the generation of positional encoding involves only table lookups or simple arithmetic operations, its impact on inference latency is minimal. Simultaneously, the positional encoding of text tokens still follows the natural sequence order, ensuring the language model's conventional understanding of text order. The resulting input sequence with positional encoding preserves the spatial layout of the original graph and is compatible with the positional encoding mechanisms of mainstream language models, laying a solid foundation for subsequent efficient pre-filling and decoding.

[0310] Step 154: Submit the input sequence with positional encoding to the multimodal model, use the positional encoding as positional information for pre-filling and decoding, and output the image inference result.

[0311] After constructing the input sequence and setting the positional encoding, this step submits the complete input sequence with positional encoding to the language model in the multimodal model, and performs the two stages of prefilling and decoding in sequence, finally outputting the inference results related to the image (such as text answers, classification labels, image descriptions, etc.).

[0312] Pre-filling stage: Multimodal models (such as language models) can read in the complete input sequence (including all text and visual tokens and their positional encodings) at once, and compute the key-value cache and context state for each token in parallel. Since all tokens in the input sequence are ready, this stage can fully utilize the parallel computing capabilities of the hardware to build an understanding of the entire input at once. The pre-filling stage does not produce output tokens; it only updates the model's internal state.

[0313] Decoding Phase: After pre-filling, the multimodal model (e.g., the language model) enters the autoregressive decoding phase. This phase generates one output token (e.g., a word or subword) at a time and incrementally updates the context using existing key-value caches. Each newly generated token is used as input for the next step, repeating this process until an end marker is generated or the preset maximum output length is reached. During decoding, the model can combine the visual and text tokens encoded in the pre-filling phase, integrating global semantics with local details to generate an accurate answer.

[0314] After decoding, the token sequence output by the multimodal model (such as the language model) undergoes post-processing such as detokenization to obtain the final image reasoning result, such as "There is a red car in the image", "The value in the second column of the third row of the table is 25", or a specific answer to the user's question.

[0315] For example, suppose the input sequence length is 994 (text prefix 4 + global 196 + local 4×196 + user question 10), and positional encoding is already set. The multimodal model (e.g., the language model within it) (e.g., a 7-parameter LLM) first performs pre-filling, processing all 994 tokens at once to build a complete key-value cache. Then it enters the decoding stage: suppose the user question is "What is the total number in the table?", the model progressively generates tokens such as "total", "number", "is", "1", "2", "3", "4", etc., until it outputs the complete answer "The total number is 1234".

[0316] Through the above embodiments, the constructed input sequence is submitted to the language model, and the efficient parallel computing capabilities of the pre-filling stage are used to quickly establish the context. Then, the answer is generated through autoregressive decoding. Since the visual tokens in the input sequence already carry global positional encodings reflecting the spatial layout of the original image, the multimodal model (such as the language model) can correctly perceive the geometric relationships between different subgraph tokens during the pre-filling stage, thus enabling it to accurately answer questions involving spatial location during decoding. Overall, this disclosure achieves efficient and accurate reasoning for images of arbitrary resolution using only a single fixed-resolution visual model. The output image reasoning results can be directly used for various downstream tasks such as document analysis, visual question answering, and remote sensing image interpretation.

[0317] Figure 7 This is a schematic flowchart illustrating an image processing method as an exemplary embodiment of this disclosure. Figure 7 As shown in the figure, an image processing method according to an embodiment of this disclosure includes a multimodal model comprising a visual model and a language model, and then the following steps are performed through the visual model and the language model respectively:

[0318] (1) Executed through a visual model:

[0319] Visual encoding is performed on a predetermined number of local block images from multiple local block images to obtain the corresponding local visual features;

[0320] Based on the spatial location of each local block image in the input image, determine the global position encoding of each visual token in the corresponding local visual features;

[0321] The local visual features corresponding to a preset number of local block images and the global position encoding of each visual token are transmitted to the language model.

[0322] (2) Execution via language model:

[0323] Receive the local visual features of the preset number of local block images and the global position code of each visual token;

[0324] The corresponding token interval is pre-filled based on the global position code to obtain the corresponding pre-filled block;

[0325] (3) Repeat the above process until all local block images have been processed to obtain all pre-filled blocks corresponding to all local block images;

[0326] (4) The complete input sequence consisting of all pre-filled blocks is decoded by the language model, and the image reasoning result is output.

[0327] Specifically, in this embodiment of the disclosure, the visual model and the language model collaboratively process multiple local block images in an asynchronous and parallel manner. For example, the visual model does not need to wait for all local block images to be encoded before handing the features to the language model; instead, it processes them in batches: first, a preset number (e.g., one or more) of local block images are visually encoded, and simultaneously, the global position encoding of each visual token is determined based on the spatial position of each local block image in the input image. Then, the local visual features and their global position encodings of this batch are transmitted to the language model. Upon receiving the data, the language model immediately performs pre-padding calculations on the token interval corresponding to that batch based on the global position encoding, generating pre-padding blocks for that portion.

[0328] While the language model performs pre-filling computation, the visual model doesn't idle; instead, it continues processing the next batch of local image blocks—encoding and calculating global positional codes—before transmitting them back to the language model. This cycle repeats, forming a highly pipelined approach to visual encoding and language model pre-filling: the visual model continuously produces new batches of visual features, while the language model simultaneously consumes ready-to-use features for pre-filling. The computational tasks of the two models overlap in time, significantly masking the latency of visual encoding.

[0329] After all local image blocks have been processed, the language model has obtained all pre-padded blocks, which together constitute the complete input sequence (containing the global visual token and all local visual tokens). The language model then decodes the complete input sequence once, generating the output token by token. This asynchronous pipelined approach significantly reduces the generation time of the first output token, making it particularly suitable for processing large numbers of local blocks in high-resolution images, improving real-time response capabilities while maintaining inference accuracy.

[0330] To make the technical solutions of the embodiments of this disclosure clearer, Figure 8 For a schematic diagram of asynchronous processing in another exemplary embodiment of this disclosure, see [link to schematic diagram]. Figure 8 The asynchronous pipeline scheme of this disclosure is described in detail below. For example, Figure 8 With three local block images (sub) Figure 1 ,son Figure 2 ,son Figure 3 Taking the example of visual model and language model pre-filling on the time axis, the overlapping execution relationship is shown.

[0331] like Figure 8 As shown, the timeline progresses from left to right. The top section shows the visual model processing stream (vit_stream) running on the NPU, the bottom section shows the language model processing stream (llm_stream) running on the NPU, and the middle section shows the data preparation and scheduling operations handled by the CPU (including text embedding vector lookup, positional encoding pre-computation, subgraph distribution, etc.). After the process begins, the CPU will distribute the subgraphs... Figure 1 Image data is sent to the visual model, and the visual model performs operations on the sub-model. Figure 1 ViT encoding is performed. Simultaneously, the CPU performs parallel pre-computation of the text token's embedding vector extraction and positional encoding, and sends the results to the language model. When the visual model completes the sub-... Figure 1 After encoding, the local visual features and the global position encoding of each visual token are transmitted to the language model, which immediately triggers the computation of the first prefill chunk (chunk1 Prefill). During the prefilling process, the CPU will... Figure 2 The image data is sent to the visual model, allowing it to continue encoding sub-images. Figure 2 At the same time, the CPU updates the position encoding information; when the child Figure 2 Upon completion of encoding, the system performs positional encoding and feature writing (scatter operation) to prepare the sub-projects. Figure 2 The required token range is determined, and then chunk 2 prefill is performed immediately after chunk 1 finishes. Similarly, the sub-... Figure 3The encoding, chunk2Prefill, and subsequent chunk3Prefill also form a pipeline. This process repeats, forming an asynchronous pipeline where the visual model (NPU), language model (NPU), and CPU work together. As shown in the diagram, the tasks of the visual model and language model closely overlap on the timeline. For every sub-image completed by the visual model, the language model completes one more pre-filled chunk, thus hiding most of the time-consuming visual encoding within the pre-filling process. Once all local image chunks have been processed, the language model has obtained all pre-filled chunks and finally enters the decoding stage to output the inference result. This diagram intuitively verifies that the asynchronous pipeline scheme disclosed in this paper can significantly reduce the generation latency of the first token and improve the overall efficiency of high-resolution image inference.

[0332] Figure 9 This is a schematic diagram of the overall architecture of an image processing method as another exemplary embodiment of this disclosure. To enable those skilled in the art to clearly, completely, and accurately understand the technical solutions of the embodiments of this disclosure, the following is combined with... Figure 9 The schematic diagram of the overall architecture of the image processing method shown provides a comprehensive description of the complete processing flow of this disclosure.

[0333] like Figure 9 As shown, the system receives single or multiple image inputs. First, resolution analysis and target size selection are performed on the input image (i.e., step 110: determining the target preset resolution based on the original resolution and the preset fixed inference resolution). Subsequently, two processing chains are constructed in parallel: ① a "global graph construction" chain (corresponding to global view image generation), which scales the original image to a fixed resolution (such as the preset fixed inference resolution); ② a "local subgraph construction" chain (corresponding to local block image generation), which scales the original image proportionally to a preset resolution (i.e., the target preset resolution), then divides it into multiple subgraphs (i.e., local block images), and scales or fills each subgraph to a fixed small resolution (i.e., the preset fixed inference resolution). The images output from both chains are uniformly fed into a fixed small resolution visual model (ViT) to extract global visual features and local visual features respectively.

[0334] While the visual model outputs features, the system performs positional encoding reconstruction based on the absolute position coordinates of the subgraph (i.e., determines the global positional encoding of each visual token), achieving output of global / local features and completing positional alignment. Then, the input sequence construction module combines the global features, local features, and their reconstructed positional encodings in a preset order to form the input sequence required by the language model. The asynchronous scheduling module is specifically shown in the figure. This module is responsible for coordinating the interleaved execution of prefilling by the visual model (ViT) and the language model: the visual model encodes multiple local image blocks in batches, and after each batch is completed, the language model is triggered to perform prefilling on the corresponding token interval, forming an asynchronous pipeline where visual encoding and prefilling run in parallel ("Interleaved execution of ViT and LLM Prefilling" in the figure). After all prefilling is completed, the language model performs the decoding stage, finally outputting the inference result.

[0335] The embodiments disclosed herein achieve efficient and accurate inference for images of arbitrary resolution by using only a fixed small-resolution visual model, through resolution adaptation, dual-link feature extraction, positional encoding reconstruction based on the absolute position of the subgraph, and asynchronous scheduling. This overcomes the problems of high cost of compiling large models or loss of detail caused by scaling and padding in related technologies.

[0336] The image processing method disclosed herein can be deployed on various hardware devices, including but not limited to neural network processors (NPUs), graphics processing units (GPUs), central processing units (CPUs), and application-specific integrated circuits (ASICs), and is particularly suitable for NPU environments that only support fixed-resolution inference. This method supports processing input images of various data types, such as RGB color images, grayscale images, depth images, or multispectral images, covering common formats such as JPEG, PNG, and BMP, as well as decoupled tensor data. It can be widely used in tasks such as table analysis and text recognition in document understanding, fine-grained target localization in visual question answering, relational reasoning in complex scenes, remote sensing image interpretation, medical image analysis, and environmental perception in autonomous driving. By dynamically adapting to arbitrary resolution input, preserving global and local information, and reconstructing spatially continuous positional encoding, a good balance is achieved between inference accuracy and hardware efficiency, demonstrating significant practical value.

[0337] In summary, by dynamically determining the target preset resolution based on the original resolution of the input image and the preset fixed inference resolution, dynamically adapting to the input size, and generating a global view image and multiple local block images respectively, a single fixed-resolution visual model can simultaneously obtain the overall semantics (global view) and high-resolution local details (local blocks) of the input image without recompiling a high-resolution model. Furthermore, by calculating the global positional encoding corresponding to the spatial location of each local visual token in the original image, it is ensured that the visual tokens of each sub-image maintain the spatial continuity of the original image in the language model input sequence, avoiding the fragmentation of positional relationships caused by simply treating sub-images as independent multi-image inputs. Finally, the global visual features, local visual features, and their global positional encoding are used to construct the input sequence and perform inference, outputting the image inference result. This achieves efficient and accurate inference on images of arbitrary resolution without increasing deployment costs, while maintaining the output effect under high-resolution input.

[0338] Correspondingly, this disclosure also provides apparatus embodiments corresponding to the above method embodiments. Figure 10 This is a structural block diagram of an image processing apparatus according to an exemplary embodiment of the present disclosure, such as... Figure 10 As shown, an image processing apparatus according to an embodiment of this disclosure is applied to a multimodal model, the multimodal model having a preset fixed inference resolution, and the apparatus may include:

[0339] The preset resolution determination module 910 is used to determine a target preset resolution in response to receiving an input image, based on the original resolution of the input image and a preset fixed inference resolution, so as to generate a global view image and multiple local block images that match the target preset resolution for the input image.

[0340] The first encoding module 920 is used to perform visual encoding on the global view image to obtain the corresponding global visual features;

[0341] The second encoding module 930 is used to perform visual encoding on the multiple local block images respectively to obtain the corresponding local visual features;

[0342] The third encoding module 940 is used to determine the global position encoding of each visual token in the corresponding local visual features based on the spatial position of each of the local block images in the input image.

[0343] The inference module 950 is used to construct the input sequence of the multimodal model and perform inference based on the global visual features, each of the local visual features and the global position encoding of each visual token, and output the image inference result.

[0344] As an optional embodiment, the preset resolution determination module 910 includes:

[0345] An acquisition unit is configured to acquire the original resolution of the input image, wherein the original resolution includes the length of the input image, and the length of the input image is the larger of the width and height of the input image;

[0346] The determining unit is configured to determine a target preset resolution based on the original resolution of the input image and the preset fixed inference resolution, wherein the target preset resolution is an integer multiple of the preset fixed inference resolution and does not exceed the long side dimension of the original resolution;

[0347] A global view generation unit is used to scale and fill the input image to the preset fixed inference resolution to generate the global view image;

[0348] The local segmented image generation unit is used to segment the input image based on the target preset resolution and the preset fixed inference resolution to obtain multiple non-overlapping local segmented images.

[0349] As an optional embodiment, the global view generation unit includes:

[0350] The first scaling subunit is used to scale the input image proportionally to obtain a first intermediate image, wherein the long side dimension of the first intermediate image is equal to the long side dimension in the preset fixed inference resolution.

[0351] A filling subunit is used to fill the blank area of ​​the intermediate image so that the resolution of the filled image reaches the preset fixed inference resolution, thereby obtaining the global view image. The blank area is the edge area formed between the short side dimension of the intermediate image and the short side dimension of the preset fixed inference resolution.

[0352] As an optional embodiment, the local block image generation unit includes:

[0353] The second scaling subunit is used to scale the input image proportionally to obtain a second intermediate image, wherein the long side of the second intermediate image is equal to the long side of the target preset resolution.

[0354] The segmentation parameter determination subunit is used to determine the number of segmentation rows and columns based on the size of the second intermediate image and the size of the preset fixed inference resolution.

[0355] The image segmentation unit is used to uniformly divide the second intermediate image into multiple non-overlapping image blocks according to the number of segmentation rows and the number of segmentation columns;

[0356] The resolution adjustment subunit is used to adjust the size of each image block to the preset fixed inference resolution to obtain the multiple local block images.

[0357] As an optional embodiment, the third encoding module 940 includes:

[0358] The position coordinate determination unit is used to determine the absolute position coordinates of each of the local block images, wherein the absolute position coordinates are the position coordinates of each local block image in the global coordinate space after the input image is scaled proportionally to the target preset resolution;

[0359] The position encoding unit is used to calculate a global height index and a global width index for each visual token in the local segment image based on the absolute position coordinates of each of the local segment images and a preset patch size, as the global position encoding of the visual token, wherein the patch size is the width and height of the pixel block used when the local segment image is divided into non-overlapping visual tokens.

[0360] As an optional embodiment, the position coordinate determination unit includes:

[0361] The acquisition subunit is used to acquire the grid position index of each of the local block images. The grid position index is the row number and column number used when the input image is scaled proportionally to the target preset resolution and then uniformly divided according to the size of the preset fixed inference resolution.

[0362] The calculation subunit is used to calculate the horizontal and vertical coordinates of a preset reference point of the local block image in the global coordinate space based on the grid position index and the size of the preset fixed inference resolution, as the absolute position coordinates of the local block image.

[0363] As an optional embodiment, the position encoding unit includes:

[0364] The row and column sequence determination subunit is used to divide each local block image into multiple non-overlapping visual tokens according to the preset patch size, and to determine the local row number and local column number of each visual token within the local block image.

[0365] The position encoding determination subunit is used to calculate the global height index and global width index of the visual token in the global coordinate space based on the absolute position coordinates of the local block image, the preset patch size, the local row number and the local column number, so as to serve as the global position encoding of the visual token.

[0366] As an optional embodiment, the position encoding determination subunit is further configured to:

[0367] Divide the ordinate of the absolute position coordinates of the local block image by the preset patch size and round down to obtain the first rounded result;

[0368] Add the first rounded result to the local row number to obtain the global height index of the visual token in the global coordinate space;

[0369] Divide the x-coordinate of the absolute position coordinates of the local block image by the preset patch size and round down to obtain the second rounding result;

[0370] Add the second rounding result to the local column number to obtain the global width index of the visual token in the global coordinate space;

[0371] The global height index and global width index of the visual token in the global coordinate space are determined as the global position code of the visual token.

[0372] As an optional embodiment, the inference module 950 includes:

[0373] The token interval determination unit is used to determine, according to a preset token arrangement order, the global visual token corresponding to the global visual feature and each visual token in each of the local visual features, the token interval occupied in the input sequence of the multimodal model.

[0374] A visual feature writing unit is used to write the global visual features and each of the local visual features into the corresponding token intervals to obtain an input sequence of filled feature vectors.

[0375] A sequence position encoding unit is used to determine the position encoding of the corresponding token position in the input sequence of the filled feature vector based on the global position encoding of each visual token, so as to obtain an input sequence with position encoding.

[0376] The inference unit is used to submit the input sequence with position encoding to the multimodal model, pre-fill and decode the position encoding as position information, and output the image inference result.

[0377] As an optional embodiment, the multimodal model includes a visual model and a language model, and the device further includes:

[0378] The second encoding module 930 performs visual encoding on a preset number of local block images in the plurality of local block images by calling the visual model to obtain the corresponding local visual features;

[0379] The third encoding module 940, by calling the visual model, determines the global position encoding of each visual token in the corresponding local visual features based on the spatial position of each of the local block images in the input image, so as to transmit the local visual features corresponding to the preset number of local block images and the global position encoding of each visual token to the language model.

[0380] The inference module 950 receives the local visual features of the preset number of local block images and the global position code of each visual token by calling the language model, so as to pre-fill the corresponding token interval according to the global position code and obtain the corresponding pre-filled block.

[0381] The second encoding module 930, the third encoding module 940, and the inference module 950 repeat the above process until all the multiple local block images are processed, and all pre-filled blocks corresponding to all local block images are obtained.

[0382] The inference module 950 also decodes the complete input sequence consisting of all pre-filled blocks by calling the language model and outputs the image inference result.

[0383] The model quantization device of this disclosure corresponds to the model quantization methods described above in terms of specific implementation and beneficial technical effects. The relevant contents can be referred to each other, and will not be repeated here.

[0384] Below, for reference Figure 11 This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0385] Figure 11 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0386] like Figure 11 As shown, the electronic device includes one or more processors and memory.

[0387] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0388] The memory can store one or more computer program products, and the memory can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program products can be stored on the computer-readable storage medium, and the processor can run the computer program products to implement the image processing methods of the various embodiments of this disclosure described above and / or other desired functions.

[0389] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0390] In addition, the input device may also include, for example, a keyboard, a mouse, etc.

[0391] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0392] Of course, for the sake of simplicity, Figure 11 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0393] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the image processing methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0394] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0395] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the image processing methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0396] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0397] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0398] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0399] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0400] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0401] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0402] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0403] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. An image processing method, characterized in that, Applied to a multimodal model having a preset fixed inference resolution, the method includes: In response to receiving an input image, a target preset resolution is determined based on the original resolution of the input image and a preset fixed inference resolution, so as to generate a global view image and multiple local block images that match the target preset resolution for the input image. The global view image is visually encoded to obtain the corresponding global visual features; Visual encoding is performed on the multiple local image blocks to obtain the corresponding local visual features; Based on the spatial position of each of the local block images in the input image, determine the global position encoding of each visual token in the corresponding local visual features; Based on the global visual features, each of the local visual features, and the global position encoding of each visual token, the input sequence of the multimodal model is constructed and inference is performed to output the image inference result.

2. The method according to claim 1, characterized in that, In response to receiving an input image, the method determines a target preset resolution based on the original resolution of the input image and a preset fixed inference resolution, and generates a global view image and multiple local block images matching the target preset resolution for the input image, including: Obtain the original resolution of the input image, wherein the original resolution includes the length of the input image, and the length of the input image is the larger of the width and height of the input image; Based on the original resolution of the input image and the preset fixed inference resolution, a target preset resolution is determined. The target preset resolution is an integer multiple of the preset fixed inference resolution and does not exceed the long side dimension of the original resolution. The input image is scaled and filled to the preset fixed inference resolution to generate the global view image; Based on the target preset resolution and the preset fixed inference resolution, the input image is segmented to obtain multiple non-overlapping local block images.

3. The method according to claim 2, characterized in that, The step of scaling and filling the input image to the preset fixed inference resolution to generate the global view image includes: The input image is scaled proportionally to obtain a first intermediate image, wherein the long side of the first intermediate image is equal to the long side of the preset fixed inference resolution. The blank areas of the intermediate image are filled so that the resolution of the filled image reaches the preset fixed inference resolution, thereby obtaining the global view image. The blank areas are the edge regions formed between the short side size of the intermediate image and the short side size of the preset fixed inference resolution.

4. The method according to claim 2 or 3, characterized in that, The step of segmenting the input image based on the target preset resolution and the preset fixed inference resolution to obtain multiple non-overlapping local block images includes: The input image is scaled proportionally to obtain a second intermediate image, wherein the long side of the second intermediate image is equal to the long side of the target preset resolution; The number of rows and columns to be segmented are determined based on the size of the second intermediate image and the size of the preset fixed inference resolution. According to the number of rows and columns to be divided, the second intermediate image is evenly divided into multiple non-overlapping image blocks; The size of each image block is adjusted to the preset fixed inference resolution to obtain the multiple local block images.

5. The method according to any one of claims 1 to 3, characterized in that, The step of determining the global position encoding of each visual token in the corresponding local visual features based on the spatial position of each of the local block images in the input image includes: Determine the absolute position coordinates of each of the local image blocks, wherein the absolute position coordinates are the position coordinates of each local image block in the global coordinate space after the input image is scaled proportionally to the target preset resolution; Based on the absolute position coordinates of each of the local segment images and the preset patch size, a global height index and a global width index are calculated for each visual token in the local segment image as the global position code of the visual token. The patch size is the width and height of the pixel block used when the local segment image is divided into non-overlapping visual tokens.

6. The method according to claim 5, characterized in that, Determining the absolute position coordinates of each of the aforementioned local image blocks includes: Obtain the grid position index of each of the local block images. The grid position index is the row number and column number used when the input image is scaled proportionally to the target preset resolution and then uniformly divided according to the size of the preset fixed inference resolution. Based on the grid position index and the size of the preset fixed inference resolution, the horizontal and vertical coordinates of the preset reference point of the local block image in the global coordinate space are calculated as the absolute position coordinates of the local block image.

7. The method according to claim 5, characterized in that, The step of calculating a global height index and a global width index for each visual token in the local segment image based on the absolute position coordinates of each segment image and a preset patch size, as the global position encoding of the visual token, includes: According to the preset patch size, each local block image is divided into multiple non-overlapping visual tokens, and the local row number and local column number of each visual token within the local block image are determined. Based on the absolute position coordinates of the local block image, the preset patch size, the local row number, and the local column number, the global height index and global width index of the visual token in the global coordinate space are calculated as the global position encoding of the visual token.

8. The method according to claim 7, characterized in that, The step of calculating the global height index and global width index of the visual token in the global coordinate space based on the absolute position coordinates of the local patch image, the preset patch size, the local row number, and the local column number, as the global position encoding of the visual token, includes: Divide the ordinate of the absolute position coordinates of the local block image by the preset patch size and round down to obtain the first rounded result; Add the first rounded result to the local row number to obtain the global height index of the visual token in the global coordinate space; Divide the x-coordinate of the absolute position coordinates of the local block image by the preset patch size and round down to obtain the second rounding result; Add the second rounding result to the local column number to obtain the global width index of the visual token in the global coordinate space; The global height index and global width index of the visual token in the global coordinate space are determined as the global position code of the visual token.

9. The method according to any one of claims 1 to 3, characterized in that, The process of constructing the input sequence of the multimodal model based on the global visual features, each of the local visual features, and the global position encoding of each visual token, and performing inference to output the image inference result includes: According to the preset token arrangement order, determine the global visual token corresponding to the global visual feature, the visual token in each of the local visual features, and the token interval occupied in the input sequence of the multimodal model. The global visual features and each of the local visual features are written into the corresponding token intervals to obtain the input sequence of filled feature vectors; Based on the global position code of each visual token, the position code of the corresponding token position in the input sequence of the filled feature vector is determined, and an input sequence with position code is obtained. The input sequence with positional encoding is submitted to the multimodal model, and the positional encoding is used as positional information for pre-filling and decoding to output image inference results.

10. The method according to any one of claims 1 to 3, characterized in that, The multimodal model includes a visual model and a language model, and the method includes: The visual model performs visual encoding on a preset number of local block images from the plurality of local block images to obtain corresponding local visual features. Based on the spatial position of each local block image in the input image, the global position encoding of each visual token in the corresponding local visual features is determined, so as to transmit the local visual features corresponding to the preset number of local block images and the global position encoding of each visual token to the language model. The language model receives the local visual features of the preset number of local block images and the global position code of each visual token, and performs pre-filling calculation on the corresponding token interval according to the global position code to obtain the corresponding pre-filled block. Repeat the above process until all the local block images have been processed, and obtain all the pre-filled blocks corresponding to all local block images; The language model is used to decode the complete input sequence consisting of all pre-filled blocks, and the image reasoning result is output.

11. An image processing apparatus, characterized in that, Applied to a multimodal model, the multimodal model having a preset fixed inference resolution, the device includes: A preset resolution determination module is used to respond to receiving an input image and determine a target preset resolution based on the original resolution of the input image and a preset fixed inference resolution, so as to generate a global view image and multiple local block images that match the target preset resolution for the input image. The first encoding module is used to perform visual encoding on the global view image to obtain the corresponding global visual features; The second encoding module is used to perform visual encoding on the multiple local block images respectively to obtain the corresponding local visual features; The third encoding module is used to determine the global position encoding of each visual token in the corresponding local visual features based on the spatial position of each of the local block images in the input image. The inference module is used to construct the input sequence of the multimodal model and perform inference based on the global visual features, each of the local visual features and the global position encoding of each visual token, and output the image inference result.

12. An electronic device, characterized in that, include: Memory, used to store computer program products; A processor for executing a computer program product stored in the memory, wherein when the computer program product is executed, it implements the method described in any one of claims 1-10.

13. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-10.

14. A computer program product comprising computer program instructions, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-10.

Images