Image-text processing method and device, electronic equipment and readable storage medium
By adjusting the image size and padding in the height direction, the computational cost and speed issues in edge deployment of the VLM model are resolved, achieving more efficient image processing.
Patent Information
- Application Number
- CN202511701407.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-19
- Publication Date
- 2026-02-24
AI Technical Summary
When VLM models are deployed on the edge, it is difficult to balance model complexity and inference speed. Especially when processing images of arbitrary resolution, existing methods can lead to increased computation or information loss, affecting the processing results.
By adjusting the image size to fill the height direction and then removing the visual feature vectors of the filled portion after processing by the visual encoder, the inference speed of the language model is ensured to remain unaffected.
Without affecting the processing results, the overall processing time of the model was shortened and the computational efficiency was improved.
Smart Images
Figure CN121564504A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, specifically to a method, apparatus, electronic device, and readable storage medium for image and text processing. Background Technology
[0002] The Visual Language Model (VLM) is a multimodal artificial intelligence model that combines visual and language processing capabilities. VLM models can understand and generate natural language related to visual content. Their core goal is to achieve cross-modal interaction and reasoning, solving complex tasks that traditional computer vision and natural language processing models cannot accomplish. VLM models can be applied in many fields, such as autonomous driving, remote sensing image analysis, and cross-modal understanding. Taking autonomous driving as an example, VLM models can be used to identify semantic and visual information in complex traffic scenes for environmental perception and decision support.
[0003] The VLM model relies on large-scale joint vision-language reasoning, which demands extremely high computing power. When deployed on the edge (meaning data processing and AI computation are performed locally on the terminal device), a balance needs to be struck between model complexity and inference speed. Although the computing power of edge AI chips has improved with advancements in chip technology, it still falls short of meeting the demands of real-time multimodal processing. Summary of the Invention
[0004] To solve the above-mentioned technical problems, or at least partially solve them, embodiments of this application provide a text and image processing method, a text and image processing apparatus, an electronic device, and a readable storage medium.
[0005] In a first aspect, embodiments of this application provide a method for processing images and text, including: The first image is scaled to the target size to obtain the second image; Determine whether the height of the second image is an integer multiple of the length of the local attention window, wherein the local attention window is the local attention window in the attention mechanism used by the visual encoder included in the large image-text model; If the height of the second image is not an integer multiple of the local attention window, the second image is filled in the height direction to obtain a third image, wherein the height of the third image is an integer multiple of the length of the local attention window; The third image is used as input data and processed by the large image-text model to obtain the processing result of the large image-text model.
[0006] Optionally, the third image is input as input data into the large image-text model for processing to obtain the processing result of the large image-text model, including: performing feature extraction processing on the third image through the visual encoder included in the large image-text model to obtain a first visual embedding vector corresponding to the third image, the first visual embedding vector including a second visual embedding vector corresponding to the second image and a third visual embedding vector corresponding to the filled part in the third image; deleting the third visual embedding vector from the first visual embedding vector so that the large image-text model performs inference based on the second visual embedding vector.
[0007] Optionally, scaling the first image to a target size to obtain the second image includes: determining the target size based on the length of the local attention window of the visual encoder and the size of the image block corresponding to the visual encoder; scaling the first image proportionally to the target size to obtain the second image.
[0008] Optionally, the target size includes the target width and the target height; determining the target size based on the length of the local attention window of the visual encoder and the size of the image patch corresponding to the visual encoder includes: Calculate the ratio of the first parameter to the second parameter; wherein, the product of the length of the local attention window of the visual encoder and the first parameter is recorded as the first product, the product of the size of the image block corresponding to the visual encoder and the second parameter is recorded as the second product, and the ratio of the first product to the second product is equal to the ratio of the width to the height of the first image; The target width is determined based on the first parameter, the length of the local attention window of the visual encoder, and the width of the first image; wherein the target width does not exceed the width of the first image. The target height is determined based on the width and height of the first image and the target width.
[0009] Optionally, the visual encoder performs feature extraction processing on the third image, including: the visual encoder uses a local attention mechanism without masking to perform feature extraction processing on the third image.
[0010] Optionally, deleting the third visual embedding vector from the first visual embedding vector includes: determining a third visual embedding vector corresponding to the filled portion in the first visual embedding vector based on the size of the filled portion in the third image; and deleting the third visual embedding vector from the first visual embedding vector.
[0011] Optionally, determining the third visual embedding vector corresponding to the filled portion in the first visual embedding vector based on the size of the filled portion in the third image includes: determining the third visual embedding vector corresponding to the filled portion in the first visual embedding vector based on the ratio between the size of the second image and the size of the filled portion in the third image.
[0012] Optionally, the method includes: when the height of the second image is an integer multiple of the local attention window, not padding the second image, directly inputting the second image as input data into the large image-text model, and obtaining the processing result of the large image-text model.
[0013] Secondly, embodiments of this application provide a graphic processing apparatus, including: The scaling module is used to scale the first image to a target size to obtain the second image; The determination module is used to determine whether the height of the second image is an integer multiple of the length of the local attention window, wherein the local attention window is the local attention window in the attention mechanism adopted by the visual encoder included in the large image-text model; A filling module is used to fill the second image in the height direction when the height of the second image is not an integer multiple of the local attention window, to obtain a third image, wherein the height of the third image is an integer multiple of the length of the local attention window; The inference module is used to input the third image as input data into the image-text big data model for processing, and to obtain the processing result of the image-text big data model.
[0014] Optionally, the inference module is further configured to: perform feature extraction processing on the third image through the visual encoder included in the large image-text model to obtain a first visual embedding vector corresponding to the third image, wherein the first visual embedding vector includes a second visual embedding vector corresponding to the second image and a third visual embedding vector corresponding to the filled portion in the third image; delete the third visual embedding vector from the first visual embedding vector so that the large image-text model performs inference based on the second visual embedding vector.
[0015] Optionally, the scaling module is used to: determine the target size based on the length of the local attention window of the visual encoder and the size of the image block corresponding to the visual encoder; and perform proportional scaling on the first image to scale the first image to the target size to obtain the second image.
[0016] Optionally, the target size includes a target width and a target height; the scaling module is used to: calculate the ratio of a first parameter to a second parameter; wherein, the product of the length of the local attention window of the visual encoder and the first parameter is recorded as the first product, the product of the size of the image block corresponding to the visual encoder and the second parameter is recorded as the second product, and the ratio of the first product to the second product is equal to the ratio of the width to the height of the first image; determine the target width based on the first parameter, the length of the local attention window of the visual encoder, and the width of the first image; wherein, the target width does not exceed the width of the first image; determine the target height based on the width and height of the first image and the target width.
[0017] Optionally, the inference module is configured to: the visual encoder uses a local attention mechanism without masking to perform feature extraction processing on the third image.
[0018] Optionally, the inference module is configured to: determine a third visual embedding vector corresponding to the filled portion in the first visual embedding vector based on the size of the filled portion in the third image; and delete the third visual embedding vector from the first visual embedding vector.
[0019] Optionally, the inference module is used to: determine the third visual embedding vector corresponding to the filled portion in the first visual embedding vector based on the ratio between the size of the second image and the size of the filled portion in the third image.
[0020] Optionally, the inference module is used to: when the height of the second image is an integer multiple of the local attention window, not to fill the second image, but to directly input the second image as input data into the large image-text model, and obtain the processing result of the large image-text model.
[0021] Thirdly, embodiments of this application provide an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the image and text processing method provided in any embodiment of this application.
[0022] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, and when the program or instructions are executed by a processor, they implement the image and text processing method provided in any embodiment of this application.
[0023] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the image and text processing method provided in any embodiment of this application.
[0024] The technical solution provided in this application has at least the following beneficial effects: The image processing method provided in this application involves scaling a first image to a target size to obtain a second image. If the height of the second image is not an integer multiple of the length of a local attention window, the second image is filled in the height direction to obtain a third image. The height of the third image is an integer multiple of the length of the local attention window, which is the local attention window in the attention mechanism used by the visual encoder included in the large image-text model. The third image is then input into the large image-text model for processing to obtain the processing result. This method adjusts the size of the image to be processed so that the image is filled in the height direction instead of the width direction. The image filled in the height direction is processed by the visual encoder to generate an overall visual feature vector. The visual feature vector corresponding to the filled part is at the end of the overall visual feature vector. Deleting the visual feature vector corresponding to the filled part before language model inference does not affect the inference speed of the language model or increase its inference time. This shortens the overall processing time of the model without affecting the processing effect of the large image-text model. Attached Figure Description
[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.
[0026] Figure 1 This diagram illustrates how a VLM model processes image input of arbitrary resolution in prior art. Figure 2 This diagram illustrates how a VLM model processes image input of arbitrary resolution in prior art. Figure 3 This diagram illustrates how a VLM model processes image input of arbitrary resolution in prior art. Figure 4 A schematic flowchart of a text and image processing method according to an embodiment of this application is shown; Figure 5 A schematic diagram of the structure of a graphic processing apparatus according to an embodiment of this application is shown; Figure 6 A schematic diagram of the structure of an electronic device according to an embodiment of the present invention is shown. Detailed Implementation
[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0028] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.
[0029] The VML model in this embodiment mainly consists of a visual encoder, a language model, and a multimodal fusion module connecting the two. The visual encoder extracts image features, transforming the image into high-dimensional feature vectors. These high-dimensional feature vectors capture the key visual features of the image for use by the subsequent multimodal fusion module. Commonly used visual encoders include Convolutional Neural Networks (CNNs) (e.g., ResNet) or Visual Transformer models (e.g., the Transformer-based computer vision model ViT). The language model processes text input and output, generating natural language descriptions or performing reasoning based on text instructions. The language model is typically a pre-trained Large Language Model (LLM), such as the Generative Pre-trained Transformer (GPT) or the Bidirectional Encoder Representations from Transformers (BERT). The multimodal fusion module converts the visual features (image embeddings) generated by the visual encoder into a format that the language model can understand through a cross-modal attention mechanism. It then uses a projection layer to project the visual features into the same semantic space as the text word embeddings, thereby achieving alignment and joint inference between visual features and text semantics.
[0030] VLM models can handle tasks such as "cross-modal understanding" and "cross-modal generation," including image description (picture description), visual question answering (VQA, such as "How many dogs are in the picture?"), text-to-image or image-to-text tasks.
[0031] This paper uses the ViT model as the visual encoder and the LLM (Large Language Model) as the language model as an example. In this embodiment, the inference process of the VLM model mainly includes a visual encoding stage and an LLM inference stage. The ViT model extracts image features, such as segmenting the image into multiple fixed-size "image patches," and transforms these image patches into semantic units similar to "tokens" in natural language processing. Then, it models the global dependencies of these image patches through the Transformer's attention mechanism, thereby achieving computer vision tasks such as image classification and object detection. The LLM inference stage includes Prefill and Decode. Prefill is the first core stage when the large language model infers and generates text. It refers to the model receiving the user's input "prompt," performing attention calculations, linear transformations, and other operations on all tokens in the prompt at once, generating and caching the "key" and "value" information (i.e., KV Cache) corresponding to each token, preparing for subsequent token-by-token text generation. Decode is the second core stage in the large language model's inference and text generation process, following Prefill. This stage utilizes the KV cache from the Prefill stage, combined with the already generated token sequence, and predicts probabilities through an attention mechanism and the Softmax function (which transforms a vector containing arbitrary real numbers into a probability distribution). It generates the next token sequentially, repeating this process until complete text that meets the "stopping conditions" (such as generating a period or reaching the maximum length) is generated. Image features extracted by the ViT model and text features extracted by the LLM model are fused within the LLM model based on the positions of image or video placeholders, ensuring smooth and efficient inference for the VLM model.
[0032] Current VLM models typically handle input images of arbitrary resolution in two ways. One approach is to segment the image into multiple sub-images, each with the same resolution as the visual encoder's base resolution, such as 448x448. However, this approach results in image fragmentation, affecting the extraction of overall visual features. To address this issue, methods such as... Figure 1As shown, images of any resolution that need to be processed can be scaled up to the same resolution as the visual encoder's base resolution. However, this method of adding a global scaling image not only increases the computational load but also loses a lot of information, resulting in poor processing performance.
[0033] Another approach is to process image inputs of arbitrary resolution using a local window attention mechanism. For example, the image of arbitrary resolution to be processed can be segmented according to the window length, and the attention calculation scope can be controlled using a mask in the attention mechanism. For instance, the mask can be used to control the similarity calculation only for the Query (representing the information that needs to be focused on) and Key (representing the features of each element in the input sequence) within the window. Figure 2 This diagram illustrates how an image is segmented according to the window length. Figure 2 The following example uses a 196*196 resolution image. Figure 2 Each small square represents a resolution of 14x14, and 64 small squares form a local attention window. For example... Figure 2 The image with a resolution of 196*196 is segmented according to the length of the local attention window (e.g.) Figure 2 (As shown by the red line in the middle), after segmentation, the small cells within the local attention window are flattened. The top-left cell flattens (converting multi-dimensional data into one-dimensional data) to 8*8=64 embeddings, the top-right cell to 8*6=48 embeddings, the bottom-left cell to 6*8=48 embeddings, and the bottom-right cell to 6*6=36 embeddings. Then, the entire flattened area is used for local attention calculation, and the scope of attention calculation is controlled by a mask. However, although this method produces correct attention values for the local window, all embeddings (low-dimensional continuous vectors corresponding to the text) actually participate in the attention calculation. Values outside the mask range are 0, and the computational cost of attention is quadratic with the length, which slows down the calculation speed.
[0034] For example, such as Figure 3 As shown, the length and width of the image to be processed at any resolution can be padded to an integer multiple of the length of the local attention window, with each window calculated independently. Figure 3 Let's take a 196*196 resolution image as an example for explanation. Figure 3 Each small grid represents a resolution of 14*14, and 64 small grids form a local attention window, with a length of 112. For example... Figure 3As shown, padding is applied to a 196*196 image in both the length and width directions. Figure 3 The small red and yellow squares are the fill area, ensuring that they are integer multiples of the length of the local attention window in both the length and width directions. Then, the window is divided according to its length (e.g., ...). Figure 3 (As shown by the red lines in the middle), after segmentation, the small squares within the local attention window are flattened. The top-left square flattens into 8*8=64 embedding vectors, the top-right square flattens into 8*8=64 embedding vectors, the bottom-left square flattens into 8*8=64 embedding vectors, and the bottom-right square flattens into 8*8=64 embedding vectors. Then, attention is calculated according to the batch size parameter. Batch size refers to the number of data samples input to the model each time in deep learning training. However, this method increases invalid padding, which in turn generates hidden vectors of visual features. These hidden vectors will increase the processing time of the subsequent multimodal fusion module, increasing the overall processing time of the VLM model.
[0035] To address, or at least partially address, the image processing method provided in this application adjusts the size of the image to be processed, filling the image in the height direction instead of the width. The image filled in the height direction is then processed by a visual encoder to generate an overall visual feature vector. The visual feature vector corresponding to the filled portion is placed at the end of the overall visual feature vector. Deleting the visual feature vector corresponding to the filled portion before language model inference does not affect the inference speed or increase the inference time of the language model. This shortens the overall processing time of the model without affecting the overall image processing performance.
[0036] The graphic processing method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.
[0037] Figure 4 A schematic flowchart of a text processing method according to an embodiment of this application is shown. Figure 1 As shown, this image and text processing method includes: Step S401: Scale the first image to the target size to obtain the second image.
[0038] The first image can be the original image that needs to be processed into a large graphic model. The size (or resolution) of the original image can be arbitrary, such as 1920*1080. The first image can include text information. As an optional example, the first image can be an image of a traffic safety sign, such as a first image that includes graphic symbols for construction ahead and words such as "Construction 100m ahead".
[0039] The first image is either reduced in size or enlarged in size to a target size, and this scaled-down first image is designated as the second image. Scaled-down of the first image facilitates processing by the visual encoder using a local window attention mechanism, thereby improving the processing speed of the visual encoder.
[0040] In an optional embodiment, the target size can be determined based on the length of the local attention window of the visual encoder and the size of the image patch corresponding to the visual encoder. Here, the image patch (patch_size) is a parameter used in deep learning for image processing, referring to the size of the image segmented into small patches for training or processing. In computer vision tasks, patch_size is commonly used for image segmentation, cutting large images into smaller patches for local feature extraction. For example, during convolution operations, patch_size determines the size of the image region processed each time.
[0041] Alternatively, the target size can be determined according to the following process: Step (1): Calculate the ratio of the first parameter to the second parameter; wherein, the product of the length of the local attention window of the visual encoder and the first parameter is recorded as the first product, the product of the size of the image block corresponding to the visual encoder and the second parameter is recorded as the second product, and the ratio of the first product to the second product is equal to the ratio of the width to the height of the first image. Step (2): Determine the target width based on the first parameter, the length of the local attention window of the visual encoder, and the width of the first image; wherein the target width does not exceed the width of the first image; Step (3): Determine the target height based on the width and height of the first image and the target width.
[0042] The first and second parameters are intermediate parameters for calculating the target size. Let the width of the first image be W1, the height be H1, the first parameter be x, the second parameter be y, the length of the local attention window of the visual encoder be L, and the size of the image patch corresponding to the visual encoder be patch_size. Equation (1) is obtained from step (1): (L*x) / (patch_size*y)==W1 / H1.
[0043] Substituting the width and height of the first image, the length of the local attention window of the visual encoder, and the size of the image patch into the above formula, we obtain the ratio of the first parameter to the second parameter. Then, according to the target constraint: the product of the length of the local attention window of the visual encoder and the first parameter, scaled by N times, does not exceed the width of the first image, we calculate the maximum value of N. For example, we calculate the quotient of the width of the first image divided by the first product, and take the largest integer less than or equal to the quotient as the value of N. The target width in the target size is the product of the length of the local attention window of the visual encoder, the first parameter, and N. Based on the width and height of the first image and the target width, we calculate the target height to perform proportional scaling on the first image.
[0044] Step S402: Determine whether the height of the second image is an integer multiple of the length of the local attention window, where the local attention window is the local attention window in the attention mechanism used by the visual encoder included in the large image-text model.
[0045] Step S403: If the height of the second image is not an integer multiple of the local attention window, padding is performed on the second image in the height direction to obtain a third image, the height of which is an integer multiple of the length of the local attention window.
[0046] Step S404: Input the third image as input data into the image-text big data model for processing, and obtain the processing result of the image-text big data model. Specifically, when the image-text big data model processes the third image, the visual encoder performs feature extraction processing on the third image to obtain the first visual embedding vector corresponding to the third image. This first visual embedding vector includes the visual embedding vector corresponding to the second image (denoted as the second visual embedding vector) and the visual embedding vector corresponding to the filled portion in the third image (denoted as the third visual embedding vector). Since the filled portion is filled in the height direction of the second image, the third visual embedding vector follows the first visual embedding vector.
[0047] Before performing inference based on the first visual embedding vector corresponding to the third image, it is necessary to delete the visual embedding vector corresponding to the padding part of the third image, that is, delete the third visual embedding vector.
[0048] Step S405: If the height of the second image is an integer multiple of the local attention window, the second image is not filled. Instead, the second image is directly used as input data into the large image-text model to obtain the processing result of the large image-text model.
[0049] The image processing method provided in this invention adjusts the size of the image to be processed, filling the image only in the height direction instead of the width. The height-filled image is then processed by a visual encoder to generate an overall visual feature vector. The visual feature vector corresponding to the filled portion is placed at the end of the overall visual feature vector. This visual feature vector is deleted before language model inference, thus not affecting the inference speed or increasing the inference time of the language model. This shortens the overall processing time of the model without compromising the image-text processing performance.
[0050] In an optional embodiment, deleting the visual embedding vector corresponding to the padding portion of the third image from the visual embedding vector includes: Based on the size of the filled portion in the third image, determine the visual embedding vector corresponding to the filled portion in the third image in the visual embedding vector; Remove the visual embedding vector corresponding to the padding portion of the third image from the visual embedding vector.
[0051] For example, based on the proportional relationship between the size of the second image and the size of the filled portion, the proportional relationship between the second visual embedding vector and the third visual embedding vector in the first visual embedding vector is determined, and then the visual embedding vector corresponding to the filled portion in the third image in the first visual embedding vector is determined.
[0052] In an optional embodiment, the visual encoder uses a maskless local attention mechanism to extract features from the third image when processing it.
[0053] For ease of understanding, we will use images A and B as examples of images requiring large-scale image-text model processing. Image A has dimensions of 1920*1080, with a width of 1920 and a height of 1080. The first parameter is x, the second parameter is y, the length of the local attention window of the visual encoder is 112, and the size of the image block corresponding to the visual encoder is 28. According to step S401, (112*x) / (28*y)==1920 / 1080, so the ratio of the first parameter x to the second parameter y is 4:9. Then we calculate 1920 / (112*4)=4.28, N is taken as 4, so the target width is 112*4*4=1792, and the target height is (1920 / 1080) / 1792=1008. Therefore, the target size is 1792*1008, that is, the size of image A' is 1792*1008. The height of image A' is 1008, and the length of the local attention window of the visual encoder is 112. Since 1008 is an integer multiple of 112, there is no need to pad image A' in the height direction. Image A' can be directly used as input data to the visual encoder for processing.
[0054] Image B has dimensions of 1600*1200, with a width of 1600 and a height of 1200. The first parameter is x, the second parameter is y, the length of the local attention window of the visual encoder is 112, and the size of the image block corresponding to the visual encoder is 28. According to step S401, (112*x) / (28*y)==1600 / 1200, so the ratio of the first parameter x to the second parameter y is 1:3. Then, 1600 / (112*1)=133.33 is calculated, and N is taken as 133. Therefore, the target width is 112*133=14869, and the target height is (1600 / 1200) / 1792=11172. Thus, the target size is 14869*11172, which is the size of image B'. Image B' has a height of 14869, and the length of the local attention window of the visual encoder is 112. Since 14869 is not an integer multiple of 112, image B' needs to be padded in the height direction to make it an integer multiple of 112. After padding, image B' is used as input data for processing by the visual encoder.
[0055] Figure 5 A schematic diagram of the structure of a graphic processing apparatus according to an embodiment of this application is shown. Figure 5 As shown, the image processing apparatus 500 includes: The scaling module 501 is used to scale the first image to a target size to obtain the second image; The determination module 502 is used to determine whether the height of the second image is an integer multiple of the length of the local attention window, wherein the local attention window is the local attention window in the attention mechanism adopted by the visual encoder included in the large image-text model; The filling module 503 is used to fill the second image in the height direction when the height of the second image is not an integer multiple of the local attention window, to obtain a third image, wherein the height of the third image is an integer multiple of the length of the local attention window; The inference module 504 is used to input the third image as input data into the image-text big data model for processing and to obtain the processing result of the image-text big data model. When the image-text big data model processes the third image, the visual encoder performs feature extraction processing on the third image to obtain a first visual embedding vector corresponding to the third image. The first visual embedding vector includes a second visual embedding vector corresponding to the second image and a third visual embedding vector corresponding to the filled portion in the third image. The third visual embedding vector is deleted from the first visual embedding vector so that the image-text big data model performs inference based on the second visual embedding vector.
[0056] Optionally, the scaling module is used to: determine the target size based on the length of the local attention window of the visual encoder and the size of the image block corresponding to the visual encoder; and perform proportional scaling on the first image to scale the first image to the target size to obtain the second image.
[0057] Optionally, the target size includes a target width and a target height; the scaling module is used to: calculate the ratio of a first parameter to a second parameter; wherein, the product of the length of the local attention window of the visual encoder and the first parameter is recorded as the first product, the product of the size of the image block corresponding to the visual encoder and the second parameter is recorded as the second product, and the ratio of the first product to the second product is equal to the ratio of the width to the height of the first image; determine the target width based on the first parameter, the length of the local attention window of the visual encoder, and the width of the first image; wherein, the target width does not exceed the width of the first image; determine the target height based on the width and height of the first image and the target width.
[0058] Optionally, the inference module is configured to: the visual encoder uses a local attention mechanism without masking to perform feature extraction processing on the third image.
[0059] Optionally, the inference module is configured to: determine a third visual embedding vector corresponding to the filled portion in the first visual embedding vector based on the size of the filled portion in the third image; and delete the third visual embedding vector from the first visual embedding vector.
[0060] Optionally, the inference module is used to: determine the third visual embedding vector corresponding to the filled portion in the first visual embedding vector based on the ratio between the size of the second image and the size of the filled portion in the third image.
[0061] Optionally, the inference module is used to: when the height of the second image is an integer multiple of the local attention window, not to fill the second image, but to directly input the second image as input data into the large image-text model, and obtain the processing result of the large image-text model.
[0062] The above-described apparatus can execute the method provided in the embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in the embodiments of the present invention.
[0063] It should be noted that the image and text processing method provided in this application embodiment can be executed by an image and text processing device, or by a control module in the image and text processing device for executing the loading image and text processing method.
[0064] The image and text processing device in this application embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.
[0065] The image processing device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.
[0066] This application also provides a non-volatile readable storage medium storing one or more modules (programs). When these modules are applied to a device, they enable the device to execute the instructions for the method steps in this application.
[0067] This application provides one or more machine-readable media storing instructions that, when executed by one or more processors, cause an electronic device to perform one or more of the methods described in the above embodiments. In this application, the electronic device includes various types of devices such as terminal devices and servers (clusters).
[0068] The embodiments of this disclosure can be implemented as an apparatus configured as desired using any suitable hardware, firmware, software, or any combination thereof, including electronic devices such as terminal devices and servers (clusters). Figure 6 A block diagram is shown of an exemplary electronic device 1100 that can be used to implement the various embodiments described in this application.
[0069] In one embodiment, Figure 6The illustrated electronic device 1100 includes one or more processors 1102, a control module (chipset) 1104 coupled to at least one of the processors 1102, a memory 1106 coupled to the control module 1104, an NVM (non-volatile memory) / storage device 1108 coupled to the control module 1104, one or more input / output devices 1110 coupled to the control module 1104, and a network interface 1112 coupled to the control module 1104. Here, NVM refers to memory that stores information that can persist for a long time after the power is turned off, making it less prone to loss.
[0070] Processor 1102 may include one or more single-core or multi-core processors, and processor 1102 may include any combination of general-purpose processors or special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, electronic device 1100 can serve as a terminal device, server (cluster), or other device as described in the embodiments of this application.
[0071] In some embodiments, electronic device 1100 may include one or more computer-readable media (e.g., memory 1106 or non-volatile memory / storage device 1108) having instructions 1114 and one or more processors 1102 that are combined with the one or more computer-readable media and configured to execute instructions 1114 to implement modules and thus perform the actions described in this disclosure.
[0072] In one embodiment, the control module 1104 may include any suitable interface controller to provide any suitable interface to at least one of the processors 1102 and / or any suitable device or component communicating with the control module 1104.
[0073] The control module 1104 may include a memory controller module to provide an interface to the memory 1106. The memory controller module may be a hardware module, a software module, and / or a firmware module.
[0074] Memory 1106 may be used, for example, to load and store data and / or instructions 1114 for electronic device 1100. In one embodiment, memory 1106 may include any suitable volatile memory, such as suitable DRAM (Dynamic Random Access Memory). In some embodiments, memory 1106 may include double data rate type quad synchronous dynamic random access memory.
[0075] In one embodiment, the control module 1104 may include one or more input / output controllers to provide an interface to the non-volatile memory / storage device 1108 and (one or more) input / output devices 1110.
[0076] For example, non-volatile memory / storage device 1108 may be used to store data and / or instructions 1114. Non-volatile memory / storage device 1108 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable (one or more) non-volatile storage devices (e.g., one or more hard disk drives, one or more optical disk drives, and / or one or more digital universal optical disk drives).
[0077] The non-volatile memory / storage device 1108 may include storage resources that are physically part of a device on which the electronic device 1100 is mounted, or that can be accessed by the device without being part of the device. For example, the non-volatile memory / storage device 1108 may be accessed via a network via one or more input / output devices 1110.
[0078] One or more input / output devices 1110 may provide an interface for electronic device 1100 to communicate with any other suitable device. Input / output devices 1110 may include communication components, audio components, sensor components, etc. Network interface 1112 may provide an interface for electronic device 1100 to communicate via one or more networks. Electronic device 1100 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, such as accessing wireless networks based on communication standards, such as WiFi (Wireless Fidelity), 2G (2-Generation wireless telephone technology), 3G (3-Generation wireless telephone technology), 4G (4-Generation wireless telephone technology), 5G (5-Generation wireless telephone technology), etc., or combinations thereof, for wireless communication.
[0079] In one embodiment, at least one of the processors 1102 may be logically packaged with one or more controllers (e.g., memory controller modules) of the control module 1104. In one embodiment, at least one of the processors 1102 may be logically packaged with one or more controllers of the control module 1104 to form a system-in-package. In one embodiment, at least one of the processors 1102 may be integrated with the logic of one or more controllers of the control module 1104 on the same die. In one embodiment, at least one of the processors 1102 may be integrated with the logic of one or more controllers of the control module 1104 on the same die to form a system-on-a-chip.
[0080] In various embodiments, electronic device 1100 may be, but is not limited to, a server, desktop computing device, or mobile computing device (e.g., laptop computing device, handheld computing device, tablet computer, netbook, etc.). In various embodiments, electronic device 1100 may have more or fewer components and / or different architectures. For example, in some embodiments, electronic device 1100 includes one or more cameras, a keyboard, a liquid crystal display screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.
[0081] The detection device may use a main control chip as a processor or control module, and sensor data, position information, etc. may be stored in a memory or non-volatile memory / storage device. The sensor group may be used as an input / output device, and the communication interface may include a network interface.
[0082] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.
[0083] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0084] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.
[0085] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more blocks of a block diagram.
[0086] These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable terminal equipment, provide steps for implementing one or more processes of a flowchart and / or one or more blocks of a block diagram that specify the function.
[0087] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.
[0088] The above provides a detailed description of a model training method and apparatus, a text generation method and apparatus, an electronic device, and a machine-readable medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A method for processing images and text, characterized in that, include: The first image is scaled to the target size to obtain the second image; Determine whether the height of the second image is an integer multiple of the length of the local attention window, wherein the local attention window is the local attention window in the attention mechanism used by the visual encoder included in the large image-text model; If the height of the second image is not an integer multiple of the local attention window, the second image is filled in the height direction to obtain a third image, wherein the height of the third image is an integer multiple of the length of the local attention window; The third image is used as input data and processed by the large image-text model to obtain the processing result of the large image-text model.
2. The method according to claim 1, characterized in that, The third image is used as input data to the large image-text model for processing, and the processing result of the large image-text model is obtained, including: The third image is processed by a visual encoder included in the large image-text model to extract features, thereby obtaining a first visual embedding vector corresponding to the third image. The first visual embedding vector includes a second visual embedding vector corresponding to the second image and a third visual embedding vector corresponding to the filled portion in the third image. The third visual embedding vector is then deleted from the first visual embedding vector so that the large image-text model can perform inference based on the second visual embedding vector to obtain the processing result of the large image-text model.
3. The method according to claim 1, characterized in that, The scaling process of the first image, scaling the first image to a target size to obtain the second image, includes: The target size is determined based on the length of the local attention window of the visual encoder and the size of the image patch corresponding to the visual encoder; The first image is scaled proportionally to the target size to obtain the second image.
4. The method according to claim 3, characterized in that, The target size includes the target width and the target height; determining the target size based on the length of the local attention window of the visual encoder and the size of the image patch corresponding to the visual encoder includes: Calculate the ratio of the first parameter to the second parameter; wherein, the product of the length of the local attention window of the visual encoder and the first parameter is recorded as the first product, the product of the size of the image block corresponding to the visual encoder and the second parameter is recorded as the second product, and the ratio of the first product to the second product is equal to the ratio of the width to the height of the first image; The target width is determined based on the first parameter, the length of the local attention window of the visual encoder, and the width of the first image; wherein the target width does not exceed the width of the first image. The target height is determined based on the width and height of the first image and the target width.
5. The method according to claim 2, characterized in that, Deleting the third visual embedding vector from the first visual embedding vector includes: Based on the size of the filled portion in the third image, determine the third visual embedding vector corresponding to the filled portion in the first visual embedding vector; Delete the third visual embedding vector from the first visual embedding vector.
6. The method according to claim 5, characterized in that, The step of determining the third visual embedding vector corresponding to the filled portion in the first visual embedding vector based on the size of the filled portion in the third image includes: The third visual embedding vector corresponding to the filled portion in the first visual embedding vector is determined based on the ratio between the size of the second image and the size of the filled portion in the third image.
7. The method according to claims 1-6, characterized in that, The method includes: If the height of the second image is an integer multiple of the local attention window, the second image is not padded. Instead, the second image is directly used as input data into the large image-text model to obtain the processing result of the large image-text model.
8. A graphic processing device, characterized in that, include: The scaling module is used to scale the first image to a target size to obtain the second image; The determination module is used to determine whether the height of the second image is an integer multiple of the length of the local attention window, wherein the local attention window is the local attention window in the attention mechanism adopted by the visual encoder included in the large image-text model; A filling module is used to fill the second image in the height direction when the height of the second image is not an integer multiple of the local attention window, to obtain a third image, wherein the height of the third image is an integer multiple of the length of the local attention window; The inference module is used to input the third image as input data into the large image-text model for processing, and to obtain the processing result of the large image-text model.
9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the method as described in any one of claims 1-7.
10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the method as described in any one of claims 1-7.