Visual language model preprocessing method and related equipment
By combining optimal scaling and global graph scaling, along with foreground object segmentation and adaptive cross-modal fusion, the problem of semantic fragmentation of image sub-blocks and insufficient modal feature interaction in visual language models is solved, thereby improving the model's recognition accuracy and multimodal understanding capabilities.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-22
- Publication Date
- 2026-04-14
Smart Images
Figure CN121861674A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision, and more particularly to a visual language model preprocessing method and related equipment. Background Technology
[0002] Currently, large Visual Language Models (VLMs) are limited by fixed input sizes. When processing large images, preprocessing of the original image is usually required, including operations such as segmentation and cropping, sliding window sampling, or scaling, dividing the image into multiple sub-image patches before inputting them into the network for training or inference. Furthermore, VLMs typically use image-text pairs as input and extract visual and textual features separately using different encoders. However, the above processing methods have several significant drawbacks: The lack of local and global semantic connections between image sub-blocks leads to fragmented contextual information, making it difficult for the model to capture cross-regional overall features in the image. When a target object is segmented into different sub-blocks, incomplete local information may cause recognition errors. Furthermore, text and image features are processed by different encoders, resulting in insufficient information interaction between different modalities and difficulty in feature alignment, which limits the model's cross-modal understanding capabilities.
[0003] To address these issues, current computer vision methods primarily employ two preprocessing approaches: overlapping cropping and non-overlapping cropping. While overlapping cropping can improve performance to some extent, it introduces redundant computation, significantly increasing memory consumption and inference time. Non-overlapping methods typically introduce constraints such as cropping quantity or aspect ratio matching, for example, selecting a cropping ratio closest to the original image's aspect ratio to mitigate target distortion caused by non-uniform scaling. However, these methods still fail to adequately guarantee the semantic coherence of the target and lack effective mechanisms for feature interaction between text and visual modalities. Summary of the Invention
[0004] This application provides a pre-data processing method for visual language models, aiming to solve the problems in the prior art of lacking local and global semantic associations between image sub-blocks and weak feature interaction between text and visual modalities.
[0005] The first aspect of this invention provides a method for preprocessing visual language models, the method comprising: Determine the text features of the target input text; The original image is cropped to obtain the first set of cropped sub-blocks; The original image is scaled to the standard input size of the visual encoder to obtain a scaled global map; Determine the set of foreground objects corresponding to the original image, and determine the second set of cropped sub-blocks based on the set of foreground objects; The visual features corresponding to the original image are determined based on the first set of cropped sub-blocks, the scaled global image, and the second set of cropped sub-blocks. The text features are fused with the visual features to obtain cross-modal fusion features.
[0006] By adopting the above technical solutions, the combination of optimal cropping and global image scaling avoids redundant calculations in overlapping cropping while ensuring the correlation between local image details and global semantics, thus solving the context fragmentation problem caused by traditional cropping. Through foreground object segmentation and targeted cropping, the foreground target is completely preserved in the second cropping sub-block set, effectively avoiding the breakage of subject information and significantly improving the model's recognition accuracy for small targets and multiple foreground objects. Through multi-source visual feature integration and adaptive cross-modal fusion mechanism, deep alignment of text and visual features is achieved, eliminating semantic bias and scale differences between modalities, and greatly enhancing the model's ability to understand and reason about multimodal content.
[0007] A second aspect of the present invention provides a visual language model pre-data processing apparatus, the apparatus comprising: The determination module is used to determine the text features of the target input text; The cropping module is used to crop the original image to obtain a first set of cropped sub-blocks; A scaling module is used to scale the original image to the standard input size of the visual encoder to obtain a scaled global image; The cropping module is further configured to determine the set of foreground objects corresponding to the original image, and determine a second set of cropping sub-blocks based on the set of foreground objects; The determining module is further configured to determine the visual features corresponding to the original image based on the first set of cropped sub-blocks, the scaled global image, and the second set of cropped sub-blocks; A fusion module is used to fuse the text features with the visual features to obtain cross-modal fusion features; In one possible design, the cutting module is specifically used for: The cropping quantity corresponding to the original image is determined by the following formula: ; Where N is the number of cropped images, W is the width of the original image, H is the height of the original image, and S*S is the standard input size of the visual language model; Enumerate all factor combinations that satisfy the product of the first factor m and the second factor n equals N, and store the factor combinations in a list of undetermined ratios, where m is the number of crop blocks in the height of the original image and n is the number of crop blocks in the width of the original image. The optimal cutting ratio is determined by iterating through the list of undetermined ratio values using a cropping aspect ratio filtering function, which is: ; According to the optimal cropping ratio, the original image is cropped into N first cropping sub-blocks; The cut sub-blocks whose areas do not meet the standard input size among the N first cut sub-blocks are completed to obtain the first cut sub-block set.
[0008] By adopting the above technical solution, the cropping module calculates the number of cropping operations based on the matching relationship between the original image size and the standard input size of the visual encoder. Then, by enumerating factor combinations and combining them with the cropping aspect ratio filtering function, the optimal cropping ratio can be determined, which can effectively avoid computational redundancy caused by overlapping areas.
[0009] In one possible design, the cutting module is further specifically used for: The N first trimmed sub-blocks whose areas do not meet the standard input size are padded to obtain the first trimmed sub-block set, which includes: Determine which of the N first cut sub-blocks has an area that does not meet the standard input size, and obtain the sub-block to be completed; The number of pixels to be filled is calculated based on the difference between the sub-block to be filled and the standard input size. Based on the number of pixels required to be filled, a zero-fill operation is performed on the sub-block to be filled in order to complete the sub-block.
[0010] By adopting the above technical solution, on the one hand, it can accurately locate the sub-blocks that do not meet the size requirements and quantify the filling needs, ensuring that all first cropped sub-blocks are uniformly the standard input size (S*S) of the visual encoder after filling, meeting the compatibility requirements of the visual language model for the input format, and avoiding feature extraction interruption or semantic deviation caused by size mismatch; on the other hand, the zero-filling operation only fills the size by supplementing the edge regions with a pixel value of 0, without changing the pixel information and semantic features such as local texture and contour of the original effective area of the sub-block, effectively avoiding the noise interference that may be introduced by non-zero filling (such as mean filling and random filling), and ensuring the semantic coherence between the sub-block and the original image.
[0011] In one possible design, the cutting module is further specifically used for: The foreground and background of the original image are segmented to determine the set of foreground objects corresponding to the original image. The set of foreground objects contains M foreground target objects, where M is an integer greater than or equal to 1. Solve for the minimum bounding rectangle for each of the M foreground target objects, and expand the minimum bounding rectangle into a square with the length of the long side based on the length of the long side; The original image is cropped according to the square, retaining the image area within the square to obtain the second set of cropped sub-blocks.
[0012] By adopting the above technical solution, the foreground target object in the original image is accurately located through foreground and background segmentation. The minimum bounding rectangle of the target foreground object is expanded into a square. On the one hand, this ensures that the foreground target object is completely contained in the second cropping sub-block set. On the other hand, it also makes the second cropping sub-block compatible with the standard input size of the visual encoder. The final second cropping sub-block set focuses on the foreground subject, providing the visual encoder with input supplements for local detail enhancement. Compared with relying solely on the global map or random cropping sub-blocks, this solution improves the model's accuracy in recognizing foreground objects. It is especially suitable for scenes containing small targets or multiple foreground objects, effectively solving the technical problem of subject information fragmentation in background technology.
[0013] In one possible design, the fusion module is specifically used for: The final output vector of the attention mechanism, the text-adaptive visual fusion feature, and the visual-adaptive text fusion feature are determined based on the text features and the visual features. The final output vector of the attention mechanism, the text-adaptive visual fusion feature, and the visual-adaptive text fusion feature are merged according to the channel dimension to obtain the merged feature vector; The merged feature vector is input into a learnable convolutional layer to determine the cross-modal fusion feature.
[0014] In one possible design, the fusion module is further specifically used for: The visual features are rearranged to obtain a first visual feature vector and a second visual feature vector, and the first visual feature vector and the second visual feature vector are multiplied to determine the attention score vector; The text features are rearranged to obtain a text feature vector; The attention score vector is multiplied by the text feature vector to obtain the final output vector of the attention mechanism; An activation function is applied to the text feature vector to obtain text weight features; The text weight features are multiplied by the final output vector of the attention mechanism to determine the text adaptive visual fusion features. An activation function is applied to the first visual feature vector to obtain visual weight features; The visual weight features are multiplied by the final output vector of the attention mechanism to determine the visual adaptive text fusion features.
[0015] In one possible design, the fusion module is further specifically used for: The final output vector of the attention mechanism is determined by the following formula: ; in The final output vector of the attention mechanism is... Let the attention score vector be... For text feature vectors, , The first visual feature vector, This is the second visual feature vector; The step of multiplying the text weight features with the final output vector of the attention mechanism to determine the text adaptive visual fusion features includes: The text adaptive visual fusion features are determined using the following formula: ; ; in For text weight features, The text is an adaptive visual fusion feature; The step of multiplying the visual weight features with the final output vector of the attention mechanism to determine the visual adaptive text fusion features includes: The visual adaptive text fusion features are determined using the following formula: ;
[0016] ; in, Visual weight features This refers to the visual adaptive text fusion feature.
[0017] By adopting the above technical solution, and by constructing the final output vector of the attention mechanism, text-adaptive visual fusion features, and visual-adaptive text fusion features, a merged feature vector is obtained, realizing deep interaction between text and visual features. The final output vector of the attention mechanism ensures the initial alignment of modal information, the text-adaptive visual fusion features make visual features focus on text semantics, and the visual-adaptive text fusion features make text features align with the visual subject. This can effectively eliminate feature scale differences and semantic biases between modalities, and significantly enhance the model's ability to understand and reason about multimodal content.
[0018] A third aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the visual language model pre-data processing method described in the first aspect above.
[0019] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the visual language model pre-data processing method described in the first aspect above.
[0020] In summary, this application includes at least one of the following beneficial technical effects: 1. By designing a cropping ratio filtering function, the number of cropping is calculated based on the matching relationship between the original image size and the standard input size of the visual encoder. Then, by enumerating factor combinations and combining them with the cropping aspect ratio filtering function, the optimal cropping ratio can be determined. This can more approximately select the optimal cropping ratio relative to the original image, and then crop the original image to obtain the required first cropping sub-block set, effectively avoiding computational redundancy caused by overlapping areas.
[0021] 2. By introducing a foreground target segmentation and cropping mechanism, the foreground target object in the original image is accurately located through foreground and background segmentation. The minimum bounding rectangle of the target foreground object is expanded into a square. This ensures that the foreground target object is completely contained in the second cropping sub-block set, and also makes the second cropping sub-block compatible with the standard input size of the visual encoder. The final second cropping sub-block set focuses on the foreground subject, providing the visual encoder with input supplements for local detail enhancement. Compared with relying solely on the global map or random cropping sub-blocks, this scheme improves the model's accuracy in recognizing foreground objects, and is especially suitable for scenes containing small targets or multiple foreground objects. It effectively solves the technical problem of subject information fragmentation in background technology.
[0022] 3. By constructing the final output vector of the attention mechanism, text-adaptive visual fusion features, and visual-adaptive text fusion features through an adaptive cross-modal attention module, a merged feature vector is obtained, realizing deep interaction between text and visual features. The final output vector of the attention mechanism ensures the initial alignment of modal information, the text-adaptive visual fusion features focus visual features toward text semantics, and the visual-adaptive text fusion features align text features toward the visual subject. This can effectively eliminate feature scale differences and semantic biases between modalities, significantly enhancing the model's understanding and reasoning ability for multimodal content, realizing deep fusion of text and image features, and further enhancing the model's understanding and reasoning ability for multimodal content. Attached Figure Description
[0023] Figure 1This is a flowchart illustrating a visual language model pre-data processing method provided in an embodiment of this application; Figure 2 This is a schematic diagram of foreground object segmentation provided in an embodiment of this application; Figure 3 This is a schematic diagram of an adaptive cross-modal cross-attention feature fusion module provided in an embodiment of this application; Figure 4 This is a schematic diagram of the virtual structure of a visual language model pre-data processing device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the hardware structure of the server provided in the embodiments of this application. Detailed Implementation
[0024] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to any or all possible combinations including one or more of the listed items.
[0025] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.
[0026] The following describes a method for processing data before visual language models from the perspective of a data processing device for visual language models. This data processing device can be a server or a service unit within a server, without any specific limitation. For the sake of simplicity, the following description uses a server as an example to illustrate the method.
[0027] Please see Figure 1 , Figure 1 A flowchart illustrating a pre-data processing method for a visual language model provided in an embodiment of the present invention includes: 101. Determine the text features of the target input text.
[0028] In this embodiment, the server can determine the text features of the target input text. Specifically, text features can be output through a text encoder such as BERT-base. Its dimensions are .
[0029] 102. Crop the original image to obtain the first set of cropped sub-blocks.
[0030] In this embodiment, the server crops the original image to obtain a first set of cropped sub-blocks, including: First, the server uses the formula The optimal cropping number for the original image is determined, where N is the cropping number, W is the width of the original image, H is the height of the original image, S*S is the standard input size of the visual language model, and ceil(·) is the floor function. In this embodiment, the standard input size of the visual language model is 448*448 pixels. Using the above formula, the optimal cropping number that best matches the standard input size of the original image can be determined very quickly and effectively. For example, if the width of the original image is 1344 pixels and the height is 896 pixels, N=6 can be easily obtained.
[0031] Secondly, enumerate all factor combinations that satisfy the condition that the product of the first factor m and the second factor n equals N, and store these factor combinations in a list of undetermined ratios. In each factor combination, m represents the number of crop blocks in the original image width, and n represents the number of crop blocks in the original image height, with both m and n being positive integers. Continuing with N=6 as an example, all combinations that satisfy this condition... The combinations are: (1, 6), (6, 1), (2, 3), (3, 2).
[0032] The optimal cropping ratio is determined by iterating through the list of undetermined ratios using the cropping aspect ratio filtering function. The cropping aspect ratio filtering function is as follows: Based on the optimal cropping ratio, the original image is cropped into N first cropping sub-blocks. Cropping sub-blocks whose dimensions do not meet the standard input dimensions are padded to obtain a set of first cropping sub-blocks. In this embodiment, zero-padding is used to pad the insufficiently sized cropping sub-blocks; the specific zero-padding steps are not discussed in detail here.
[0033] It is understood that the cropping aspect ratio filtering function provided in this embodiment is a relatively optimal filtering function. This function calculates the absolute value between the factor ratio and the original image's aspect ratio, and determines the factor combination with the smallest function return value as the optimal cropping ratio. Depending on the specific application scenario, there can be various different filtering functions. Continuing with the example of the original image with N=6 and factor combinations (1, 6), (6, 1), (2, 3), and (3, 2), the cropping aspect ratio filtering function easily yields the factor combination with the smallest function return value as (3, 2).
[0034] It should be noted that the original image provided in this embodiment can be a regular quadrilateral or an irregular image. When the original image is an irregular shape, the smallest bounding rectangle of the original image is determined, and it is filled to make it a regular quadrilateral before the subsequent process is executed.
[0035] 103. Scale the original image to the standard input size of the visual encoder to obtain a scaled global map.
[0036] In this embodiment, the server scales the original image to the standard input size of the visual encoder, which is also the standard input size S*S of the visual language model. The visual encoder can use VisionTransformer and employ a bilinear interpolation algorithm to maintain image sharpness.
[0037] 104. Determine the set of foreground objects corresponding to the original image, and determine the set of second cropping sub-blocks based on the set of foreground objects; In this embodiment, the server can determine the set of foreground objects corresponding to the original image, and determine the second set of cropped sub-blocks based on the set of foreground objects. For ease of understanding, the following will be combined with... Figure 2 A detailed explanation of how to determine the second set of cut sub-blocks is provided: Please refer to Figure 2 , Figure 2 This is a schematic diagram of foreground object segmentation provided in an embodiment of this application. First, the server segments the foreground and background of the original image to determine the set of foreground objects corresponding to the original image. The set of foreground objects contains M foreground target objects, where M is an integer greater than or equal to 1. Figure 2 As shown. The foreground object can be determined by the range of pixel coordinates. For example, when one foreground object is identified, its coordinate range is: x1=500, y1=300; x2=800, y2=600, then the set of foreground objects O={o1} (M=1). It is easy to understand that the segmentation of foreground and background here can be achieved using traditional digital image processing methods or deep learning-based methods, without any specific limitation.
[0038] Secondly, based on the acquired set of foreground objects, the server needs to solve for the minimum bounding rectangle for each of the M foreground objects. If the minimum bounding rectangle is already a square, it does not need to be expanded. If the minimum bounding rectangle is a rectangle, it is expanded into a square with the length of the long side, in order to avoid deformation when scaling to the standard input size of the visual encoder.
[0039] Finally, the server crops the original image based on the square regions corresponding to the smallest bounding rectangles of the M foreground objects, retains the image regions within the square regions, and then scales the images within the square regions to the standard input size of the visual encoder to obtain the second set of cropped sub-blocks.
[0040] It should be noted that the server can obtain the first set of clipped sub-blocks through step 102, determine the scaled global graph through step 103, and determine the second set of clipped sub-blocks through step 104. However, there is no restriction on the execution order of these three steps. Step 102 can be executed first, or step 103 can be executed first, or step 104 can be executed first, or they can be executed simultaneously. There is no specific limitation.
[0041] 105. Determine the visual features corresponding to the original image based on the first set of cropped sub-blocks, the scaled global image, and the second set of cropped sub-blocks.
[0042] In this embodiment, the service determines the visual features corresponding to the original image based on the first set of cropped sub-blocks, the scaled global image, and the second set of cropped sub-blocks by: inputting the first set of cropped sub-blocks, the scaled global image, and the second set of cropped sub-blocks into a visual encoder to obtain a first sub-block feature set, a global image feature vector, and a second sub-block feature set, respectively; performing feature aggregation operations on the first sub-block feature set and the second sub-block feature set to obtain a first aggregated feature and a second aggregated feature, respectively; performing normalization processing on the first aggregated feature, the second aggregated feature, and the global image feature vector to unify the modulus and eliminate scale differences between features from different sources; concatenating the normalized first aggregated feature, the second aggregated feature, and the global image feature vector, integrating channel information through linear transformation, and processing through an activation function to finally output the visual features corresponding to the original image.
[0043] The first sub-block feature set and the second sub-block feature set correspond to the local visual features of the first cropped sub-block set and the second cropped sub-block, respectively. These local visual features mainly include detailed information such as edges, textures, and local contours. The global image feature vector corresponds to the overall composition, main color distribution, and large-scale structure of the original image, as well as other macroscopic information. The server aggregates these local visual feature vectors, that is, it obtains the first aggregated feature and the second aggregated feature. The server then normalizes the first aggregated feature, the second aggregated feature, and the global image feature vector to unify the modulus, eliminate scale differences, and integrates their channel information through linear transformation to output the visual features corresponding to the original image.
[0044] It is easy to understand that the above aggregation operation can aggregate multiple local features through pooling or statistical calculation. The normalization of feature vectors is often achieved by L2 normalization, batch normalization, etc. to eliminate scale differences. Channel integration is achieved through convolution or preset projection matrix to obtain the visual features corresponding to the original image. The specific process will not be discussed in detail here.
[0045] 106. Fuse textual features with visual features to obtain cross-modal fusion features.
[0046] In this embodiment, after determining textual and visual features, the server fuses the text and visual features to determine cross-modal fusion features. Specifically, an adaptive cross-modal cross-attention module is used to acquire adaptive cross-modal fusion features. For ease of understanding, the following will be combined with... Figure 3 A detailed explanation of how to determine cross-modal fusion features is provided: Please see Figure 3 , Figure 3 This is a schematic diagram of an adaptive cross-modal cross-attention feature fusion module provided in an embodiment of this application: First, the adaptive cross-modal cross-attention module uses the output of the text encoder. and the output of the visual encoder As input, since an attention mechanism is used between text and visual modalities, the two feature sequences must first be made to have the same dimension. Therefore, we first assume that the output of the visual encoder... Dimensions Where B is the batch size, i.e., the number of image samples input into the model at one time; C is the channel dimension of the feature map; H is the vertical dimension of the feature map, corresponding to the top and bottom dimensions of the image; and W is the horizontal dimension of the feature map. Perform feature rearrangement and copy it into two feature vectors, respectively. and That is, the first visual feature vector and the second visual feature vector, where N is the number of tokens. The first visual feature vector can be considered as the K value, and the second visual feature vector as the V value. and Vector multiplication yields the visual feature weight vector. A score matrix representing the importance of each feature;
[0047] Secondly, regarding text features In this embodiment, its dimension is It should be noted that the dimensions of text features are generally (B, L, D), where B represents the batch size, L represents the length of the text sequence (e.g., the number of words), and D represents the feature dimension of a single character or word. For example, (8, 32, 768) represents 8 text samples, each with 32 words, and each word with 768 features. In this scheme, since the text features need to be fused with the visual features through convolutional layers, while preserving the semantic relevance of the text, the text features need to complete the dimensional mapping from sequence features to spatial features. First, interpolation is used to align the length of the text feature sequence with the total number of pixels in the H×W space of the visual features, and linear projection is used to match the number of channels of the visual features. Then, through dimensional transpose and reshaping, and optionally superimposed with learnable spatial encoding and 1×1 convolution to enhance semantic relevance, the text features are transformed from (B, L, D) to (B, C, H, W). After feature rearrangement, we get This vector can be used as the Q-value, and its core can serve as a semantic retrieval probe for text, accurately capturing the semantic relationships between text and images, ensuring that cross-modal fusion does not deviate from the textual intent. Next, and Multiply, we get Fc is the final output vector of the attention mechanism, representing Weighted characteristics:
[0048] The above describes the conventional cross-modal attention mechanism. This method also utilizes text information... After passing through the sigmoid function, we get That is, text weight features, which are then compared with... Dot product is used to obtain the visual information features that the text adaptively focuses on. :
[0049] Similarly, visual information Visual weight features are obtained using the sigmoid function. The visual weight feature is combined with Dot product to obtain text information features that adaptively focus on visual information. :
[0050] Finally Channel merging is performed to preserve the complete semantic information of each of the three types of features, avoiding the loss of details, in order to obtain the merged feature vector. It should be noted that... The batch size and number of tokens are consistent across all three types of features. This is the core premise for channel merging and cross-modal fusion. All three types of features come from the same batch of input text-image samples (e.g., processing 8 samples simultaneously). Throughout the entire process from text and visual feature extraction to attention calculation, the batch dimension is always bound to the number of input samples, ensuring that the text and visual features of each sample correspond one-to-one and there will be no sample mismatch. In the feature rearrangement stage, the number of tokens for text features and visual features has been unified to the same N through operations such as dimension mapping, sequence space transformation, and linear projection. The attention mechanism calculation does not change the token number dimension, so the number of channels in the merged feature vector is 3C (e.g., when C=768, the number of channels after merging is 2304). Then, it is input into a learnable convolutional layer to output cross-modal fusion features with adaptive functionality.
[0051] In summary, the embodiments provided in this application effectively solve the core problems of semantic fragmentation of image sub-blocks, broken foreground subject information, and insufficient cross-modal feature interaction in the prior art by forming a synergistic technical effect through multi-dimensional optimization strategies. Specifically, in the image cropping stage, based on the matching relationship between the original image size and the standard input size of the visual encoder, the optimal cropping ratio is accurately determined by calculating the cropping quantity, enumerating factor combinations, and using the aspect ratio filtering function. This avoids redundant calculations of overlapping cropping and preserves the semantic relationship between the local and global parts of the image to the greatest extent. In the foreground object processing stage, foreground-background segmentation, minimum bounding rectangle expansion, and targeted cropping ensure that the foreground target object is completely contained in the second cropping sub-block set, significantly improving the recognition accuracy in scenarios with small targets and multiple foreground objects, and completely solving the technical pain point of broken subject information. In the feature fusion stage, an adaptive cross-modal cross-attention mechanism is used to construct the attention mechanism output vector, text-adaptive visual fusion features, and visual-adaptive text fusion features. Through channel merging and convolution integration, the depth alignment of text and visual features is achieved, eliminating scale differences and semantic deviations between modalities. This method balances processing efficiency and feature quality in its overall process. While reducing computational overhead, it significantly enhances the model's understanding and reasoning capabilities for multimodal content, providing superior preprocessing support for the training and inference of visual language models. The embodiments of this application have been described above from the perspective of a method for pre-processing data of a visual language model. The embodiments of this application will now be described below from the perspective of a device for pre-processing data of a visual language model.
[0052] Please see Figure 4 , Figure 4 This is a virtual structural diagram of a visual language model pre-data processing device 400 provided in an embodiment of this application. The visual language model pre-data processing device 400 includes: The determination module 401 is used to determine the text features of the target input text; The cropping module 402 is used to crop the original image to obtain a first set of cropped sub-blocks; The scaling module 403 is used to scale the original image to the standard input size of the visual encoder to obtain a scaled global image; The cropping module 402 is further configured to determine the set of foreground objects corresponding to the original image, and determine the second set of cropping sub-blocks based on the set of foreground objects; The determining module 401 is further configured to determine the visual features corresponding to the original image based on the first set of cropped sub-blocks, the scaled global image, and the second set of cropped sub-blocks; The fusion module 404 is used to fuse the text features with the visual features to obtain cross-modal fusion features.
[0053] Through the above scheme, the visual language model pre-data processing device provided in this embodiment, through the collaborative design and functional division of labor among modules, accurately implements the technical concept of the above optimization method, and has significant structural advantages and practical value. The determination module, cropping module, scaling module, and fusion module each perform their respective functions and work in deep coordination: the cropping module achieves comprehensive capture of global image information, local details, and main features through a dual cropping mechanism, namely, optimal scaling of the first cropping sub-block set and foreground-targeted scaling of the second cropping sub-block set; the scaling module ensures the compatibility between the global image and the input size of the visual encoder, laying the foundation for the integration of multi-source visual features; the determination module outputs visual features with both integrity and consistency through the aggregation, normalization, and linear integration of multi-source features; and the fusion module achieves deep interaction and alignment between text and visual features through an adaptive cross-modal cross-attention structure. The device is compact and highly adaptable, and can be flexibly connected to different types of visual encoders and text encoders. While avoiding redundant calculations, it efficiently outputs cross-modal fusion features, which not only ensures the semantic coherence between image sub-blocks and the integrity of foreground subject information, but also significantly improves the accuracy of visual language models in understanding multimodal content and inference efficiency, providing reliable device support for the engineering application of related equipment.
[0054] This embodiment also discloses a computer device, which may be a server, and its internal structure diagram may be as follows. Figure 5As shown. The computer device includes at least one server 501, at least one network interface 504 or other user interface 503, memory 505, and at least one communication bus 502. The server 500 may optionally include the user interface 503, including a display, keyboard, or click device. The memory 505 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device. The memory 505 stores execution instructions. When the server 500 is running, the processor 501 communicates with the memory 505, and the processor 501 calls the instructions stored in the memory 505 to execute the aforementioned visual language model pre-data processing method. An operating system 506 contains various programs for implementing various basic business operations and handling tasks based on the hardware.
[0055] The server provided in this application embodiment can execute the technical solution of the above-described embodiment of a visual language model pre-data processing method. Its implementation principle and technical effect are similar, and will not be repeated here.
[0056] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0057] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.
[0058] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0059] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0060] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0061] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0062] This invention also provides a computer program product comprising computer software instructions that, when executed on a processing device, cause the processing device to perform actions such as... Figure 1 The process in the corresponding embodiment.
[0063] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).
[0064] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0065] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.
[0066] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0067] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0068] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0069] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for pre-processing data for visual language models, characterized in that, The method includes: Determine the text features of the target input text; The original image is cropped to obtain the first set of cropped sub-blocks; The original image is scaled to the standard input size of the visual encoder to obtain a scaled global map; Determine the set of foreground objects corresponding to the original image, and determine the second set of cropped sub-blocks based on the set of foreground objects; The visual features corresponding to the original image are determined based on the first set of cropped sub-blocks, the scaled global image, and the second set of cropped sub-blocks. The text features are fused with the visual features to obtain cross-modal fusion features.
2. The data processing method according to claim 1, characterized in that, The process of cropping the original image to obtain the first set of cropped sub-blocks includes: The cropping quantity corresponding to the original image is determined by the following formula: ； Where N is the number of cropped images, W is the width of the original image, H is the height of the original image, and S*S is the standard input size of the visual language model; Enumerate all factor combinations that satisfy the product of the first factor m and the second factor n equals N, and store the factor combinations in a list of undetermined ratios, where m is the number of crop blocks in the height of the original image and n is the number of crop blocks in the width of the original image. The optimal cutting ratio is determined by iterating through the list of undetermined ratio values using a cropping aspect ratio filtering function, which is: ； According to the optimal cropping ratio, the original image is cropped into N first cropping sub-blocks; The cut sub-blocks whose areas do not meet the standard input size among the N first cut sub-blocks are completed to obtain the first cut sub-block set.
3. The method according to claim 2, characterized in that, The step of completing the first set of clipping sub-blocks by filling in the areas of the N first clipping sub-blocks that do not meet the standard input size, to obtain the first set of clipping sub-blocks, includes: Determine which of the N first cut sub-blocks has an area that does not meet the standard input size, and obtain the sub-block to be completed; The number of pixels to be filled is calculated based on the difference between the sub-block to be filled and the standard input size. Based on the number of pixels required to be filled, a zero-fill operation is performed on the sub-block to be filled in order to complete the sub-block.
4. The data processing method according to claim 1, characterized in that, Determining the set of foreground objects corresponding to the original image, and determining the second set of cropped sub-blocks based on the set of foreground objects, includes: The foreground and background of the original image are segmented to determine the set of foreground objects corresponding to the original image. The set of foreground objects contains M foreground target objects, where M is an integer greater than or equal to 1. Solve for the minimum bounding rectangle for each of the M foreground target objects, and expand the minimum bounding rectangle into a square with the length of the long side based on the length of the long side; The original image is cropped according to the square, retaining the image area within the square to obtain the second set of cropped sub-blocks.
5. The data processing method according to claim 1, characterized in that, The step of fusing the text features with the visual features to obtain cross-modal fusion features includes: The final output vector of the attention mechanism, the text-adaptive visual fusion feature, and the visual-adaptive text fusion feature are determined based on the text features and the visual features. The final output vector of the attention mechanism, the text-adaptive visual fusion feature, and the visual-adaptive text fusion feature are merged according to the channel dimension to obtain the merged feature vector; The merged feature vector is input into a learnable convolutional layer to determine the cross-modal fusion feature.
6. The method according to claim 5, characterized in that, The step of determining the final output vector of the attention mechanism, the text-adaptive visual fusion feature, and the visual-adaptive text fusion feature based on the text features and the visual features includes: The visual features are rearranged to obtain a first visual feature vector and a second visual feature vector, and the first visual feature vector and the second visual feature vector are multiplied to determine the attention score vector; The text features are rearranged to obtain a text feature vector; The attention score vector is multiplied by the text feature vector to obtain the final output vector of the attention mechanism; An activation function is applied to the text feature vector to obtain text weight features; The text weight features are multiplied by the final output vector of the attention mechanism to determine the text adaptive visual fusion features. An activation function is applied to the first visual feature vector to obtain visual weight features; The visual weight features are multiplied by the final output vector of the attention mechanism to determine the visual adaptive text fusion features.
7. The method according to claim 6, characterized in that, The step of multiplying the attention score vector with the text feature vector to obtain the final output vector of the attention mechanism includes: The final output vector of the attention mechanism is determined by the following formula: ； in The final output vector of the attention mechanism is... Let the attention score vector be... For text feature vectors, , The first visual feature vector, This is the second visual feature vector; The step of multiplying the text weight features with the final output vector of the attention mechanism to determine the text adaptive visual fusion features includes: The text adaptive visual fusion features are determined using the following formula: ； in For text weight features, The text is an adaptive visual fusion feature; The step of multiplying the visual weight features with the final output vector of the attention mechanism to determine the visual adaptive text fusion features includes: The visual adaptive text fusion features are determined using the following formula: ； ； in, Visual weight features This refers to the visual adaptive text fusion feature.
8. A data processing apparatus, characterized in that, include: The determination module is used to determine the text features of the target input text; The cropping module is used to crop the original image to obtain a first set of cropped sub-blocks; A scaling module is used to scale the original image to the standard input size of the visual encoder to obtain a scaled global image; The cropping module is further configured to determine the set of foreground objects corresponding to the original image, and determine a second set of cropping sub-blocks based on the set of foreground objects; The determining module is further configured to determine the visual features corresponding to the original image based on the first set of cropped sub-blocks, the scaled global image, and the second set of cropped sub-blocks; The fusion module is used to fuse the text features with the visual features to obtain cross-modal fusion features.
9. An electronic device, characterized in that, include: The electronic device includes a processor, a memory, a user interface, and a network interface, wherein the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed, perform the method as described in any one of claims 1-7.