Visual information processing methods, devices, electronic devices and storage media
By identifying and cropping valid text regions in images and adjusting the image scaling based on the semantic prediction of the query text, the problem of redundant data in visual language models is solved, achieving faster inference speed and lower computational resource consumption.
Patent Information
- Application Number
- CN202611460744.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-18
- Publication Date
- 2026-11-17
AI Technical Summary
In existing technologies, visual language models are easily affected by redundant data when processing images with complex backgrounds, resulting in slow inference speed and high consumption of computing resources.
By identifying valid text regions in the original image for cropping, predicting the target scaling rate based on the semantics of the query text, and adjusting the cropped image, redundant visual data input to the visual language model is reduced.
While preserving the core semantic information of the image, redundant visual data input to the visual language model is significantly reduced, thereby accelerating the inference speed of the downstream model and reducing the consumption of computing resources.
Smart Images

Figure CN123119476A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of data processing technology, and specifically to a visual information processing method, apparatus, electronic device, and storage medium. Background Technology
[0002] With the rapid development of multimodal large model technology, visual language models have been widely used in document understanding, optical character recognition, and visual question answering. These models typically divide the input image into image patches and map them to visual tokens, which are then fed together with the input query text into the model for joint analysis and reasoning. In real-world scenarios (such as processing photographed or scanned document images), the input image often contains large areas of invalid background (such as blank page margins, photographed desktops, etc.).
[0003] In related technologies, an end-to-end full-image input method is typically used. This involves uniformly dividing a high-resolution original image containing the complete scene (covering complex backgrounds, the main body of the document, and internal blank areas) into image patches. These patches are then converted into a visual token sequence by a visual encoder and directly input into a large visual language model. The model, relying on its internal self-attention mechanism, adaptively learns to distinguish between document content and background noise from all tokens in the full image, directly outputting a structured answer, thus achieving end-to-end joint modeling from image to semantics. Alternatively, a two-stage method based on document parsing can be used. The first stage uses an independent document parsing model to perform text detection and recognition on the input image, outputting a set of structured information containing text content, location coordinates, and confidence scores. The second stage uses this sequence (often after layout reorganization) as the sole input to a large language model for field extraction and question answering. In this process, visual information such as font, color, layout, and non-text elements of the original image is stripped away, and inference is entirely based on the text symbol sequence.
[0004] In this approach, a large amount of redundant data may be input into the visual language model, which can easily affect the inference speed of downstream models and increase additional computational overhead. Summary of the Invention
[0005] This disclosure aims to at least partially address one of the technical problems in the related art.
[0006] Therefore, the purpose of this disclosure is to propose a visual information processing method, apparatus, electronic device, and storage medium, which can significantly reduce redundant visual data input into a visual language model while preserving the core semantic information of an image, thereby accelerating the inference speed of downstream models and reducing computational resource consumption.
[0007] To achieve the above objectives, the visual information processing method proposed in the first aspect of this disclosure includes: Retrieve a data pair containing the original image and the query text; Identify valid text regions in the original image, and crop the original image based on the valid text regions to obtain a cropped image; Based on the semantics of the query text and the cropped image, predict the corresponding target scaling ratio; The cropped image is adjusted based on the target scaling ratio to obtain a scaled image; The scaled image and the query text are processed as input to a visual language model.
[0008] To achieve the above objectives, a visual information processing apparatus according to a second aspect of this disclosure includes: The acquisition module is used to acquire data pairs containing the original image and the query text; The recognition module is used to identify valid text regions in the original image and crop the original image according to the valid text regions to obtain a cropped image; The prediction module is used to predict the corresponding target scaling ratio based on the semantics of the query text and the cropped image; An adjustment module is used to adjust the cropped image based on the target scaling ratio to obtain a scaled image; The processing module is used to process the scaled image and the query text as input to the visual language model.
[0009] The electronic device proposed in the third aspect of this disclosure includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the visual information processing method proposed in the first aspect of this disclosure.
[0010] A fourth aspect of this disclosure provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the visual information processing method as proposed in the first aspect of this disclosure.
[0011] A fifth aspect of this disclosure provides a computer program product in which, when instructions in the computer program product are executed by a processor, the visual information processing method as described in a first aspect of this disclosure is performed.
[0012] The visual information processing method, apparatus, electronic device, and storage medium disclosed herein acquire a data pair containing an original image and query text; identify valid text regions in the original image and crop the original image based on the valid text regions to obtain a cropped image; predict the corresponding target scaling ratio based on the semantics of the query text and the cropped image; adjust the cropped image based on the target scaling ratio to obtain a scaled image; and process the scaled image and query text as input to a visual language model. Therefore, while preserving the core semantic information of the image, redundant visual data input to the visual language model can be significantly reduced, thereby accelerating the inference speed of downstream models and reducing computational resource consumption.
[0013] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure. Attached Figure Description
[0014] The above and / or additional aspects and advantages of this disclosure will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, in which: Figure 1 This is a schematic flowchart of a visual information processing method proposed in an embodiment of this disclosure; Figure 2 This is a schematic flowchart of a visual information processing method according to another embodiment of this disclosure; Figure 3 This is a flowchart of the model inference process after adding a visual information adaptive compression module according to the present disclosure; Figure 4 This is a schematic diagram of the visual information extraction module processing flow proposed in this disclosure; Figure 5 This is a schematic diagram of the processing flow of the adaptive resolution adjustment module proposed in this disclosure; Figure 6 This is a schematic diagram of the structure of a visual information processing device according to an embodiment of the present disclosure; Figure 7 This is a block diagram of an electronic device according to an embodiment of this application. Detailed Implementation
[0015] Embodiments of this disclosure are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are used only to explain this disclosure, and should not be construed as limiting this disclosure. Rather, embodiments of this disclosure include all variations, modifications, and equivalents falling within the spirit and scope of the appended claims.
[0016] It should be noted that the examples in this application may involve user data, data acquisition, and / or use. These aspects all comply with relevant laws, regulations, and provisions. In the examples of this application, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, when implementing each example, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained through appropriate means in accordance with relevant laws and regulations. The specific methods of notification and / or authorization can vary depending on the actual situation and application scenario, and the scope of the solution is not limited in this regard.
[0017] In this manual and the sample solutions, any processing of personal information will be conducted only under legal grounds (such as obtaining the consent of the data subject or being necessary for the performance of a contract) and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.
[0018] Figure 1 This is a schematic flowchart of a visual information processing method proposed in an embodiment of this disclosure.
[0019] It should be noted that the execution subject of the visual information processing method in this embodiment is a visual information processing device, which can be implemented by software and / or hardware. The device can be configured in an electronic device, which may include, but is not limited to, a terminal, a server, etc., such as a mobile phone, a PDA, etc.
[0020] like Figure 1 As shown, the visual information processing method includes: S101: Retrieve a data pair containing the original image and the query text.
[0021] The original image refers to the initial image that contains the visual content to be analyzed but has not been processed by this scheme.
[0022] The query text refers to the natural language text entered by the user to guide or limit the objectives of the image processing task.
[0023] In this context, a data pair refers to a pair of data that appear in pairs and have a relationship (in this embodiment, it refers to the original image and the corresponding query text).
[0024] In this embodiment, by acquiring data pairs containing the original image and query text, the input object for data processing can be clearly defined, providing a data foundation for subsequent joint visual and text processing.
[0025] S102: Identify the valid text regions in the original image and crop the original image based on the valid text regions to obtain the cropped image.
[0026] Among them, the effective text region refers to the image region in the original image that actually contains valuable information such as text and characters.
[0027] Cropped images refer to the partial images retained after removing invalid edges or background from the original image.
[0028] In this embodiment, by identifying the valid text regions in the original image and cropping the original image based on the valid text regions, a cropped image can be obtained. This can remove invalid background and interference regions in the image, increase the focus on regions containing core information, and reduce the amount of redundant data in subsequent processing from a spatial perspective.
[0029] S103: Based on the semantics of the query text and the cropped image, predict the corresponding target scaling ratio.
[0030] The semantics of the query text refers to the specific meaning, intent, or task orientation expressed by the query text.
[0031] The target scaling factor is a proportional value calculated based on the current image content and query task requirements to adjust the image size.
[0032] In this embodiment, by predicting the corresponding target scaling rate based on the semantics of the query text and the cropped image, the appropriate compression level for retaining the required information can be adaptively determined in combination with the specific query task requirements, avoiding information loss or insufficient compression caused by using a fixed scaling ratio.
[0033] S104: Adjust the cropped image based on the target scaling factor to obtain the scaled image.
[0034] The scaled image refers to the image obtained by enlarging or reducing the resolution of the image according to a determined target scaling factor.
[0035] In this embodiment, by adjusting the cropped image based on the target scaling rate, a scaled image is obtained, which can further reduce the data size of the image at the pixel level, thereby completing secondary compression.
[0036] S105: Process the scaled image and query text as input to the visual language model.
[0037] Among them, the visual language model refers to an artificial intelligence model that can simultaneously receive, understand, and process both visual (image) and natural language (text) modal data.
[0038] In this embodiment, a data pair containing the original image and query text is acquired; valid text regions in the original image are identified, and the original image is cropped based on the valid text regions to obtain a cropped image; based on the semantics of the query text and the cropped image, the corresponding target scaling ratio is predicted; the cropped image is adjusted based on the target scaling ratio to obtain a scaled image; the scaled image and the query text are then used as input to a visual language model for processing. This significantly reduces redundant visual data input to the visual language model while preserving the core semantic information of the image, thereby accelerating the inference speed of downstream models and reducing computational resource consumption.
[0039] Figure 2 This is a schematic flowchart of a visual information processing method proposed in another embodiment of this disclosure.
[0040] like Figure 2 As shown, the visual information processing method includes: S201: Retrieve the data pair containing the original image and the query text.
[0041] For a detailed description of S201, please refer to the above embodiments, which will not be repeated here.
[0042] S202: Perform text detection on the original image to determine the initial text boxes for each text line in the original image.
[0043] Text detection refers to the process of automatically locating text content in an image.
[0044] Among them, a text line refers to a sequence of text arranged horizontally or vertically in a line in an image.
[0045] The initial text box refers to the geometric bounding box that initially identifies each line of text and surrounds it, as determined by the text detection algorithm.
[0046] S203: Remove initial text boxes whose area is smaller than the area threshold or whose aspect ratio is not within the set range from the obtained multiple initial text boxes to obtain valid text boxes.
[0047] The area threshold is a pre-set reference value used to determine whether a text box is too small (e.g., has noise).
[0048] The aspect ratio refers to the ratio between the length and width of a text box.
[0049] The preset ratio range refers to a pre-determined range of aspect ratio values that conforms to the shape characteristics of normal text lines.
[0050] Among them, a valid text box refers to a bounding box that has been retained after being filtered by geometric conditions and is deemed to contain real text content.
[0051] S204: Based on valid text boxes, identify valid text regions in the original image.
[0052] Optionally, in some embodiments, when removing initial text boxes with areas smaller than an area threshold or aspect ratios outside a set range from the resulting multiple initial text boxes to obtain valid text boxes, the process can be as follows: Project the original image along both the vertical and horizontal directions to determine blank rows and columns not covered by valid text boxes; mark areas where consecutive blank rows are longer than a first threshold as vertical blank bands, and areas where consecutive blank columns are longer than a second threshold as horizontal blank bands; use the vertical and horizontal blank bands as dividing boundaries to divide the original image into multiple non-overlapping content blocks; determine the minimum bounding rectangle of the valid text boxes contained within each content block, and extend it outward by a preset pixel distance to obtain the valid text area of each content block. Thus, by locating the blank intervals of the document through row and column projection scanning and using these as boundaries to divide content blocks, redundant blank spaces between paragraphs and rows are eliminated to the greatest extent possible while preserving the original document layout structure, resulting in a more compact cropped image.
[0053] Among them, projection scanning can refer to a method of statistically analyzing the pixel features of an image (such as the coverage of an effective text box) row by row or column by column along a specific direction.
[0054] Blank lines refer to pixel lines that are not covered by any valid text boxes when scanning horizontally.
[0055] Blank columns refer to pixel columns that are not covered by any valid text boxes when scanning in the vertical direction.
[0056] Here, blank row length and blank column length refer to the number of consecutive blank rows or blank columns.
[0057] The first threshold is a preset reference value used to determine whether consecutive blank lines are sufficient to form a vertical dividing band.
[0058] The second threshold is a preset reference value used to determine whether consecutive blank columns are sufficient to form a horizontal dividing band.
[0059] Among them, the vertical blank band refers to the textless area in the image where the vertically extending continuous blank lines exceed the set standard.
[0060] Among them, the horizontal blank band refers to the textless area in the image where the horizontally extended continuous blank column exceeds the set standard.
[0061] In this context, the dividing boundary refers to the dividing line or region used to divide an image into different parts.
[0062] Among them, the content block refers to a local image region containing valid text information, which is divided by a blank band.
[0063] The minimum bounding rectangle is the smallest rectangle that can completely enclose a set of geometric shapes (in this case, valid text boxes within a content block).
[0064] The preset pixel distance refers to the pre-set safe margin value used to expand the rectangular boundary outward, preventing the text edges from being truncated.
[0065] S205: Merge all valid text regions into a target rectangle, and crop the original image according to the target rectangle to obtain a cropped image.
[0066] The target rectangle refers to a single rectangular boundary that completely covers all valid text regions after summarizing and enclosing their boundaries.
[0067] In other words, in this embodiment of the present disclosure, text detection can be performed on the original image to determine the initial text boxes for each text line in the original image; initial text boxes with an area smaller than an area threshold or an aspect ratio outside a set range are removed from the obtained initial text boxes to obtain valid text boxes; based on the valid text boxes, valid text regions in the original image are identified; all valid text regions are merged into target rectangles, and the original image is cropped according to the target rectangles to obtain a cropped image. Therefore, by combining text detection with geometric rule filtering, noise, decorative lines, and other non-real text interference in the image can be accurately removed, ensuring that the final cropped image region accurately covers the core text information and improving the accuracy of the first-stage visual information extraction.
[0068] S206: Extract the visual feature vector of the cropped image through the image branch of the scaling factor prediction model, and extract the semantic feature vector of the query text through the text branch of the scaling factor prediction model.
[0069] The scaling factor prediction model is a network model used to calculate and output the optimal scaling ratio of an image.
[0070] Among them, the image branch refers to the network branch in the prediction model that receives image data and extracts its features.
[0071] Among them, visual feature vectors refer to the image visual information (such as complexity, sharpness, etc.) extracted by the network and represented in the form of numerical vectors.
[0072] The text branch refers to the network branch in the prediction model that receives text data and extracts its features.
[0073] Among them, semantic feature vectors refer to the textual semantic information extracted through the network and represented in the form of numerical vectors.
[0074] Optionally, in some embodiments, the scaling factor prediction model is trained based on training data constructed in a self-supervised manner. The steps for constructing the training data include: acquiring training samples containing document images and natural language query text; scaling the document images proportionally according to a set ratio to generate a scaled image sequence; calling a visual language model to process each image in the scaled image sequence and the natural language query text respectively to obtain the corresponding text response sequence; calculating the semantic similarity of each text response in the text response sequence relative to the text response of the original-sized image; and selecting the lowest scaling ratio that satisfies a semantic similarity greater than or equal to a preset performance threshold as the ground truth scaling factor for the corresponding training sample. Thus, a self-supervised method is used to automatically generate the ground truth scaling factor, thereby acquiring large-scale training data without manual annotation, enabling the model to accurately learn the minimum tolerable resolution while ensuring task completion quality.
[0075] Training data refers to the dataset used as input to the model to adjust its parameters and enable it to make specific predictions.
[0076] Training samples can refer to a single data instance in the training data.
[0077] Among them, document images refer to images containing text, tables, and other document layout content.
[0078] Natural language query text refers to query instructions expressed in everyday human language.
[0079] Among them, proportional scaling can refer to the operation of changing the size of an image while keeping the original aspect ratio unchanged.
[0080] Among them, a scaled image sequence refers to a set of images obtained by scaling the same original image by different proportions.
[0081] The text response sequence refers to the series of corresponding text results output by the model in response to a set of scaled image sequences.
[0082] Text response refers to the text output by the visual language model after receiving input.
[0083] The original-size image text response refers to the text output for an unscaled image, which serves as a reference benchmark.
[0084] Semantic similarity refers to the degree to which two text fragments are close in meaning.
[0085] Among them, the preset performance threshold refers to a pre-set numerical indicator that measures whether the semantic similarity has reached an acceptable standard.
[0086] The minimum scaling ratio refers to the minimum scaling ratio that meets performance requirements.
[0087] The truth scaling factor refers to the optimal scaling value that balances performance and compression ratio, serving as the standard answer for training the model.
[0088] Optionally, in some embodiments, the scaling factor prediction model includes an image branch composed of a convolutional network, a text branch composed of a word embedding network, and a multilayer perceptron; wherein, the target scaling rate is obtained by constraining the activation function and linearly mapping it to a preset output range; the scaling factor prediction model uses mean absolute error as the loss function during training. Thus, low latency in prediction can be guaranteed based on a lightweight network structure, and the output can be ensured to conform to the actual physical meaning of scaling based on the activation function and linear mapping. Using mean absolute error as the loss function can improve the model's robustness to noise interference in self-supervised data.
[0089] Among them, word embedding network refers to a network structure that maps discrete words in a text into continuous vector representations.
[0090] Among them, the multilayer perceptron refers to a feedforward neural network composed of multiple layers of nodes, which is used for feature mapping and nonlinear transformation.
[0091] The preset output range refers to the upper and lower limits of the reasonable scaling factor (e.g., 0.2 to 1.0) set in advance.
[0092] Among them, mean absolute error refers to the mean absolute error. S207: Concatenate the visual feature vector and the semantic feature vector, and output the target scaling rate through fusion and regression processing.
[0093] The fusion and regression processing can refer to the process in this embodiment of the disclosure of comprehensively calculating features from different sources and mapping them to a continuous numerical value (i.e., scaling factor).
[0094] In other words, in this embodiment of the disclosure, the visual feature vector of the cropped image can be extracted through the image branch of the scaling factor prediction model, and the semantic feature vector of the query text can be extracted through the text branch of the scaling factor prediction model. The visual feature vector and the semantic feature vector are concatenated, and the target scaling rate is output through fusion and regression processing. Thus, by first extracting the visual features of the image and the semantic features of the text separately, and then fusing them, the predicted scaling rate can simultaneously take into account the image sharpness and other attributes as well as the task requirements, achieving task-aware dynamic compression.
[0095] S208: Adjust the cropped image based on the target scaling factor to obtain the scaled image.
[0096] S209: Process the scaled image and query text as input to the visual language model.
[0097] For a detailed description of S208 and S209, please refer to the above embodiments, and they will not be repeated here.
[0098] In this embodiment, text detection is performed on the original image to determine the initial text boxes for each text line. Initial text boxes with areas smaller than a threshold or aspect ratios outside a set range are removed to obtain valid text boxes. Based on these valid text boxes, valid text regions in the original image are identified. All valid text regions are merged into target rectangles, and the original image is cropped according to these target rectangles to obtain a cropped image. Thus, by combining text detection with geometric rule filtering, noise, decorative lines, and other non-realistic text interference in the image can be accurately removed, ensuring that the final cropped image region accurately covers the core text information and improving the accuracy of the first-stage visual information extraction. The visual feature vector of the cropped image is extracted using the image branch of the scaling factor prediction model, and the semantic feature vector of the query text is extracted using the text branch of the scaling factor prediction model. The visual and semantic feature vectors are concatenated, and the target scaling rate is output through fusion and regression processing. Therefore, by first extracting the visual features of the image and the semantic features of the text separately, and then fusing them, the predicted scaling rate can simultaneously take into account the image's sharpness and other attributes as well as the task requirements, thus achieving task-aware dynamic compression.
[0099] In summary, this disclosure places a pluggable adaptive visual information compression module before the input of the large visual language model. Through effective text region cropping guided by text detection results and adaptive resolution adjustment oriented towards task perception, redundant visual tokens are significantly reduced from the source while preserving the complete visual information of the document, achieving synergistic optimization of efficiency and accuracy. Figure 3 As shown, Figure 3This is a flowchart of the model inference process based on the addition of a visual information adaptive compression module proposed in this disclosure. The visual information adaptive compression module mainly includes two sequential processing stages: effective region cropping guided by text detection (visual information extraction module) and task-aware adaptive resolution adjustment (adaptive resolution adjustment module). During inference, the input text-image pair is first processed by the visual information extraction module, which automatically identifies and crops out compact images of all effective text regions. This cropped image is then fed into the adaptive resolution adjustment module, which dynamically predicts the optimal scaling rate based on the task semantics and completes image compression. Finally, the compressed image and the original query text are input into the visual language model to drive downstream task execution, thereby significantly reducing redundant visual tokens while preserving core semantics, improving system efficiency and response speed.
[0100] (a) Visual Information Extraction Module The purpose of this module is to remove large areas of invalid background (such as blank margins, shooting environment, etc.) from the original image, retaining only the text areas that carry valid information, while ensuring that the original layout structure of the document is not destroyed. The specific process is as follows: Figure 4 As shown, Figure 4 This is a schematic diagram of the visual information extraction module processing flow proposed in this disclosure, and its specific steps include: Step 1: Text Detection and Geometric Filtering. First, text detection is performed on the input document image to obtain a set of bounding rectangles for all text lines in the image. Then, this set is filtered according to preset geometric rules to remove detection boxes with too small an area (which may be noise) or abnormal aspect ratios (which may be decorative lines or page frames), thus obtaining a set of valid text boxes representing the real text content.
[0101] Step Two: Projection Analysis and Blank Area Identification. First, perform vertical projection analysis on the image: scan pixel by pixel (Y-axis) along the image's height. If the current pixel row is covered by any valid text box, mark it as a valid row (denoted as "1"); otherwise, mark it as a blank row (denoted as "0"). This generates a binary sequence representing the text distribution in the vertical direction of the image. In the generated vertical projection sequence, identify the length of consecutive blank rows. If the length exceeds a preset threshold, the interval is identified as a vertical blank band. Subsequently, using the same principle, perform horizontal projection analysis. If consecutive blank columns are identified that exceed a preset threshold, they are identified as horizontal blank bands.
[0102] Step 3: Segmentation of Valid Content Regions. Using the identified vertical and horizontal blank bands as boundaries, the original image is segmented into several non-overlapping "content blocks" in the vertical and horizontal directions. For each segmented content block, the minimum bounding rectangle of all valid text boxes within it is calculated. This rectangle's boundary is extended outward by several pixels as a safety margin, forming the candidate valid region of that content block.
[0103] Step 4: Merging Text Areas. Since the blank areas consist of complete rows and columns, all valid content areas are ultimately integrated into a single rectangular frame that completely covers the core information of the document, without omissions or fragmentation. This area eliminates redundant background whitespace around the edges and inside, while preserving the document's internal visual structure, achieving the first level of visual information purification.
[0104] (ii) Adaptive resolution adjustment module Based on the effective region image output by the visual information extraction module, this module, combined with the user's specific query task semantics, autonomously determines a minimum image resolution that meets the task requirements and performs secondary compression, thereby further optimizing the number of visual tokens at the pixel level. The specific process is as follows: Figure 5 As shown, Figure 5 This is a schematic diagram of the processing flow of the adaptive resolution adjustment module proposed in this disclosure, and its specific steps include: Step 1: Self-Supervised Truth Generation. To train the task-aware resolution prediction model, this application employs a self-supervised approach to construct the dataset. First, sample pairs consisting of document images and corresponding natural language queries are collected. For each sample, a sequence of proportionally scaled images is generated, ranging from the original size (1.0) to a preset minimum (e.g., 0.2). Each scaled image and query is processed by the target visual language large model to obtain a series of text responses. The semantic similarity of each scaled version of the response is calculated based on the response of the original-size image. A performance threshold (e.g., 0.95) is set, and the scaling ratio of the last segment in the sequence with a similarity not lower than this threshold is defined as the "truth scaling factor" of that sample, representing the minimum acceptable resolution for maintaining semantic understanding performance under the current query task.
[0105] Step 2: Lightweight Scaling Factor Prediction Model. This application designs a lightweight two-stream neural network for online prediction of scaling factors. The image branch is constructed using a lightweight convolutional network to extract visual feature vectors such as complexity and sharpness of the effective region image; the text branch is constructed using a lightweight word embedding network to encode the query text into semantic feature vectors. After concatenating the two feature vectors, they are fused and regressed through a shallow multilayer perceptron, and finally constrained by the Sigmoid function and linearly mapped to a preset output range (e.g., [0.2, 1.0]) to obtain the predicted adaptive scaling factor. The model training uses the mean absolute error (L1 Loss) as the loss function, which is insensitive to noise that may exist in the self-supervised ground truth, enabling the model to learn more robust compression decision rules, avoid overfitting, and improve generalization ability.
[0106] Figure 6 This is a schematic diagram of the structure of a visual information processing device according to an embodiment of the present disclosure.
[0107] like Figure 6 As shown, the visual information processing device 60 includes: The acquisition module 601 is used to acquire data pairs containing the original image and the query text; The recognition module 602 is used to recognize the valid text regions in the original image and crop the original image according to the valid text regions to obtain the cropped image; The prediction module 603 is used to predict the corresponding target scaling ratio based on the semantics of the query text and the cropped image; The adjustment module 604 is used to adjust the cropped image based on the target scaling ratio to obtain a scaled image; The processing module 605 is used to process the scaled image and the query text as input to the visual language model.
[0108] It should be noted that the foregoing explanation of the visual information processing method also applies to the visual information processing device of this embodiment, and will not be repeated here.
[0109] In this embodiment, a data pair containing the original image and query text is acquired; valid text regions in the original image are identified, and the original image is cropped based on the valid text regions to obtain a cropped image; based on the semantics of the query text and the cropped image, the corresponding target scaling ratio is predicted; the cropped image is adjusted based on the target scaling ratio to obtain a scaled image; the scaled image and the query text are then used as input to a visual language model for processing. This significantly reduces redundant visual data input to the visual language model while preserving the core semantic information of the image, thereby accelerating the inference speed of downstream models and reducing computational resource consumption.
[0110] According to embodiments of this application, this application also provides an electronic device and a readable storage medium.
[0111] Figure 7 This is a block diagram of an electronic device according to an embodiment of this application.
[0112] like Figure 7 As shown, the electronic device includes: The memory 701, the processor 702, and computer instructions stored in the memory 701 and executable on the processor 702.
[0113] When the processor 702 executes instructions, it implements the visual information processing method provided in the above embodiments.
[0114] Furthermore, electronic devices also include: Communication interface 703 is used for communication between memory 701 and processor 702.
[0115] Memory 701 is used to store computer instructions that can be executed on processor 702.
[0116] The memory 701 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.
[0117] The processor 702 is used to implement the visual information processing method of the above embodiments when executing a program.
[0118] If the memory 701, processor 702, and communication interface 703 are implemented independently, then the communication interface 703, memory 701, and processor 702 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0119] Optionally, in a specific implementation, if the memory 701, processor 702, and communication interface 703 are integrated on a single chip, then the memory 701, processor 702, and communication interface 703 can communicate with each other through an internal interface.
[0120] The processor 702 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.
[0121] This application also proposes a computer program product that implements the visual information processing method of the embodiments of this application when the instruction processor in the computer program product is executed.
[0122] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0123] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0124] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.
[0125] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.
[0126] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0127] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.
[0128] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
[0129] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.
Claims
1. A visual information processing method, characterized in that, include: Retrieve a data pair containing the original image and the query text; Identify valid text regions in the original image, and crop the original image based on the valid text regions to obtain a cropped image; Based on the semantics of the query text and the cropped image, predict the corresponding target scaling ratio; The cropped image is adjusted based on the target scaling ratio to obtain a scaled image; The scaled image and the query text are processed as input to a visual language model.
2. The method as described in claim 1, characterized in that, The process of identifying valid text regions in the original image and cropping the original image based on the valid text regions to obtain a cropped image includes: Text detection is performed on the original image to determine the initial text box for each text line in the original image; In the resulting set of initial text boxes, those with an area smaller than the area threshold or an aspect ratio outside the set range are removed to obtain valid text boxes. Based on the valid text boxes, identify the valid text regions in the original image; All the valid text regions are merged into a target rectangle, and the original image is cropped according to the target rectangle to obtain the cropped image.
3. The method as described in claim 2, characterized in that, The step of identifying valid text regions in the original image based on the valid text boxes includes: The original image is projected and scanned along the vertical and horizontal directions respectively to determine the blank rows and columns not covered by the effective text boxes; Regions with consecutive blank rows longer than the first threshold are marked as vertical blank bands, and regions with consecutive blank columns longer than the second threshold are marked as horizontal blank bands. Using the vertical and horizontal blank bands as dividing boundaries, the original image is divided into multiple non-overlapping content blocks; Determine the minimum bounding rectangle of the valid text boxes contained in each content block, and expand it outward by a preset pixel distance to obtain the valid text area of each content block.
4. The method as described in claim 1, characterized in that, The step of predicting the corresponding target scaling ratio based on the semantics of the query text and the cropped image includes: The visual feature vector of the cropped image is extracted by the image branch of the scaling factor prediction model, and the semantic feature vector of the query text is extracted by the text branch of the scaling factor prediction model. The visual feature vector and the semantic feature vector are concatenated, and the target scaling rate is output through fusion and regression processing.
5. The method as described in claim 4, characterized in that, The scaling factor prediction model is trained based on training data constructed in a self-supervised manner. The steps for constructing the training data include: Obtain training samples containing document images and natural language query text; The document image is scaled proportionally according to a set ratio to generate a scaled image sequence. The visual language model is invoked to process each image in the scaled image sequence and the natural language query text respectively, to obtain the corresponding text response sequence; Calculate the semantic similarity of each text response in the text response sequence relative to the text response of the original-sized image; Select the lowest scaling ratio that satisfies semantic similarity greater than or equal to a preset performance threshold as the ground value scaling factor for the corresponding training sample.
6. The method as described in claim 4, characterized in that, The scaling factor prediction model includes an image branch composed of a convolutional network, a text branch composed of a word embedding network, and a multilayer perceptron; wherein, The target scaling factor is obtained by constraining the activation function and linearly mapping it to a preset output range; The scaling factor prediction model uses mean absolute error as the loss function during training.
7. A visual information processing device, characterized in that, include: The acquisition module is used to acquire data pairs containing the original image and the query text; The recognition module is used to identify valid text regions in the original image and crop the original image according to the valid text regions to obtain a cropped image; The prediction module is used to predict the corresponding target scaling ratio based on the semantics of the query text and the cropped image; An adjustment module is used to adjust the cropped image based on the target scaling ratio to obtain a scaled image; The processing module is used to process the scaled image and the query text as input to the visual language model.
8. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.
9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, in, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.
10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-6.
