Non-uniform image data processing method and device based on visual language model

Abstract

This disclosure presents a method and apparatus for processing non-uniform image data based on a visual language model. One specific implementation of the method includes: performing text box localization processing on the image to be processed to obtain a set of text box coordinate information; obtaining a density matrix; performing grid clustering processing on the density matrix to obtain an initial set of region bounding boxes; performing density adaptive expansion processing on the initial set of region bounding boxes to obtain a set of target region bounding boxes; performing multi-scale regional extraction processing on the image to be processed to obtain a global thumbnail and a set of local sub-images; performing fusion inference processing on the global thumbnail and the set of local sub-images to obtain a target content information set; performing differential compression processing on the image to be processed to obtain archived image data; and saving the archived image data to a storage medium. This implementation can reduce the lexical consumption of visual language model inference, reduce computational resource consumption, and reduce the storage space occupied by images.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of computer technology, and more specifically to a method and apparatus for processing non-uniform image data based on a visual language model. Background Technology

[0002] Non-uniform image data can be large-format technical document images (e.g., A0-sized high-resolution engineering drawings) containing both dense information regions (e.g., text regions) and sparse information regions (e.g., white space regions or regions containing only sparse lines). A visual language model-based non-uniform image data processing method can be a method that uses a visual language model to perform content recognition on the non-uniform image data and stores the image and the corresponding content recognition results. For the aforementioned visual language model-based non-uniform image data processing method, the typical approach is as follows: First, the image is divided into multiple image blocks according to a preset segmentation size; then, each image block is input into the visual language model for content recognition; finally, the content recognition results output by the visual language model are merged and stored with the image.

[0003] However, in practice, it has been found that when using the above method to process image data, the following technical problem often occurs: The above method does not distinguish between dense and sparse information regions of non-uniform image data, causing sparse information regions to be input into the visual language model for content recognition as well. This leads to an increase in the consumption of lexical units and computational resources during the inference process of the visual language model. At the same time, sparse information regions are also stored at high resolution, which increases the storage space occupied when storing images.

[0004] The information disclosed in this background section is only intended to enhance the understanding of the background of the present disclosure concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of this disclosure propose a method and apparatus for processing non-uniform image data based on a visual language model to solve one or more of the technical problems mentioned in the background section above.

[0007] In a first aspect, some embodiments of this disclosure provide a non-uniform image data processing method based on a visual language model, comprising: performing text box localization processing on an acquired image to be processed to obtain a set of text box coordinate information; performing density quantization processing on the image to be processed according to the text box coordinate information set to obtain a density matrix; performing grid clustering processing on the density matrix to obtain an initial set of region bounding boxes; performing density adaptive expansion processing on the initial set of region bounding boxes according to the density matrix to obtain a set of target region bounding boxes; performing multi-scale regional extraction processing on the image to be processed according to the target region bounding box set to obtain a global thumbnail and a set of local sub-images; performing fusion inference processing on the global thumbnail and the set of local sub-images using a visual language model to obtain a set of target content information; performing differential compression processing on the image to be processed according to the target content information set, the target region bounding box set, and the set of local sub-images to obtain archived image data; and saving the archived image data to a storage medium.

[0008] Secondly, some embodiments of this disclosure provide a non-uniform image data processing apparatus based on a visual language model, comprising: an acquisition unit configured to perform text box localization processing on an acquired image to be processed to obtain a set of text box coordinate information; a quantization unit configured to perform density quantization processing on the image to be processed according to the text box coordinate information set to obtain a density matrix; a clustering unit configured to perform grid clustering processing on the density matrix to obtain an initial set of region bounding boxes; and an expansion unit configured to perform density adaptive expansion processing on the initial set of region bounding boxes according to the density matrix to obtain... The system includes: a target region bounding box set; an extraction unit configured to perform multi-scale regional extraction processing on the image to be processed based on the target region bounding box set to obtain a global thumbnail and a local sub-image set; an inference unit configured to perform fusion inference processing on the global thumbnail and the local sub-image set using a visual language model to obtain a target content information set; and a compression unit configured to perform differential compression processing on the image to be processed based on the target content information set, the target region bounding box set, and the local sub-image set to obtain archived image data, and to save the archived image data to a storage medium.

[0009] The above embodiments of this disclosure have the following beneficial effects: The non-uniform image data processing method based on visual language models in some embodiments of this disclosure can reduce lexical consumption during the visual language model inference process, reduce the computational resource consumption required for visual language model inference, and reduce the storage space occupied when storing images. Specifically, the reasons for the increased lexical consumption, increased computational resource consumption, and increased storage space occupied when storing images during visual language model inference are as follows: Existing methods do not distinguish between dense and sparse information regions of non-uniform image data, causing sparse information regions to be input into the visual language model for content recognition as well, resulting in increased lexical consumption and increased computational resource consumption during the visual language model inference process; at the same time, sparse information regions are also stored at high resolution, resulting in increased storage space occupied when storing images.

[0010] The non-uniform image data processing method based on a visual language model, according to some embodiments of this disclosure, firstly performs text box localization processing on the acquired image to be processed, obtaining a set of text box coordinate information. Here, the text box coordinate information set reflects the spatial distribution of text boxes in the image to be processed, providing a data basis for subsequent density quantization of the spatial distribution of information in the image to be processed. Next, based on the aforementioned text box coordinate information set, the image to be processed is subjected to density quantization processing, obtaining a density matrix. Here, discrete text box position coordinates are transformed into continuous density data reflecting the spatial distribution of information in the image to be processed, providing a data basis for subsequent spatial differentiation between dense and sparse information regions. Then, the density matrix is ​​subjected to grid clustering processing to obtain an initial set of region bounding boxes. Here, the location range of dense information regions in the image to be processed is initially identified, spatially distinguishing dense and sparse information regions, ensuring that subsequent multi-scale regionalization extraction processing only targets dense information regions, avoiding invalid content recognition by calling the visual language model for sparse information regions. Afterwards, based on the aforementioned density matrix, the initial set of region bounding boxes is subjected to density adaptive expansion processing to obtain a target region bounding box set. Here, the target region bounding box set, relative to the initial region bounding box set, undergoes density-adaptive expansion of its boundaries. This ensures that it fully encompasses structures such as tables and text blocks within dense information regions, avoiding the situation where dense information regions are segmented into multiple bounding boxes due to boundary cutoff, requiring repeated inference by the visual language model. Then, based on the target region bounding box set, the image to be processed undergoes multi-scale regional extraction processing, resulting in a global thumbnail and a set of local sub-images. Here, the local sub-images only carry the details of dense information regions in the image to be processed, while the global thumbnail carries the sparse information regions and the overall layout of the image to be processed at a lower resolution. This allows the visual language model to only identify the details of dense information regions, while perceiving sparse information regions as a whole through the global thumbnail, reducing the lexical consumption caused by the visual language model's detailed identification of sparse information regions. Finally, the visual language model performs fusion inference processing on the global thumbnail and the local sub-images to obtain the target content information set. Here, the visual language model combines the overall layout context carried by the global thumbnail with the dense information details carried by the local sub-image set to output the target content information set. This ensures that the input to the visual language model's inference process only includes the overall layout context and dense information details, avoiding the input of sparse information regions for detail recognition. This reduces lexical consumption during the visual language model's inference process and lowers the computational resource consumption required for inference. Finally, based on the aforementioned target content information set, the aforementioned target region bounding box set, and the aforementioned local sub-image set, the image to be processed undergoes differential compression processing to obtain archived image data, which is then saved to a storage medium.Here, dense information regions in the archived image data are preserved at higher resolution, while sparse information regions are preserved at lower resolution. This avoids storing sparse information regions at the same high resolution, thereby reducing the storage space occupied when storing the images to be processed. Therefore, this non-uniform image data processing method based on a visual language model can reduce lexical consumption during visual language model inference, lower the computational resource consumption required for visual language model inference, and reduce the storage space occupied when storing the images to be processed. Attached Figure Description

[0011] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0012] Figure 1 This is a flowchart of some embodiments of the non-uniform image data processing method based on a visual language model according to the present disclosure; Figure 2 This is a schematic diagram of the structure of some embodiments of the non-uniform image data processing apparatus based on a visual language model according to the present disclosure; Detailed Implementation Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0013] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0014] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0015] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0016] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0017] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0018] Figure 1 A flow 100 of some embodiments of a visual language model-based non-uniform image data processing method according to the present disclosure is shown. This visual language model-based non-uniform image data processing method includes the following steps: Step 101: Perform text box positioning processing on the acquired image to be processed to obtain a set of text box coordinate information.

[0019] In some embodiments, the execution entity may perform text box positioning processing on the acquired image to be processed to obtain a set of text box coordinate information. The image to be processed may be non-uniform image data. For example, the image to be processed may be an A0-sized architectural drawing image with a resolution of 35 megapixels. The text box coordinate information in the set of text box coordinate information can reflect the position and size of individual text boxes included in the image to be processed within the image.

[0020] In some optional implementations of certain embodiments, the above-described text box positioning processing of the acquired image to be processed to obtain a set of text box coordinate information may include the following steps: The first step is to convert the image to grayscale to obtain a grayscale image. This grayscale image can be a single-channel image where the pixel value of each pixel represents its brightness. In practice, the execution entity can use the `cvtColor` function in the OpenCV library to convert the image to be processed from an RGB three-channel image to a single-channel image, thus obtaining a grayscale image.

[0021] The second step is to denoise the aforementioned grayscale image to obtain a denoised grayscale image. This denoised grayscale image can be a grayscale image after suppressing image acquisition noise and scanning noise. In practice, the executing entity can use a preset denoising algorithm to denoise the aforementioned grayscale image to obtain the denoised grayscale image. The preset denoising algorithm can include, but is not limited to, at least one of the following: median filtering algorithm, bilateral filtering algorithm, and Gaussian filtering algorithm.

[0022] The third step involves performing contrast enhancement processing on the denoised grayscale image to obtain an enhanced image. This enhanced image can be an image where the grayscale difference between the text foreground and background is amplified. In practice, the execution entity can first use an anti-sharpening mask algorithm to sharpen the edges of the denoised grayscale image, obtaining a sharpened grayscale image. Then, a thresholding algorithm is used to separate the text foreground and background in the sharpened grayscale image, resulting in a binary image. This thresholding algorithm can include, but is not limited to, at least one of the following: Otsu's thresholding algorithm or an adaptive Gaussian thresholding algorithm. Finally, morphological dilation is performed on the binary image to thicken and connect thin and broken strokes, resulting in the enhanced image.

[0023] The fourth step is to perform block positioning processing on the enhanced image to obtain the text box coordinate information set.

[0024] Optionally, the above-mentioned block-based localization processing of the enhanced image to obtain the text box coordinate information set may include the following steps: The first step is to perform non-overlapping segmentation of the enhanced image according to a preset segmentation size, resulting in an image block set. The preset segmentation size can be a pre-defined size used to divide the enhanced image into image blocks. For example, the preset segmentation size could be 4096 pixels. The image blocks in the image block set can be image regions obtained after segmenting the enhanced image. In practice, the execution entity can utilize the slicing operation of the NumPy library to uniformly divide the enhanced image into multiple non-overlapping sub-regions along the horizontal and vertical directions according to the preset segmentation size, thus obtaining the image block set.

[0025] The second step involves locating text boxes for each image block in the aforementioned image block set, resulting in a local text box coordinate set. The local text box coordinates in this set represent the position and size of a single text box within the corresponding image block relative to the top-left corner of that image block. In practice, the execution entity can utilize an OCR engine to locate text boxes for each image block in the aforementioned image block set, obtaining the local text box coordinate set. For example, the OCR engine could be the PaddleOCR engine.

[0026] The third step involves performing global coordinate mapping on the local text box coordinate set based on the offset of each image block in the enhanced image. This yields a text box coordinate information set. The offset can be the x-coordinate and y-coordinate of the top-left corner of the image block in the enhanced image. In practice, the executing entity can add the top-left x-coordinate and top-left y-coordinate of each local text box coordinate in the local text box coordinate set to the horizontal and vertical offsets of the image block to which that local text box belongs, respectively, to obtain the global coordinates of the text box in the enhanced image. This global coordinate set is then used as the text box coordinate information set.

[0027] Step 102: Based on the text box coordinate information set, perform density quantization on the image to be processed to obtain the density matrix.

[0028] In some embodiments, the execution entity may perform density quantization processing on the image to be processed based on the text box coordinate information set to obtain a density matrix. The density matrix characterizes the spatial density of the text boxes included in the image to be processed.

[0029] In some optional implementations of certain embodiments, the process of performing density quantization on the image to be processed based on the text box coordinate information set to obtain a density matrix may include the following steps: The first step is to perform a gridding process on the image to be processed, resulting in a set of grid cells. Each grid cell in the set can be a non-overlapping rectangular region of the image to be processed. In practice, the executing entity can uniformly divide the image to be processed in both the horizontal and vertical directions according to a preset grid side length (e.g., 100 pixels) to obtain the grid cell set.

[0030] The second step involves performing text box density statistical processing on the aforementioned grid cell set based on the text box coordinate information set, resulting in an initial density matrix. Each element in this initial density matrix represents the number of text boxes within the corresponding grid cell. In practice, the executing entity can first determine the initial density value as the ratio of the number of text boxes contained in each grid cell to the area of ​​the corresponding grid cell, thus obtaining an initial density value set. Then, the initial density value set is arranged into a two-dimensional matrix according to the relative positions of each grid cell in the image to be processed, yielding the initial density matrix.

[0031] The third step is to normalize the initial density matrix to obtain the density matrix. In practice, the execution entity can use the maximum-minimum normalization method to normalize each element in the initial density matrix to obtain the density matrix.

[0032] Step 103: Perform grid clustering on the density matrix to obtain the initial set of region bounding boxes.

[0033] In some embodiments, the execution entity may perform grid clustering on the density matrix to obtain an initial set of region bounding boxes. The initial region bounding boxes in the initial set may be rectangular boxes reflecting the location and size of dense text box regions in the image to be processed. The initial region bounding boxes may include, but are not limited to, at least one of the following: the x-coordinate of the top-left corner of the bounding box, the y-coordinate of the top-left corner of the bounding box, the width of the bounding box, and the height of the bounding box.

[0034] In some optional implementations of certain embodiments, the above-described grid clustering process of the density matrix to obtain an initial set of region bounding boxes includes: The first step is to select grid cells with density values ​​greater than or equal to a preset density threshold from the density matrix above, and use these as the target grid cell set. The preset density threshold can be a pre-defined critical value used to determine whether a grid cell is a high-density grid cell. For example, the preset density threshold could be 0.1.

[0035] The second step involves performing density-connected clustering on the target grid cell set based on a preset neighborhood radius and a preset sample number threshold, resulting in a target grid cell cluster set. The preset neighborhood radius can be a pre-defined distance threshold used to determine whether two target grid cells belong to the same neighborhood. For example, the preset neighborhood radius could be 200 pixels. The preset sample number threshold can be a pre-defined lower limit on the number of target grid cells required to form a target grid cell cluster. For example, the preset sample number threshold could be 4. In practice, the execution entity can use the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm to perform density-connected clustering on the target grid cell set based on the preset neighborhood radius and preset sample number threshold, thus obtaining the target grid cell cluster set.

[0036] The third step is to extract the bounding rectangle of each target grid cell cluster in the above target grid cell cluster set to obtain the initial region bounding box set.

[0037] In practice, the aforementioned execution entity can use the boundingRect function in the OpenCV library to calculate the minimum bounding rectangle for the pixel range occupied by each target mesh cell cluster in the target mesh cell cluster set, thereby obtaining the initial region bounding box set.

[0038] Step 104: Based on the density matrix, perform density adaptive expansion processing on the initial region bounding box set to obtain the target region bounding box set.

[0039] In some embodiments, the execution entity may perform density adaptive expansion processing on the initial region bounding box set according to the density matrix to obtain a target region bounding box set. The target region bounding boxes in the target region bounding box set may be the location boxes corresponding to image regions containing complete dense information in the image to be processed.

[0040] In some optional implementations of certain embodiments, the above-mentioned density adaptive expansion processing of the initial region bounding box set based on the density matrix to obtain the target region bounding box set includes: The first step is to perform the following extension steps for each initial region bounding box in the above initial region bounding box set: Sub-step 1 involves extending the four boundaries of the initial region boundary frame outwards by a predetermined extension ratio, resulting in an extended region group. The extended region in this group can be a rectangular area formed by extending one boundary of the initial region boundary frame outwards. The extended region group includes four extended regions corresponding to the top, bottom, left, and right boundaries of the initial region boundary frame. The predetermined extension ratio can be a pre-set value used to determine the proportional relationship between the extension distance and the corresponding directional dimension of the initial region boundary frame. For example, the predetermined extension ratio can be 10%. In practice, the executing entity can extend the top and bottom boundaries of the initial region boundary frame outwards by multiplying the predetermined extension ratio by the height of the initial region boundary frame, respectively, to obtain extended regions corresponding to the top and bottom boundaries; and extend the left and right boundaries of the initial region boundary frame outwards by multiplying the predetermined extension ratio by the width of the initial region boundary frame, respectively, to obtain extended regions corresponding to the left and right boundaries.

[0041] Sub-step 2: Based on the aforementioned density matrix, perform density mean statistical processing on each extended region in the extended region group to obtain an extended region density mean group. The extended region density mean in the extended region density mean group can be a numerical value reflecting the density of text boxes within the corresponding extended region. In practice, the execution entity can perform the following steps on each extended region in the extended region group: First, determine the density values ​​corresponding to all grid cells whose grid center points fall within the extended region in the aforementioned density matrix as the extended region density value set. Second, determine the arithmetic mean of the extended region density value set as the extended region density mean.

[0042] Sub-step 3 involves performing shape classification processing on the initial region bounding box to obtain shape category labels. These shape category labels can be discrete labels reflecting the shape characteristics of the initial region bounding box. For example, the shape category labels may include, but are not limited to, at least one of the following: ultra-elongated label, elongated label, and square label. In practice, the executing entity can first determine the ratio of the larger to the smaller value of the width and height of the initial region bounding box as the shape feature ratio. Then, in response to the shape feature ratio being greater than or equal to a first preset threshold (e.g., 8), the shape category label is determined to be an ultra-elongated label; in response to the shape feature ratio being greater than or equal to a second preset threshold (e.g., 3) and less than the first preset threshold, the shape category label is determined to be an elongated label; and in response to the shape feature ratio being less than the second preset threshold, the shape category label is determined to be a square label.

[0043] Sub-step 4: Based on the shape category labels, determine the basic expansion ratios of the initial region bounding box in each direction, obtaining a basic expansion ratio group. The basic expansion ratios in this group can be values ​​reflecting the expansion scale of the initial region bounding box in the corresponding direction. This basic expansion ratio group can include four basic expansion ratios corresponding to the up, down, left, and right directions, respectively. In practice, the executing entity can match the shape category labels with a preset shape expansion ratio mapping table to obtain the basic expansion ratio group. This preset shape expansion ratio mapping table can be a pre-established lookup table used to query the basic expansion ratios corresponding to the up, down, left, and right directions based on the shape category labels.

[0044] Sub-step 5: Based on the aforementioned extended region density mean group, perform density scaling on the aforementioned basic expansion ratio group to obtain the target expansion ratio group. The target expansion ratio in the target expansion ratio group can be a value that combines the density of text box distribution within the corresponding direction of the extended region and reflects the expansion scale in the corresponding direction of the initial region boundary box. The target expansion ratio group can include four target expansion ratios corresponding to the upward, downward, leftward, and rightward directions, respectively. In practice, the executing entity can perform the following density scaling steps on the upward, downward, leftward, and rightward directions of the initial region boundary box: First, determine the density scaling coefficient for the current direction by the ratio of the extended region density mean group corresponding to the current direction to the aforementioned preset density threshold. Second, determine the initial expansion ratio for the current direction by multiplying the basic expansion ratio in the aforementioned basic expansion ratio group corresponding to the current direction by the aforementioned density scaling coefficient. Third, in response to the initial expansion ratio being greater than the preset expansion ratio upper limit (e.g., 20%), the preset expansion ratio upper limit is determined as the target expansion ratio in the current direction; in response to the initial expansion ratio being less than or equal to the preset expansion ratio upper limit, the initial expansion ratio is determined as the target expansion ratio in the current direction.

[0045] Sub-step 6: Based on the aforementioned target expansion ratio group, perform density convergence expansion processing on the initial region bounding box to obtain the expanded bounding box. The expanded bounding box can be a rectangular frame reflecting the position and size of the dense text box region and its surrounding extended content in the image to be processed. In practice, the executing entity can first perform the following density convergence expansion steps in the four directions (up, down, left, and right) of the initial region bounding box: First, multiply the target expansion ratio corresponding to the current direction in the aforementioned target expansion ratio group by the size of the initial region bounding box corresponding to the current direction, and determine the maximum expansion distance in the current direction. For the up and down directions, the size corresponding to the current direction is the height of the initial region bounding box; for the left and right directions, the size corresponding to the current direction is the width of the initial region bounding box. Second, divide the maximum expansion distance equally according to a preset number of steps (e.g., 10) to obtain the single-step step distance. The third step involves offsetting the boundary of the initial region bounding box corresponding to the current direction by one single-step distance along the outer direction to the preset number of single-step distances, thus obtaining a candidate boundary sequence. The fourth step involves determining the edge density value corresponding to each candidate boundary in the candidate boundary sequence by the arithmetic mean of the density values ​​of the grid cells within a strip with a width of one grid cell located outside the candidate boundary in the density matrix, thus obtaining an edge density value sequence. The fifth step involves determining the first candidate boundary in the candidate boundary sequence whose corresponding edge density value is less than a preset convergence density threshold (e.g., 50% of the preset density threshold) as the extended boundary in the current direction, in response to the presence of edge density values ​​less than the preset convergence density threshold in the edge density value sequence; and determining the last candidate boundary in the candidate boundary sequence as the extended boundary in the current direction, in response to all edge density values ​​in the edge density value sequence being greater than or equal to the preset convergence density threshold. Finally, the extended boundaries obtained in the top, bottom, left, and right directions are combined to obtain the extended bounding box.

[0046] The second step involves overlapping and merging the obtained expanded bounding box set to obtain the target region bounding box set. In practice, the execution entity can first combine the expanded bounding boxes in the expanded bounding box set in pairs to obtain an expanded bounding box pairing set; then, for each expanded bounding box pairing in the expanded bounding box pairing set, the following filtering steps are performed: First, the two expanded bounding boxes included in the expanded bounding box pairing are respectively determined as the first expanded bounding box and the second expanded bounding box. Second, in response to the existence of an intersecting rectangle between the first and second expanded bounding boxes, the ratio of the area of ​​the intersecting rectangle to the area of ​​the first expanded bounding box is determined as the first coverage degree, and the ratio of the area of ​​the intersecting rectangle to the area of ​​the second expanded bounding box is determined as the second coverage degree. Third, in response to the first coverage degree being greater than or equal to a preset merging threshold (e.g., 0.3) or the second coverage degree being greater than or equal to the preset merging threshold, the expanded bounding box pairing is determined as a pairing to be merged. Then, each expanded bounding box in the aforementioned expanded bounding box set is used as a node, and each pair to be merged in the aforementioned pairing set is used as an edge connecting the corresponding two expanded bounding box nodes, constructing an undirected association graph. Subsequently, the disjoint-set data structure algorithm is used to extract connected components from the aforementioned undirected association graph to obtain a set of expanded bounding box groups. Each expanded bounding box group in the aforementioned expanded bounding box group set can be at least one expanded bounding box corresponding to a connected component in the aforementioned undirected association graph. Finally, for each expanded bounding box group in the aforementioned expanded bounding box group set, the smallest bounding rectangle of all expanded bounding boxes contained in that expanded bounding box group is determined as the target region bounding box corresponding to that expanded bounding box group, thus obtaining the target region bounding box set.

[0047] Step 105: Based on the target region bounding box set, perform multi-scale regional extraction processing on the image to be processed to obtain a global thumbnail and a local sub-image set.

[0048] In some embodiments, the execution entity may perform multi-scale regional extraction processing on the image to be processed based on the target region bounding box set to obtain a global thumbnail and a set of local sub-images. The global thumbnail may be an image obtained by downsampling the image to be processed, reflecting the overall macroscopic layout of the image. The local sub-images in the set of local sub-images may be images extracted from the image to be processed according to the corresponding target region bounding box, reflecting the details of the region corresponding to the target region bounding box.

[0049] In addressing the technical problems mentioned above by employing technical solutions, the following technical problem arises in the application scenario: the image to be processed contains multiple dense information regions with significant differences in the density and area of ​​text boxes between these regions (e.g., a general description of architectural construction drawings containing both large, moderately dense design description text blocks and small, highly dense title tables). This often leads to the following second technical problem: the aforementioned method uses the same extraction resolution for all target region bounding boxes' corresponding local sub-images, resulting in large dense information regions being extracted at the original resolution. This increases the number of visual units carried by the local sub-image set, leading to increased unit consumption and computational resource consumption during the visual language model's inference process. Furthermore, the local sub-images corresponding to these large dense information regions are stored at the original resolution in the archived image data, increasing the storage space required for image storage. Considering the following requirements for this application scenario: differentiated allocation of resolution across multiple regions, joint trade-off between density and area, and cross-scale location traceability, we have decided to adopt the following solution: Optionally, the above-mentioned multi-scale regionalization extraction processing of the image to be processed based on the target region bounding box set to obtain a global thumbnail and a local sub-image set may include the following steps: The first step is to perform coordinate truncation verification on the aforementioned target region bounding box set to obtain a verified bounding box set. The verified bounding boxes in this set can be target region bounding boxes whose coordinate values ​​all lie within the boundary range of the image to be processed. In practice, the execution entity can perform the following verification steps on each target region bounding box in the set: First, using the `clip` function from the NumPy library, constrain the x-coordinates of the top-left and bottom-right corners of the target region bounding box according to a preset x-coordinate range, and constrain the y-coordinates of the top-left and bottom-right corners according to a preset y-coordinate range, to obtain a coordinate-constrained bounding box. The preset x-coordinate range can be a pre-defined parameter used to limit the upper and lower limits of the x-coordinate values. For example, the preset x-coordinate range could have a lower limit of 0 and an upper limit equal to the width of the image to be processed. The preset y-coordinate range can also be a pre-defined parameter used to limit the upper and lower limits of the y-coordinate values. For example, the preset y-coordinate range could have a lower limit of 0 and an upper limit equal to the height of the image to be processed. Then, using the `floor` and `ceil` functions from the NumPy library, the coordinates of the top-left corner of the bounding box constrained by the above coordinates are rounded down, and the coordinates of its bottom-right corner are rounded up, resulting in the rounded bounding box. Finally, in response to the determination that the width and height of the rounded bounding box are both greater than zero, the rounded bounding box is identified as the validated bounding box.

[0050] The second step involves performing an extensional cropping process on the image to be processed based on the aforementioned set of verified bounding boxes, resulting in an initial set of local sub-images. The initial local sub-images in this set can be image regions within the image to be processed, defined by the corresponding verified bounding box and its surrounding extension margin. This extension margin can be a pre-defined pixel distance used to extend outwards from the verified bounding box to avoid the corresponding region boundary being too close to the text or table lines. For example, the extension margin can be half the side length of the preset grid. In practice, the executing entity can perform the following extensional cropping steps on each verified bounding box in the set: First, offset the upper left corner coordinates of the verified bounding box to the left and upwards by the distance corresponding to the extension margin, and offset the lower right corner coordinates to the right and downwards by the distance corresponding to the extension margin, respectively, to obtain the extended bounding box. Then, truncate the horizontal and vertical coordinates of the extended bounding box to the pixel range of the image to be processed, respectively, to obtain the corresponding cropping bounding box. Finally, using the OpenCV library, the corresponding image regions are extracted from the image to be processed according to the cropping bounding box described above, and used as the corresponding initial local sub-images.

[0051] The third step involves performing density-area weighted fusion processing on the aforementioned set of verified bounding boxes and the aforementioned density matrix to obtain a set of fusion weight values. The fusion weight values ​​in this set can reflect the area of ​​the corresponding verified bounding box and the density of text boxes within that area. In practice, the executing entity can perform the following weighted fusion steps on each verified bounding box in the set: First, the ratio of the area of ​​the image region corresponding to the verified bounding box to the total area of ​​the image to be processed is determined as the area component of the verified bounding box. Then, the arithmetic mean of the density values ​​corresponding to the grid cells whose grid center points are located within the region corresponding to the verified bounding box in the aforementioned density matrix is ​​determined as the density component of the verified bounding box. Finally, the sum of the product of the area component and a preset area weight coefficient, and the product of the density component and a preset density weight coefficient, is determined as the fusion weight value of the verified bounding box. The preset area weight coefficient can be a pre-set coefficient used to weight the area component. The preset density weight coefficient can also be a pre-set coefficient used to weight the density component. For example, the preset area weighting coefficient can be 0.3, and the preset density weighting coefficient can be 0.7.

[0052] The fourth step involves performing a linked sorting process on the aforementioned fusion weight value set, the aforementioned verified bounding box set, and the aforementioned initial local sub-image set to obtain a fusion weight value sequence, a verified bounding box sequence, and an initial local sub-image sequence. In practice, the executing entity can first arrange the fusion weight values ​​in the aforementioned fusion weight value set in descending order to obtain a fusion weight value sequence. Then, it can synchronously arrange the verified bounding boxes in the aforementioned verified bounding box set according to the position of their corresponding fusion weight values ​​in the aforementioned fusion weight value sequence to obtain a verified bounding box sequence; and synchronously arrange the initial local sub-images in the aforementioned initial local sub-image set according to the position of their corresponding fusion weight values ​​in the aforementioned fusion weight value sequence to obtain an initial local sub-image sequence.

[0053] Fifth, based on the aforementioned fusion weight value sequence, the initial local sub-image sequence is subjected to hierarchical downsampling processing to obtain a downsampled sub-image sequence. The downsampled sub-images in the downsampled sub-image sequence can be images reflecting the same image region as the initial local sub-images in the same position in the initial local sub-image sequence, but with different image sizes. In practice, the execution entity can perform the following hierarchical downsampling steps on each initial local sub-image in the initial local sub-image sequence: First, the fusion weight values ​​in the same position in the aforementioned fusion weight value sequence are matched with a preset weight level mapping table to obtain the downsampling ratio corresponding to that initial local sub-image. The preset weight level mapping table can be a pre-defined lookup table used to query the downsampling ratio based on the value range of the fusion weight values. Then, using the resize function in the OpenCV library, the initial local sub-image is downsampled proportionally according to the downsampling ratio corresponding to that initial local sub-image to obtain the corresponding downsampled sub-image.

[0054] Step 6: Perform proportional downsampling on the image to be processed to obtain an initial global thumbnail and scaling information. The initial global thumbnail can be an image obtained by proportionally downsampling the image to be processed according to a preset thumbnail side length. The scaling information reflects the size ratio between the initial global thumbnail and the image to be processed. The preset thumbnail side length can be a pre-defined size parameter used to constrain the number of pixels on the longer side of the initial global thumbnail. For example, the preset thumbnail side length can be 512 pixels. In practice, the execution entity can first determine the scaling information as the ratio of the preset thumbnail side length to the number of pixels on the longer side of the image to be processed. Then, using the `resize` function in the OpenCV library, proportional downsampling is performed on the image to be processed according to the scaling information to obtain the initial global thumbnail.

[0055] Step 7: Perform color-coded cross-labeling on the above-mentioned verified bounding box sequence, the above-mentioned initial global thumbnail, the above-mentioned scaling information, and the above-mentioned downsampled sub-image sequence to obtain a global thumbnail and a local sub-image set. In practice, the above-mentioned execution entity can first assign a mutually distinguishable color identifier to each verified bounding box in the above-mentioned verified bounding box sequence to obtain a color identifier sequence. The color identifier can be a color used to establish a visual association between the above-mentioned initial global thumbnail and the corresponding downsampled sub-image. Then, for each verified bounding box in the above-mentioned verified bounding box sequence, map its vertex coordinates to the coordinate system of the above-mentioned initial global thumbnail according to the above-mentioned scaling information to obtain the corresponding thumbnail annotation box. The thumbnail annotation box can be the position box of the image region corresponding to the above-mentioned verified bounding box on the above-mentioned initial global thumbnail. Next, using the rectangle function in the OpenCV library, draw each thumbnail annotation box on the above-mentioned initial global thumbnail according to the color identifiers with the same position in the above-mentioned color identifier sequence to obtain the global thumbnail. Finally, for each downsampled sub-image in the above downsampled sub-image sequence, the rectangle function in the OpenCV library is used to draw a color border along the outer edge according to the color identifiers in the above color identifier sequence, so as to obtain the corresponding sub-image with color borders, and the obtained sub-images with color borders are determined as a local sub-image set.

[0056] The above technical solution and its related content, as an inventive point of this disclosure, combined with "Step 107" below, solves the second technical problem: "Increased lexical consumption, increased computational resource consumption, and increased storage space occupied when storing images in the visual language model during inference." The factors leading to increased lexical consumption, increased computational resource consumption, and increased storage space occupied when storing images in the visual language model during inference are often as follows: The above method uses the same extraction resolution for all local sub-images corresponding to the bounding boxes of the target regions, resulting in large areas of dense information regions being extracted at the original resolution to obtain corresponding local sub-images, leading to an increase in the number of visual lexical units carried by the local sub-image set; simultaneously, the local sub-images corresponding to the large areas of dense information regions are stored in the archived image data at the original resolution, leading to an increase in the storage space occupied when storing images. If the above factors are solved, the effects of reducing lexical consumption, reducing computational resource consumption, and reducing the storage space occupied when storing images in the visual language model can be achieved. To achieve this effect, this disclosure first performs coordinate truncation verification processing on the above-mentioned target region bounding box set to obtain a verified bounding box set. Secondly, based on the verified bounding box set, the image to be processed is subjected to extensional cropping to obtain an initial local sub-image set. Here, the verified bounding box set and the initial local sub-image set provide the basic data for subsequent differential resolution allocation of each local sub-image based on density and area. Next, the verified bounding box set and the density matrix are subjected to density-area weighted fusion processing to obtain a fusion weight value set. Then, the fusion weight value set, the verified bounding box set, and the initial local sub-image set are subjected to linked sorting processing to obtain a fusion weight value sequence, a verified bounding box sequence, and an initial local sub-image sequence. Here, the fusion weight value set jointly represents the density and area of ​​each dense information region as a single comparable quantity, enabling subsequent hierarchical downsampling processing to distinguish between small-area high-density regions and large-area medium-density regions. Then, based on the fusion weight value sequence, the initial local sub-image sequence is subjected to hierarchical downsampling processing to obtain a downsampled sub-image sequence. Here, in the downsampled sub-image sequence, the resolution of local sub-images with low fusion weight values ​​is compressed, reducing the number of visual terms carried by the local sub-image set compared to the method extracted at the original resolution. Then, the images to be processed are downsampled proportionally to obtain an initial global thumbnail and scaling information. Finally, color-coded cross-labeling is performed on the above-mentioned validated bounding box sequence, the above-mentioned initial global thumbnail, the above-mentioned scaling information, and the above-mentioned downsampled sub-image sequence to obtain a global thumbnail and a local sub-image set.Here, a cross-scale positional correspondence is established between the global thumbnail and the downsampled sub-image sequence through color-coded cross-labeling. This allows the visual language model to accurately locate the position of each local sub-image within the image to be processed, even when local sub-images are differentially downsampled. Therefore, the above technical solution and its related content can reduce the lexical consumption of the visual language model during inference, reduce computational resource consumption, and reduce the storage space occupied when storing images.

[0057] Step 106: Using a visual language model, perform fusion reasoning processing on the global thumbnail and local sub-image sets to obtain the target content information set.

[0058] In some embodiments, the aforementioned execution entity can utilize a visual language model to perform fusion inference processing on the aforementioned global thumbnail and the aforementioned local sub-image set to obtain a target content information set. The aforementioned visual language model (VLM) can be a model that is accessed via a preset invocation method, fuses and infers input visual information and text data, and outputs structured information. The aforementioned preset invocation method can include, but is not limited to, at least one of the following: invoking a visual language model deployed on a remote server via a preset application programming interface; running it locally after loading a preset visual language model weight file; or accessing a server with the aforementioned visual language model deployed via a preset network interface. For example, the aforementioned visual language model can be a Qwen2-VL model, an InternVL2 model, or an LLaVA-1.6 model. The target content information in the aforementioned target content information set can be structured text reflecting the text content and semantic structure within the bounding box region of the corresponding target region in the aforementioned image to be processed. For example, the aforementioned target content information can include, but is not limited to, at least one of the following: field name, field value, and field source identifier.

[0059] In the process of adopting technical solutions to solve the technical problems of the above-mentioned background technology, for the application scenario: the scenario in which there are both dense and sparse text box parts in the target region bounding box corresponding to the image to be processed (for example, a large-format technical document image that simultaneously contains an identifier region for carrying document metadata and a list region for carrying item detail data, and both the identifier region and the list region simultaneously contain both dense and sparse text box parts), the following technical problem three often occurs: Since the above-mentioned local sub-image is extracted as a whole according to the target region bounding box corresponding to its corresponding region, the visual feature vectors corresponding to the sparse text box parts and the visual feature vectors corresponding to the dense text box parts in the target region bounding box are input into the visual language model for inference. This causes the number of visual feature vectors processed by the visual language model in a single fusion inference to be greater than the number of visual feature vectors that actually carry effective information. Consequently, the number of inference tokens consumed by the visual language model in a single fusion inference is greater than the number of inference tokens actually required, resulting in increased token consumption and increased computational resource consumption during the inference process of the visual language model. Considering the following requirements for this application scenario: visual feature vectors must be density-separable and the number of visual feature vectors must be compressible, we have decided to adopt the following solution: In some optional implementations of certain embodiments, the above-described method of using a visual language model to perform fusion reasoning processing on the global thumbnail and the local sub-image set to obtain a target content information set may include the following steps: The first step involves visual feature extraction processing of the global thumbnail and the local sub-image set to obtain a global visual feature vector sequence and a local visual feature vector set. The global visual feature vectors in the global visual feature vector sequence represent the visual features of the image blocks obtained by segmenting the global thumbnail. The local visual feature vector set in the local visual feature vector set can be a vector group composed of the visual feature vectors of each image block corresponding to a local sub-image. In practice, the execution entity can use a pre-defined visual encoder to extract visual features from each local sub-image in the global thumbnail and the local sub-image set, respectively, to obtain the global visual feature vector sequence and the local visual feature vector set. The pre-defined visual encoder can include, but is not limited to, at least one of the following: a visual encoder for the ViT (Vision Transformer) model or a visual encoder for the CLIP (Contrastive Language-Image Pre-training) model.

[0060] The second step involves spatially linking and sorting the target region bounding box set and the local visual feature vector set to obtain a target region bounding box sequence and a local visual feature vector set sequence. The target region bounding box sequence can be an ordered set of target region bounding boxes arranged from top to bottom and left to right according to their corresponding spatial positions. Local visual feature vector sets with the same position in the local visual feature vector set sequence correspond to the target region bounding boxes in the target region bounding box sequence. In practice, the execution entity can first arrange the target region bounding boxes in the target region bounding box set in ascending order of their top-left ordinate, and if the top-left ordinates are the same, arrange them in ascending order of their top-left abscissa, to obtain the target region bounding box sequence. Then, the local visual feature vector sets in the local visual feature vector set sequence are simultaneously arranged according to the position of their corresponding target region bounding boxes in the target region bounding box sequence to obtain the local visual feature vector set sequence.

[0061] The third step involves performing global position encoding on the local visual feature vector group sequence based on the target region bounding box sequence, resulting in an encoded visual feature vector group sequence. The encoded visual feature vector group in this sequence can be a vector group formed by superimposing the corresponding global position encoding vectors onto each local visual feature vector. The global position encoding vector represents the global spatial position of the corresponding local visual feature vector in the image to be processed. In practice, the executing entity can perform the following encoding steps on each local visual feature vector group in the sequence: First, based on the top-left corner coordinates of the target region bounding boxes with the same position in the target region bounding box sequence, and the relative positions of the image blocks corresponding to each local visual feature vector in the local visual feature vector group within the corresponding local sub-images, the global coordinates of the image blocks corresponding to each local visual feature vector in the local visual feature vector group in the image to be processed are determined. Then, a sinusoidal position encoding method is used to convert the global coordinates into a global position encoding vector with the same dimension as the local visual feature vector. Finally, each local visual feature vector in the local visual feature vector group is added element-wise to the corresponding global position encoding vector to obtain the corresponding encoded visual feature vector.

[0062] Fourth, based on the density matrix and the target region bounding box sequence, the encoded visual feature vector group sequence is subjected to density sparsification to obtain the target visual feature vector group sequence. The target visual feature vector group in the target visual feature vector group sequence can be a vector group obtained by retaining a portion of the encoded visual feature vectors according to a density ratio. The number of vectors in the target visual feature vector group is less than or equal to the number of vectors in the corresponding encoded visual feature vector group. In practice, the execution entity can perform the following sparsification steps on each encoded visual feature vector group in the encoded visual feature vector group sequence: First, the arithmetic mean of the density values ​​corresponding to the grid cells within the target region bounding box region where the grid center point in the density matrix is ​​located at the same position in the target region bounding box sequence is determined as the region density value corresponding to that encoded visual feature vector group. Then, the region density value is determined as the retention ratio corresponding to that encoded visual feature vector group. Finally, the product of the retention ratio and the number of encoded visual feature vectors in the encoded visual feature vector group is rounded down to determine the retention quantity corresponding to the encoded visual feature vector group. Then, the encoded visual feature vectors in the encoded visual feature vector group are sampled at equal intervals according to the retention quantity to obtain the corresponding target visual feature vector group.

[0063] The fifth step involves generating location description information from the aforementioned target region bounding box sequence to obtain a location description information sequence. This location description information can be natural language text describing the coordinates and dimensions of the corresponding target region bounding box within the image to be processed. For example, the location description information could be "its top-left corner coordinates in the original image are (5000, 3000), width is 800 pixels, and height is 600 pixels." In practice, the executing entity can fill the corresponding placeholders in a preset location description template with the top-left x-coordinate, top-left y-coordinate, width, and height of each target region bounding box in the target region bounding box sequence to obtain the corresponding location description information. This preset location description template can be a pre-defined location description prompt text containing coordinate and size placeholders.

[0064] Step 6: Process the above location description information sequence and preset task template to generate text guidance information. This text guidance information can be a prompt message used to guide the visual language model in performing fusion inference on the concatenated visual feature vector sequence. The preset task template can be a pre-defined prompt text containing location description placeholders and task instruction text. In practice, the executing entity can first concatenate the various location description information in the above location description information sequence according to their order in the sequence to obtain concatenated location description text. Then, replace the location description placeholders in the preset task template with the concatenated location description text to obtain the text guidance information.

[0065] Step 7: Perform visual feature concatenation processing on the aforementioned global visual feature vector sequence and the aforementioned target visual feature vector group sequence to obtain a concatenated visual feature vector sequence. This concatenated visual feature vector sequence can be constructed by sequentially concatenating the aforementioned global visual feature vector sequence as a prefix and the aforementioned target visual feature vector group sequence as a subsequent part. In practice, the executing entity can first arrange each global visual feature vector in the aforementioned global visual feature vector sequence according to its position within the sequence. Then, it can arrange each target visual feature vector group in the aforementioned target visual feature vector group sequence according to its position within the sequence, and each target visual feature vector within each target visual feature vector group according to its position within that target visual feature vector group, sequentially after the aforementioned global visual feature vector sequence to obtain the concatenated visual feature vector sequence.

[0066] Step 8: Input the above-mentioned spliced ​​visual feature vector sequence and the above-mentioned text guidance information into the above-mentioned visual language model to obtain the target content information set.

[0067] The above technical solution and its related content, as an inventive point of this disclosure, combined with step "107" below, solves technical problem three: "increased lexical consumption and increased computational resource consumption during the inference process of the visual language model." The factors leading to increased lexical consumption and increased computational resource consumption during the inference process of the visual language model are often as follows: the aforementioned local sub-images are extracted as a whole according to the region corresponding to the bounding box of the target region. This results in the visual feature vectors corresponding to the sparse parts of the text boxes within the bounding box of the target region and the visual feature vectors corresponding to the dense parts of the text boxes being input into the visual language model for inference. This causes the number of visual feature vectors processed by the visual language model in a single fusion inference to be greater than the number of visual feature vectors that actually carry effective information. If the above factors are solved, the effect of reducing lexical consumption and computational resource consumption during the inference process of the visual language model can be achieved. To achieve this effect, this disclosure first performs visual feature extraction processing on the aforementioned global thumbnail and the aforementioned local sub-image set to obtain a global visual feature vector sequence and a local visual feature vector set. Secondly, the target region bounding box set and the local visual feature vector set are spatially linked and sorted to obtain the target region bounding box sequence and the local visual feature vector set sequence. Next, based on the target region bounding box sequence, the local visual feature vector set sequence is globally encoded to obtain the encoded visual feature vector set sequence. Here, the encoded visual feature vector set sequence corresponds one-to-one with the target region bounding box sequence in spatial location, and each encoded visual feature vector carries its global spatial location in the image to be processed, providing a basis for accurately distinguishing the visual feature vectors corresponding to dense and sparse parts of text boxes when filtering visual feature vectors by density. Then, based on the density matrix and the target region bounding box sequence, the encoded visual feature vector set sequence is density-sparsed to obtain the target visual feature vector set sequence. Here, the number of vectors in the target visual feature vector group is determined according to the density ratio of the corresponding target region bounding box. This significantly reduces the number of visual feature vectors corresponding to sparse parts of the text box after equal-interval sampling, while retaining the visual feature vectors corresponding to dense parts of the text box. This ensures that the number of visual feature vectors processed by the visual language model in a single fusion inference step matches the number of visual feature vectors actually carrying effective information, thereby reducing the lexical consumption and computational resource consumption of the visual language model during inference. Then, the target region bounding box sequence is processed to generate location description information, resulting in a location description information sequence. Subsequently, the location description information sequence and the preset task template are processed to generate text guidance information, resulting in text guidance information. Finally, the global visual feature vector sequence and the target visual feature vector group sequence are concatenated to obtain a concatenated visual feature vector sequence.Finally, the concatenated visual feature vector sequence and the textual guidance information are input into the visual language model to obtain the target content information set. Here, the textual guidance information supplements the visual language model with the positional information of the bounding boxes of each target region, enabling the visual language model to accurately locate the spatial position corresponding to each visual feature vector even after the visual feature vector density is sparsified. Therefore, the above technical solution and its related content can reduce the lexical consumption of the visual language model during the inference process and reduce the consumption of computational resources.

[0068] Step 107: Based on the target content information set, the target region bounding box set, and the local sub-image set, perform differential compression processing on the image to be processed to obtain archived image data, and save the above archived image data to the storage medium.

[0069] In some embodiments, the executing entity may perform differential compression processing on the image to be processed based on the target content information set, the target region bounding box set, and the local sub-image set to obtain archived image data, and save the archived image data to a storage medium. The archived image data may be composite data reflecting the foreground and background regions in the image to be processed, along with corresponding semantic information. The storage medium may be a hardware carrier for storing the archived image data. For example, the storage medium may include, but is not limited to, at least one of the following: a solid-state drive, a hard disk drive, or an object storage server.

[0070] In addressing the technical problems mentioned above by employing technical solutions, the following technical problem arises in the application scenario: where the target region bounding box contains multiple types of content with significantly different compression response characteristics (e.g., the target region bounding box in a large-format technical document image that simultaneously contains text and table lines). This often leads to the following fourth technical problem: because the foreground differential compression processing applies the same set of adjusted compression mapping parameters to each target region bounding box without further subdividing the content type differences within that region, the compressed data volume of local sub-image sets exceeds the actual required compressed data volume, resulting in increased storage space occupied by archived image data on the storage medium. Considering the following requirements for this application scenario: separability of content types within the region and adaptability of compression parameters to content types, we have decided to adopt the following solution: In some optional implementations of certain embodiments, the differential compression processing of the image to be processed based on the target content information set, the target region bounding box set, and the local sub-image set to obtain archived image data may include the following steps: The first step involves semantically encapsulating the target content information set, the target region bounding box set, the local sub-image set, and the global thumbnail to obtain a structured semantic information set. The structured semantic information in this set can be structured data reflecting the mapping relationship between the position, visual appearance, and semantic content of a single foreground region in the image to be processed. This structured semantic information may include, but is not limited to, at least one of the following: a unique region identifier, a target region bounding box, target content information, a local sub-image reference path, and a global thumbnail reference path. In practice, the executing entity can first assign a unique region identifier to each target region bounding box in the target region bounding box set, obtaining a unique region identifier set. Then, for each target region bounding box in the target region bounding box set, the unique region identifier, the target region bounding box itself, the target content information, the local sub-image reference path, and the global thumbnail reference path corresponding to the target region bounding box are written into the corresponding fields of a preset JSON template to obtain the structured semantic information corresponding to that target region bounding box. Finally, the obtained structured semantic information is summarized into a structured semantic information set.

[0071] The second step involves aggregating the aforementioned structured semantic information set and the aforementioned target region bounding box set to obtain an aggregated bounding box information set. The aggregated bounding box information in this set can reflect the overall attributes of several target region bounding boxes under the same semantic category. This aggregated bounding box information may include, but is not limited to, at least one of the following: semantic category labels, target region bounding box groups, the average density of the target region bounding box groups, and the average resolution of the sub-images corresponding to the target region bounding box groups. The semantic category labels may include, but are not limited to, at least one of the following: image label, explanatory text label, table label, and other labels. In practice, the executing entity can first match the target content information included in each structured semantic information set with a preset semantic category keyword dictionary to obtain a matching result. The semantic category label corresponding to the matching result is then determined as the semantic category label of the target region bounding box corresponding to that structured semantic information. The preset semantic category keyword dictionary can be a pre-defined two-dimensional table recording the keyword set under each semantic category label. For example, keywords under "Illustration Labels" can include "Project Name," "Drawing Number," "Design Unit," and "Scale," while keywords under "Explanatory Text Labels" can include "Project Overview," "Design Basis," and "Material Requirements." Then, the target region bounding box set is grouped according to these semantic category labels to obtain target region bounding box sets. Each target region bounding box set corresponds to a semantic category label. Next, for each target region bounding box set, the arithmetic mean of the density values ​​of the grid cells whose grid center points fall within the bounding boxes of that target region bounding box set is determined as the density mean of that target region bounding box set; and the arithmetic mean of the total number of pixels in the local sub-images corresponding to each target region bounding box in that target region bounding box set is determined as the sub-image resolution mean of that target region bounding box set. Finally, for each target region bounding box group, the semantic category label corresponding to the target region bounding box group, the target region bounding box group, the mean density value corresponding to the target region bounding box group, and the mean sub-image resolution value are determined as bounding box aggregation information, thus obtaining the bounding box aggregation information set.

[0072] The third step involves performing parameter mapping processing on the aforementioned bounding box aggregation information set to obtain an initial compression mapping parameter set. The initial compression mapping parameters in this set can be a group of values ​​used to configure the compression process. These initial compression mapping parameters may include, but are not limited to, at least one of the following: compression quality value, chromaticity subsampling factor, and downsampling ratio. In practice, the executing entity can match each bounding box aggregation information in the aforementioned bounding box aggregation information set with a preset parameter mapping table to obtain the corresponding initial compression mapping parameters, and then define the obtained initial compression mapping parameters as the initial compression mapping parameter set. The preset parameter mapping table can be a pre-defined lookup table used to query the corresponding initial compression mapping parameters based on the value range to which the bounding box aggregation information belongs.

[0073] The fourth step involves continuously inversely adjusting the bounding box aggregation information set and the initial compression mapping parameter set to obtain the adjusted compression mapping parameter set. The adjusted compression mapping parameters in this set can be a group of compression configuration values ​​that match the image content of the corresponding local sub-image. In practice, the execution entity can perform the following continuous inverse adjustment steps on each target region bounding box in the target region bounding box set: First, according to the initial compression mapping parameters corresponding to the target region bounding box in the initial compression mapping parameter set, perform trial compression processing on the local sub-images corresponding to the target region bounding box in the local sub-image set to obtain trial compressed sub-images. Second, using the PSNR function in the OpenCV library, perform peak signal-to-noise ratio quantization on the trial compressed sub-images and the corresponding local sub-images to obtain peak signal-to-noise ratio values. In the third step of reverse adjustment, in response to the peak signal-to-noise ratio (PSNR) being less than a preset lower limit (e.g., 30), the compression quality value in the corresponding initial compression mapping parameters is upgraded by a preset adjustment step size (e.g., 5) to obtain the reverse-adjusted initial compression mapping parameters; in response to the peak PSNR being greater than a preset upper limit (e.g., 40), the compression quality value in the corresponding initial compression mapping parameters is downgraded by the preset adjustment step size to obtain the reverse-adjusted initial compression mapping parameters; in response to the peak PSNR being between the preset lower limit and the preset upper limit, the corresponding initial compression mapping parameters are determined as the adjusted compression mapping parameters corresponding to the bounding box of the target region. In the fourth step of the reverse adjustment, in response to the initial compression mapping parameters obtained in the third step of the reverse adjustment and the current iteration number being less than the preset maximum iteration number (e.g., 5 times), the initial compression mapping parameters obtained in the reverse adjustment are used as the corresponding initial compression mapping parameters for the new round, and the first to third steps of the reverse adjustment are executed again; in response to the initial compression mapping parameters obtained in the third step of the reverse adjustment and the current iteration number being equal to the preset maximum iteration number, the initial compression mapping parameters obtained in the reverse adjustment are determined as the adjusted compression mapping parameters corresponding to the bounding box of the target region.

[0074] The fifth step involves performing background compression on the image to be processed and the target region bounding box set to obtain a compressed background image and background layer metadata. The compressed background image can be an image obtained by compressing the portion of the image to be processed outside the corresponding region of the target region bounding box set according to a uniform compression configuration. The background layer metadata can be descriptive information used to restore the position of the compressed background image in the image to be processed and the compression configuration during decompression. The background layer metadata may include, but is not limited to, at least one of the following: background image resolution, background compression quality value, and mask coordinates corresponding to the target region bounding box set. In practice, the execution entity can first construct a binary mask matrix according to the size of the image to be processed, set the element values ​​corresponding to the positions within the target region bounding box set in the binary mask matrix to 0, and set the element values ​​corresponding to the remaining positions to 1, thus obtaining a background mask. Then, the pixel values ​​corresponding to the positions in the image to be processed where the element value in the background mask is 0 are set to zero, thus obtaining an initial background image. Next, the `imencode` function in the OpenCV library is used to perform JPEG compression on the initial background image at a preset background compression quality value (e.g., 30) to obtain the compressed background image. Finally, the resolution of the compressed background image, the preset background compression quality value, and the coordinates of the bounding boxes of each target region in the target region bounding box set are determined as background layer metadata.

[0075] Step 6: Perform foreground differential compression processing on the image to be processed and the target region bounding box set to obtain a foreground sub-image set and foreground layer metadata. The foreground sub-images in the foreground sub-image set can be images obtained by compressing the image region corresponding to the target region bounding box according to the adjusted compression mapping parameters corresponding to the target region bounding box. The foreground layer metadata can be descriptive information used to locate the position of each foreground sub-image in the image to be processed and to restore its compression configuration during decompression. The foreground layer metadata may include, but is not limited to, at least one of the following: the coordinates of each target region bounding box, the adjusted compression mapping parameters corresponding to each target region bounding box, and the resolution of each foreground sub-image. In practice, the execution entity can perform the following foreground compression steps on each target region bounding box in the target region bounding box set: First, extract the image region corresponding to the target region bounding box from the image to be processed to obtain the corresponding foreground region image. Then, using the imencode function in the OpenCV library, perform JPEG compression processing on the foreground region image according to the adjusted compression mapping parameters corresponding to the target region bounding box in the adjusted compression mapping parameter set to obtain the corresponding foreground sub-image. Next, the obtained foreground sub-images are defined as a foreground sub-image set. Finally, the coordinates of each target region bounding box in the target region bounding box set, each adjusted compression mapping parameter in the adjusted compression mapping parameter set, and the resolution of each foreground sub-image in the foreground sub-image set are defined as foreground layer metadata.

[0076] Step 7: Based on the compressed background image, background layer metadata, foreground sub-image set, foreground layer metadata, and structured semantic information set, generate archived image data and save the archived image data to a storage medium. The preset archive format can be a pre-defined container format used to encapsulate the compressed background image, background layer metadata, foreground sub-image set, foreground layer metadata, and structured semantic information set into a single file. For example, the preset archive format can be a custom container format based on TIFF tag extensions or a custom container format based on HDF5 (Hierarchical Data Format Version 5). In practice, the executing entity can first serialize the background layer metadata, foreground layer metadata, and structured semantic information set into JSON format files. Then, using an archiving tool, write the compressed background image, each foreground sub-image in the foreground sub-image set, and each serialized JSON format file into an archive file according to a preset archive directory structure to obtain the archived image data. The archiving tool can include, but is not limited to, at least one of the following: the tarfile module in the Python standard library, or the zipfile module in the Python standard library.

[0077] The above technical solution and related content, as an inventive point of this disclosure, solves technical problem four: "increased storage space occupied by archived image data in the storage medium." Factors leading to increased storage space occupied by archived image data in the storage medium are often as follows: the foreground differentiation compression processing applies the same set of adjusted compression mapping parameters to the entire region corresponding to each target region bounding box, without subdividing the content type differences within the region corresponding to the target region bounding box, resulting in the compressed data volume of the local sub-image set being greater than the actual required compressed data volume. Solving these factors can reduce the storage space occupied by archived image data in the storage medium. To achieve this effect, this disclosure first performs semantic information encapsulation processing on the target content information set, the target region bounding box set, the local sub-image set, and the global thumbnail to obtain a structured semantic information set; and then performs aggregation processing on the structured semantic information set and the target region bounding box set to obtain a bounding box aggregation information set. Here, the bounding box aggregation information in the bounding box aggregation set groups the bounding boxes of each target region according to semantic category labels. This allows subsequent parameter mapping processing to distinguish the mixed content type features within the corresponding regions of the target region bounding boxes under different semantic categories, providing a grouping basis for differentiated allocation of compression mapping parameters based on content type. Next, parameter mapping processing is performed on the above bounding box aggregation information set to obtain the initial compression mapping parameter set. Subsequently, the above bounding box aggregation information set and the above initial compression mapping parameter set are continuously adjusted in reverse to obtain the adjusted compression mapping parameter set. Here, each adjusted compression mapping parameter in the adjusted compression mapping parameter set undergoes feedback adjustment based on peak signal-to-noise ratio to match the mixed content type features within the corresponding regions of the target region bounding boxes. This ensures that, while maintaining the quality of the compressed image, the amount of compressed data for the corresponding local sub-image is compressed to the required level, reducing the storage space occupied by the archived image data on the storage medium. Then, background compression processing is performed on the above image to be processed and the above target region bounding box set to obtain the compressed background image and background layer metadata. Next, foreground differential compression processing is performed on the aforementioned image to be processed and the aforementioned target region bounding box set to obtain a foreground sub-image set and foreground layer metadata. Here, each foreground sub-image in the foreground sub-image set is differentially compressed according to the corresponding adjusted compression mapping parameters, so that the compression configuration of each foreground sub-image is adapted to the mixed features of its internal text content, table line content, and other content types. The overall compressed data volume of the foreground sub-image set is significantly reduced compared to the method of using uniform compression mapping parameters for all foreground sub-images, thereby reducing the storage space occupied by the archived image data in the storage medium. Finally, based on the aforementioned compressed background image, the aforementioned background layer metadata, the aforementioned foreground sub-image set, the aforementioned foreground layer metadata, and the aforementioned structured semantic information set, archived image data is generated and the aforementioned archived image data is saved to the storage medium.Here, archived image data is obtained by layering and combining a compressed background image compressed according to a uniform compression configuration and a foreground sub-atlas compressed according to differentiated compression mapping parameters. Compared to saving the entire image to be processed at its original resolution, the storage space occupied by archived image data on the storage medium is significantly reduced. Therefore, the above technical solution and its related content can reduce the storage space occupied by archived image data on the storage medium.

[0078] The non-uniform image data processing method based on a visual language model, according to some embodiments of this disclosure, firstly performs text box localization processing on the acquired image to be processed, obtaining a set of text box coordinate information. Here, the text box coordinate information set reflects the spatial distribution of text boxes in the image to be processed, providing a data basis for subsequent density quantization of the spatial distribution of information in the image to be processed. Next, based on the aforementioned text box coordinate information set, the image to be processed is subjected to density quantization processing, obtaining a density matrix. Here, discrete text box position coordinates are transformed into continuous density data reflecting the spatial distribution of information in the image to be processed, providing a data basis for subsequent spatial differentiation between dense and sparse information regions. Then, the density matrix is ​​subjected to grid clustering processing to obtain an initial set of region bounding boxes. Here, the location range of dense information regions in the image to be processed is initially identified, spatially distinguishing dense and sparse information regions, ensuring that subsequent multi-scale regionalization extraction processing only targets dense information regions, avoiding invalid content recognition by calling the visual language model for sparse information regions. Afterwards, based on the aforementioned density matrix, the initial set of region bounding boxes is subjected to density adaptive expansion processing to obtain a target region bounding box set. Here, the target region bounding box set, relative to the initial region bounding box set, undergoes density-adaptive expansion of its boundaries. This ensures that it fully encompasses structures such as tables and text blocks within dense information regions, avoiding the situation where dense information regions are segmented into multiple bounding boxes due to boundary cutoff, requiring repeated inference by the visual language model. Then, based on the target region bounding box set, the image to be processed undergoes multi-scale regional extraction processing, resulting in a global thumbnail and a set of local sub-images. Here, the local sub-images only carry the details of dense information regions in the image to be processed, while the global thumbnail carries the sparse information regions and the overall layout of the image to be processed at a lower resolution. This allows the visual language model to only identify the details of dense information regions, while perceiving sparse information regions as a whole through the global thumbnail, reducing the lexical consumption caused by the visual language model's detailed identification of sparse information regions. Finally, the visual language model performs fusion inference processing on the global thumbnail and the local sub-images to obtain the target content information set. Here, the visual language model combines the overall layout context carried by the global thumbnail with the dense information details carried by the local sub-image set to output the target content information set. This ensures that the input to the visual language model's inference process only includes the overall layout context and dense information details, avoiding the input of sparse information regions for detail recognition. This reduces lexical consumption during the visual language model's inference process and lowers the computational resource consumption required for inference. Finally, based on the aforementioned target content information set, the aforementioned target region bounding box set, and the aforementioned local sub-image set, the image to be processed undergoes differential compression processing to obtain archived image data, which is then saved to a storage medium.Here, dense information regions in the archived image data are preserved at higher resolution, while sparse information regions are preserved at lower resolution. This avoids storing sparse information regions at the same high resolution, thereby reducing the storage space occupied when storing the images to be processed. Therefore, this non-uniform image data processing method based on a visual language model can reduce lexical consumption during visual language model inference, lower the computational resource consumption required for visual language model inference, and reduce the storage space occupied when storing the images to be processed.

[0079] Further reference Figure 2 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a non-uniform image data processing apparatus based on a visual language model. These apparatus embodiments are similar to... Figure 1 Corresponding to the method embodiments shown, this non-uniform image data processing device based on a visual language model can be specifically applied to various electronic devices.

[0080] like Figure 2 As shown, a non-uniform image data processing device 200 based on a visual language model includes: an acquisition unit 201, a quantization unit 202, a clustering unit 203, an expansion unit 204, an extraction unit 205, an inference unit 206, and a compression unit 207. The acquisition unit 201 is configured to perform text box localization processing on the acquired image to be processed to obtain a set of text box coordinate information. The quantization unit 202 is configured to perform density quantization processing on the image to be processed based on the text box coordinate information set to obtain a density matrix. The clustering unit 203 is configured to perform grid clustering processing on the density matrix to obtain an initial set of region bounding boxes. The expansion unit 204 is configured to perform density adaptive expansion processing on the initial set of region bounding boxes based on the density matrix to obtain a set of target region bounding boxes. The extraction unit 205 is configured to perform multi-scale regional extraction processing on the image to be processed based on the target region bounding box set to obtain a global thumbnail and a set of local sub-images. The inference unit 206 is configured to: use a visual language model to perform fusion inference processing on the aforementioned global thumbnail and the aforementioned local sub-image set to obtain a target content information set. The compression unit 207 is configured to: perform differential compression processing on the aforementioned image to be processed based on the aforementioned target content information set, the aforementioned target region bounding box set, and the aforementioned local sub-image set to obtain archived image data, and save the aforementioned archived image data to a storage medium.

[0081] It is understandable that the units described in the non-uniform image data processing apparatus 200 based on a visual language model and the reference Figure 1The steps in the described method correspond accordingly. Therefore, the operations, features, and beneficial effects described above for the method also apply to the non-uniform image data processing device 200 based on the visual language model and the units contained therein, and will not be repeated here.

[0082] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0083] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including an acquisition unit, a quantization unit, a clustering unit, an expansion unit, an extraction unit, an inference unit, and a compression unit. The names of these units do not necessarily limit the specific unit; for example, an acquisition unit may be described as "a unit that performs text box positioning processing on the acquired image to be processed to obtain a set of text box coordinate information."

[0084] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0085] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for processing non-uniform image data based on a visual language model, characterized in that, include: The acquired image to be processed is subjected to text box positioning processing to obtain a set of text box coordinate information; Based on the set of text box coordinate information, the image to be processed is subjected to density quantization to obtain a density matrix; The density matrix is ​​subjected to grid clustering to obtain an initial set of region bounding boxes; Based on the density matrix, the initial region bounding box set is subjected to density adaptive expansion processing to obtain the target region bounding box set; Based on the target region bounding box set, the image to be processed is subjected to multi-scale regional extraction processing to obtain a global thumbnail and a local sub-image set; Using a visual language model, the global thumbnail and the local sub-image set are fused and reasoned to obtain the target content information set; Based on the target content information set, the target region bounding box set, and the local sub-image set, the image to be processed is subjected to differential compression processing to obtain archived image data, and the archived image data is saved to a storage medium.

2. The method according to claim 1, characterized in that, The step of performing text box positioning processing on the acquired image to be processed to obtain a set of text box coordinate information includes: The image to be processed is converted to grayscale to obtain a grayscale image; The grayscale image is denoised to obtain a denoised grayscale image; The denoised grayscale image is subjected to contrast enhancement processing to obtain the enhanced image; The enhanced image is segmented and localized to obtain a set of text box coordinate information.

3. The method according to claim 2, characterized in that, The step of performing block localization processing on the enhanced image to obtain a set of text box coordinate information includes: The enhanced image is segmented without overlap according to the preset block size to obtain an image block set; For each image block in the image block set, text box positioning is performed to obtain a local text box coordinate set; Based on the offset of each image block in the enhanced image from the image block set, the local text box coordinate set is subjected to global coordinate mapping to obtain the text box coordinate information set.

4. The method according to claim 1, characterized in that, The step of performing density quantization processing on the image to be processed based on the text box coordinate information set to obtain a density matrix includes: The image to be processed is divided into grids to obtain a set of grid cells; Based on the set of text box coordinate information, perform text box density statistical processing on the grid cell set to obtain an initial density matrix; The initial density matrix is ​​normalized to obtain the density matrix.

5. The method according to claim 1, characterized in that, The step of performing grid clustering on the density matrix to obtain an initial set of region bounding boxes includes: Filter the grid cells whose density values ​​are greater than or equal to a preset density threshold from the density matrix to form the target grid cell set; Based on a preset neighborhood radius and a preset sample number threshold, the target grid cell set is subjected to density connectivity clustering to obtain a target grid cell cluster set; The bounding rectangle extraction process is performed on each target grid cell cluster in the target grid cell cluster set to obtain the initial region bounding box set.

6. The method according to claim 1, characterized in that, The step of performing density adaptive expansion processing on the initial region bounding box set based on the density matrix to obtain the target region bounding box set includes: For each initial region bounding box in the initial region bounding box set, perform the following extension steps: The four boundaries of the initial region boundary box are extended outward by a distance corresponding to a preset extension ratio to obtain an extended region group; Based on the density matrix, perform density mean statistical processing on each extended region in the extended region group to obtain the extended region density mean group. The initial region bounding box is subjected to shape classification processing to obtain shape category labels; Based on the shape category label, determine the basic expansion ratio of the initial region bounding box in each direction to obtain the basic expansion ratio group; Based on the average density group of the extended region, the basic extension ratio group is subjected to density scaling to obtain the target extension ratio group; According to the target expansion ratio group, the initial region bounding box is subjected to density convergence expansion processing to obtain the expanded bounding box; The expanded bounding box set is then overlapped and merged to obtain the target region bounding box set.

7. A non-uniform image data processing device based on a visual language model, characterized in that, include: The acquisition unit is configured to perform text box positioning processing on the acquired image to be processed to obtain a set of text box coordinate information. The quantization unit is configured to perform density quantization on the image to be processed based on the set of text box coordinate information to obtain a density matrix; Clustering units are configured to perform grid clustering on the density matrix to obtain an initial set of region bounding boxes; The expansion unit is configured to perform density adaptive expansion processing on the initial region bounding box set according to the density matrix to obtain the target region bounding box set; The extraction unit is configured to perform multi-scale regional extraction processing on the image to be processed based on the target region bounding box set to obtain a global thumbnail and a local sub-image set; The reasoning unit is configured to use a visual language model to perform fusion reasoning processing on the global thumbnail and the local sub-image set to obtain a target content information set. The compression unit is configured to perform differential compression processing on the image to be processed based on the target content information set, the target region bounding box set, and the local sub-image set to obtain archived image data, and to save the archived image data to a storage medium.

Images