Image processing method and device, electronic equipment and storage medium

Abstract

The application discloses an image processing method and device, electronic equipment and storage medium, relates to the computer technical field, and especially relates to the artificial intelligence field of deep learning and computer vision. The specific implementation scheme is as follows: obtaining a to-be-processed task; wherein the to-be-processed task comprises an encoding code stream of a to-be-processed image; extracting first encoding information of each first encoding block of the to-be-processed image from the encoding code stream; determining the region importance weight of the image block embedding vector of the to-be-processed image according to the first encoding information; wherein the image block embedding vector is obtained by segmenting the to-be-processed image into multiple image regions by using a visual model and performing embedding mapping on the image regions; the region importance weight is used for representing the importance degree of the image region corresponding to the image block embedding vector; and the visual model is used for performing the to-be-processed task on the to-be-processed image based on the region importance weight to obtain a task processing result.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more particularly to the field of artificial intelligence, including deep learning and computer vision, specifically to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of deep learning technology, attention-based visual models, such as ViT (VisionTransformer) and its derivative structures, have been widely used in fields such as image generation, image understanding and multimodal generation.

[0003] In related technologies, modeling the importance of image regions typically employs prior methods based on saliency or edge detection, or methods based on multi-scale feature fusion. However, saliency detection or multi-scale feature extraction usually requires additional network structures, which increases the number of model parameters and inference latency. Summary of the Invention

[0004] This application provides an image processing method, apparatus, electronic device, and storage medium. The specific solutions are as follows:

[0005] According to one aspect of this application, an image processing method is provided, comprising: Obtain the task to be processed; wherein, the task to be processed includes the encoded bitstream of the image to be processed; Extract the first encoding information of each first encoding block of the image to be processed from the encoded bitstream; Based on the first encoding information, the region importance weights of each image patch embedding vector of the image to be processed are determined; wherein, the image patch embedding vectors are obtained by segmenting the image to be processed into multiple image regions using a visual model and embedding the image regions; the region importance weights are used to characterize the importance of the image region corresponding to the image patch embedding vector. The visual model is used to perform the processing task on the image to be processed based on the region importance weights, and the task processing result is obtained.

[0006] According to another aspect of this application, an image processing apparatus is provided, comprising: An acquisition module is used to acquire a task to be processed; wherein, the task to be processed includes the encoded bitstream of the image to be processed; An extraction module is used to extract the first encoding information of each first encoding block of the image to be processed from the encoded bitstream; The determining module is configured to determine the regional importance weights of each image patch embedding vector of the image to be processed based on the first encoding information; wherein, the image patch embedding vector is obtained by segmenting the image to be processed into multiple image regions using a visual model and embedding the image regions; the regional importance weight is used to characterize the importance of the image region corresponding to the image patch embedding vector. The task processing module is used to perform the task to be processed on the image to be processed based on the region importance weight using the visual model, and obtain the task processing result.

[0007] According to another aspect of this application, an electronic device is provided, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method described in the above embodiments.

[0008] According to another aspect of this application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to perform the method described in the above embodiments.

[0009] According to another aspect of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described in the above embodiments.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided for a better understanding of this solution and do not constitute a limitation of this application. Wherein: Figure 1 A schematic flowchart of an image processing method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the coded block distribution of an image. Figure 3 A schematic flowchart illustrating an image processing method provided in another embodiment of this application; Figure 4 A schematic flowchart illustrating an image processing method provided in another embodiment of this application; Figure 5 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application; Figure 6 This is a block diagram of an electronic device used to implement the image processing method of the embodiments of this application. Detailed Implementation

[0012] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0013] It should be noted that the acquisition, storage, use, and processing of data in this application comply with the relevant provisions of national laws and regulations and do not violate public order and good morals.

[0014] The image processing method, apparatus, electronic device, and storage medium of this application are described below with reference to the accompanying drawings.

[0015] Figure 1 This is a schematic flowchart of an image processing method provided in an embodiment of this application.

[0016] The image processing method of this application embodiment can be executed by the image processing device of this application embodiment, which can be configured in an electronic device.

[0017] Among them, electronic devices can be any device with computing capabilities, such as personal computers, mobile terminals, servers, etc. Mobile terminals can be hardware devices with various operating systems, touch screens and / or displays, such as in-vehicle devices, mobile phones, tablets, personal digital assistants, wearable devices, etc.

[0018] like Figure 1 As shown, the image processing method includes: Step 101: Obtain the tasks to be processed.

[0019] In this application, the task to be processed can be any of the following image tasks: image to image (i.e., image-to-image), image to text (i.e., image-to-text), image to video (i.e., image-to-video), image classification, object detection, semantic segmentation, etc., without limitation.

[0020] In this application, the task to be processed may include an encoded bitstream of the image to be processed. The encoded bitstream is a binary data sequence generated after processing the original image to be processed using a compression encoding algorithm. It can be understood that the encoded bitstream is a compressed bit sequence.

[0021] For example, the visual model used to process the image to be processed can be a single-task model or a multi-task model. For instance, the visual model can perform image-to-image tasks or image-to-video tasks. If the visual model is a multi-task model, the task to be processed can also include task type information to indicate which image task among multiple tasks the visual model should perform.

[0022] Step 102: Extract the first encoding information of each first encoding block of the image to be processed from the encoded bitstream of the image to be processed.

[0023] Since the encoded bitstream is not a pixel matrix, it cannot be directly used for display or deep learning model input. It needs to be parsed or decoded first to obtain the decompressed pixel image. In this application, the first encoding information of each first encoding block can be extracted from the encoded bitstream during or after the decoding process.

[0024] For example, the first coding block can be a macroblock, coding unit, coding tree unit, transform block, etc. Alternatively, macroblocks, coding units, coding tree units, transform blocks, etc., are specific implementations of coding blocks under different image coding standards.

[0025] For example, a first coding block may include sub-blocks. The number of sub-blocks can express the complexity of the information in the first coding block. The more sub-blocks there are, the more complex the first coding block is. The fewer sub-blocks there are, the flatter and simpler the first coding block is.

[0026] For example, such as Figure 2 As shown, the image contains multiple coded blocks with a side length of 'a' (i.e., squares with a side length of 'a' in the image). Taking the coded block in the upper left corner of the image as an example, this coded block is divided into 4 smaller squares. The upper right square of these 4 smaller squares is further divided into 4 newer smaller squares. These 4 smaller squares can be further divided. These small squares of different sizes in the coded block are called sub-blocks. The richer the image information, the more sub-block levels are guaranteed, and the more sub-blocks are contained. For example, the number of sub-blocks contained in the coded block in the human portrait area of ​​the image is greater than the number of sub-blocks contained in the coded block in the background area.

[0027] For example, the first encoding information may include, but is not limited to, the spatial location, size information, hierarchical structure, sub-block partitioning depth information, codeword length, bit consumption information, etc. of the first encoding block. For example, the larger the amount of information in the first encoding block and the larger the number of bits used, the longer the codeword length.

[0028] Step 103: Determine the regional importance weights of the embedding vectors of each image block of the image to be processed based on the first encoding information.

[0029] In this application, the image patch embedding vector can be obtained by segmenting the image to be processed into multiple image regions using a visual model and then embedding and mapping these image regions. These multiple image regions do not overlap. That is, a visual model can be used to divide the image to be processed into multiple non-overlapping image regions, and each image region can be mapped to a corresponding image patch embedding vector.

[0030] For example, the input embedding layer of the visual model can be used to spatially partition and linearly project the image to be processed, where spatial partitioning divides the image to be processed into multiple non-overlapping image regions.

[0031] For example, each image patch embedding vector represents the semantic or visual features of a local region (i.e., an image region) in the image to be processed. The image patch embedding vector can be a fixed-dimensional vector (e.g., 768-dimensional) that can be mapped from the original pixel patch by a visual model through linear projection or an embedding layer.

[0032] Suppose a 224×224 RGB image is divided into 16×16 image regions. Each image region has 16×16×3=768 pixel values. Through a linear layer (such as a fully connected layer), these 768 values ​​are mapped into a 768-dimensional vector, which is an image patch embedding vector. The entire image will generate (224 / 16)²=196 image patch embedding vectors.

[0033] In this application, the regional importance weight of the image patch embedding vector can be used to characterize the importance of the image region corresponding to the image patch embedding vector.

[0034] Since the number of sub-blocks in a coding block can express the complexity of the information in the coding block, in this application, the regional importance weight of the embedding vector of each image block of the image to be processed can be determined based on the coding information of each first coding block of the image to be processed.

[0035] For example, for each image patch embedding vector, a coding block corresponding to the image patch embedding vector can be determined, and the regional importance weight of the image patch embedding vector can be determined based on the first coding information of the coding block corresponding to the image patch embedding vector.

[0036] For example, the visual model used in this application can be a regular visual model or a visual model based on an attention mechanism, such as ViT or a derivative structure of ViT.

[0037] In some embodiments, "image region" may also be described as "image patch", "patch", etc., and this application does not limit the name of "image region".

[0038] In some embodiments, "image patch embedding vector" may also be described as "image token", "image embedding unit", "image patch embedding", etc. This application does not limit the name of "image patch embedding vector".

[0039] Step 104: Using a visual model based on region importance weights, perform the processing task on the image to be processed and obtain the task processing result.

[0040] In this application, regional importance weights can be introduced into the visual model, and based on the regional importance weights, the processing task can be performed on the image to be processed to obtain the task processing result.

[0041] For example, each image patch embedding vector can be multiplied by its corresponding region importance weight to obtain an updated image patch embedding vector. These updated image patch embedding vectors can form an image embedding sequence, which can then be input into the encoder of the visual model for feature extraction.

[0042] For example, the task execution result matches the task type of the task to be executed. For instance, if the task to be executed is to generate an image from an image, then the task processing result may include the generated image; if the task to be executed is to generate text from an image, then the task processing result may include the generated text.

[0043] In this embodiment, by extracting the encoding information of each first encoding block from the compressed domain encoded bitstream of the image to be processed, and using the encoding information naturally generated during the encoding process of the image to be processed, the regional importance weights of the embedding vectors of each image block of the image to be processed are determined. Thus, without introducing additional network structures or increasing the number of model parameters, the visual model can be guided to focus on high-information regions that are allocated more resources during the compression encoding process. This not only reduces computational overhead and cost, but also improves the model's ability to perceive key semantic regions and enhances the overall execution efficiency of image processing tasks.

[0044] Figure 3 This is a schematic flowchart of an image processing method provided in another embodiment of this application.

[0045] like Figure 3 As shown, the image processing method includes: Step 301: Obtain the tasks to be processed.

[0046] Step 302: Extract the first encoding information of each first encoding block of the image to be processed from the encoded bitstream of the image to be processed.

[0047] In this application, steps 301-302 can be implemented in any of the embodiments of this application, so they will not be described in detail here.

[0048] Step 303: Determine the set of encoding blocks corresponding to the image block embedding vector from each first encoding block according to the first encoding information.

[0049] Typically, the size of a coding block differs from the size of an image region (patch). To address the inconsistency between the spatial partitioning of image coding blocks and the spatial partitioning of image block embedding vectors, this application allows the determination of the coding block set corresponding to the image block embedding vector from each first coding block.

[0050] In this application, the set of coding blocks corresponding to the image block embedding vector can be understood as the set of first coding blocks that overlap with the image region corresponding to the image block embedding vector in the image to be processed.

[0051] For example, a first coding block that overlaps with the image region corresponding to the image block embedding vector can be used as a second coding block, and the set of coding blocks includes at least one second coding block.

[0052] For example, the overlap between the first coding block and the image region can be understood as the overlap between the coverage area of ​​the image region in the pixel space of the image to be processed and the coverage area of ​​the first coding block in the pixel space.

[0053] For example, the overlap here can include complete overlap and partial overlap, that is, the first coding block located entirely in the image area or partially in the image area can be used as the second coding block.

[0054] For example, there are 3 first coding blocks that overlap with the image region corresponding to the image block embedding vector. Among them, 2 first coding blocks are completely covered by the image region, and 1 first coding block is not completely covered by the image region. These 3 first coding blocks can all be used as second coding blocks, that is, the set of coding blocks includes 3 second coding blocks.

[0055] In some embodiments, coding block spatial mapping information of the image to be processed can be constructed based on the first coding information of each first coding block, and then a set of coding blocks can be determined from each first coding block based on the coding block spatial mapping information.

[0056] For example, coded block space mapping information can be used to describe the region division of the image to be processed in the coded space.

[0057] For example, the coding block space mapping information may include, but is not limited to, the correspondence between the first coding block and the image pixel space, the coverage of the first coding block in the pixel space of the image to be processed, the hierarchy or adjacency relationship between the first coding blocks, etc.

[0058] For example, the following method can be used to determine the set of coding blocks from each first coding block based on the coding block space mapping information: the first coding block in each first coding block that overlaps with the image region corresponding to the image block embedding vector can be determined based on the coding block space mapping information, and the set of coding blocks can be determined based on the first coding block that overlaps with the image region corresponding to the image block embedding vector.

[0059] For example, the coverage range of the image block embedding vector in the pixel space can be determined based on the position of the image region in the pixel space of the image to be processed. For example, the coverage range of the image region in the pixel space can be used as the coverage range of the image block embedding vector in the pixel space. Based on the coding block space mapping information, the first coding block that overlaps with the coverage range in each first coding block can be determined. The first coding block that overlaps with the coverage range is determined as the first coding block that overlaps with the image region corresponding to the image block embedding vector.

[0060] Therefore, by utilizing the spatial mapping information of the coding blocks, all first coding blocks that spatially overlap with the image region covered by each image block embedding vector are identified, and these coding blocks are aggregated into a coding block set associated with the embedding vector. This achieves fine-grained alignment from pixel domain semantic units to compressed domain syntax units, which can effectively bridge the semantic gap between the compressed domain and the pixel domain, and improve the visual model's ability to perceive key regions and its task performance under limited computing resources.

[0061] As an example, if the coded block space mapping information includes the coverage range of the first coded block in the pixel space, it is possible to determine which first coded blocks have an overlap with the coverage range of the image block embedding vector in the pixel space based on the coverage range of the first coded block in the pixel space and the coverage range of the image block embedding vector in the pixel space. These first coded blocks with overlap are the first coded blocks that overlap with the image region corresponding to the image block embedding vector.

[0062] As another example, if the coding block space mapping information includes the correspondence between the first coding block and the pixel space, the coverage of the first coding block in the pixel space can be determined based on the correspondence. Furthermore, based on the coverage of the first coding block in the pixel space, the method in the previous example can be used to determine the first coding block that overlaps with the image region corresponding to the image block embedding vector.

[0063] Therefore, based on the original image region position corresponding to the image patch embedding vector, its coverage range in pixel space is determined, and the mapping relationship between the coding block and the pixel space is used to filter out the coding blocks that have spatial overlap with the coverage range. Thus, by establishing the association between the geometric coverage range of the image patch embedding vector in pixel space and the spatial mapping relationship of the first coding block in the coding stream, the precise alignment of the compressed domain structure information and the visual model input unit (i.e., the image patch embedding vector) is achieved.

[0064] For example, a first coding block that overlaps with the image region corresponding to the image block embedding vector can be used as a second coding block, and a coding block set can be formed by the second coding blocks.

[0065] In this embodiment, by parsing the first encoding information, the encoding block space mapping information of the image to be processed in the compression domain is constructed. This mapping information accurately describes the topological structure of the regions divided in the image during the encoding process and their geometric correspondence in the pixel space. Based on this, the set of encoding blocks corresponding to the image block embedding vector is selected, which can realize the fine-grained and geometrically aligned association between the image block embedding vector and the encoding block. This can provide a structure-aware and position-accurate compression domain prior for subsequent region importance assessment.

[0066] Step 304: Determine the regional importance weight based on the first coding information of the second coding block in the coding block set.

[0067] Since different bit resources have been allocated to different regions during the image encoding and compression process through quantization, transformation and entropy coding, in this application, the region importance weight can be determined based on the first coding information of the second coding block in the coding block set corresponding to the image block embedding vector.

[0068] In some embodiments, the information content of the second coding block can be determined based on the first coding information of the second coding block, and the information content of the second coding block in the coding block set can be aggregated to obtain the regional importance weight.

[0069] For example, the information content of the second coding block can reflect the degree of importance attached to the image region corresponding to the second coding block during the compression coding process. For example, the information content of the second coding block can be characterized by one or more of the following: codeword length, partition depth, bit distribution density, quantization intensity, etc. of the second coding block.

[0070] For example, aggregation processing may include, but is not limited to, weighted summation, normalization, finding the maximum value, and calculating the mean.

[0071] As an example, for any image patch embedding vector, the information content of the second coded block in the set of coded blocks can be compared, and the largest information content can be selected as the region importance weight.

[0072] As another example, for any image patch embedding vector, the average information content of the second coded block in the set of coded blocks can be calculated, and the average value can be used as the region importance weight.

[0073] As another example, the information content of the second coding block can be weighted according to the information weight of the second coding block in the coding block set, and the weighted result can be used as the regional importance weight.

[0074] For example, the information weight of the second coding block can be determined based on the number of sub-blocks contained in the second coding block; the more sub-blocks, the greater the information weight.

[0075] For example, the information weight of the second coding block can also be determined based on the degree of coverage of the image region corresponding to the image block embedding vector to the second coding block. Here, the degree of coverage can be used to indicate the degree of overlap between the image region and the second coding block in pixel space, or the proportion of overlapping area, or positional inclusion relationship, etc. For example, the greater the coverage, the greater the information weight.

[0076] For example, the coverage can be equal to the ratio of the number of pixels in the overlapping portion to the maximum number of pixels, where the maximum number of pixels is the maximum of the number of pixels in the image region and the number of pixels in the coverage area of ​​the second coding block in the pixel space.

[0077] Therefore, by quantifying the spatial coverage of the image region corresponding to the image block embedding vector to the second coding block, each second coding block is dynamically assigned an information weight, thereby reflecting the contribution of the coding block to the current image region. On this basis, the information carried by each second coding block is weighted and fused according to its corresponding information weight, and finally a region importance weight that accurately matches the image block embedding vector is generated. This can achieve adaptive mapping from compressed domain information to pixel domain semantic units, and can avoid weight distortion caused by simple averaging or hard allocation. This makes the importance assessment consider both the "information density" of the coding block itself and its spatial correlation with the target image region, thereby significantly improving the sensitivity and robustness of the visual model to key regions in the image. For example, it can provide more refined and reliable spatial guidance signals for the attention mechanism of the visual model, effectively improving the model's ability to focus on key content and task performance.

[0078] In this embodiment, the first coding information of the second coding block in the coding set corresponding to the image block embedding vector is used to quantify the amount of information it carries in the compressed domain. By aggregating the information of multiple second coding blocks in the coding block set, a region importance weight that is precisely aligned with the image block embedding vector is generated. Thus, the visual model can focus on high information density regions that are allocated more resources during the coding process, which can significantly improve the discrimination ability of the attention mechanism and the performance robustness of downstream tasks.

[0079] Step 305: Using a visual model based on region importance weights, perform the processing task on the image to be processed and obtain the task processing result.

[0080] In this application, step 305 can be implemented in any of the embodiments of this application, so it will not be described in detail here.

[0081] In this embodiment, the first coding block spatially associated with the image block embedding vector can be accurately selected based on the coding information of each first coding block in the image to be processed, forming its corresponding coding block set. The information importance of the image region is quantified by using the compression domain features reflected by the first coding information of the coding blocks in the coding block set, thereby generating the region importance weight corresponding to the image block embedding vector. This can realize the semantic bridge from the original compression structure to the image block embedding vector, so that the region importance weight is not only derived from the pixel content, but also rooted in the amount of image information implicit in the coding process, which can significantly improve the sensitivity and robustness of the visual model to key regions in the image.

[0082] Figure 4 This is a schematic flowchart of an image processing method provided in another embodiment of this application.

[0083] like Figure 4 As shown, the image processing method includes: Step 401: Obtain the tasks to be processed.

[0084] In this application, step 401 can be implemented in any of the embodiments of this application, so it will not be described in detail here.

[0085] Step 402: Extract the first encoding information of each first encoding block of the image to be processed from the encoded bitstream of the image to be processed.

[0086] In this application, step 402 can be implemented in any of the embodiments of this application, so it will not be described in detail here.

[0087] In some embodiments, the first encoding information of each encoding block can be extracted from the encoded bitstream in the following manner: the encoded bitstream of the image to be processed can be parsed according to the compression encoding specification to obtain the second encoding information of the image to be processed, and the first encoding information can be extracted from the encoded bitstream based on the second encoding information.

[0088] For example, the second encoding information can be understood as global encoding information, and the first encoding information can be understood as local encoding information.

[0089] For example, the second encoded information may include, but is not limited to, the basic image attributes, encoding parameters, block partitioning structure, entropy encoding configuration, metadata, etc. of the image to be processed.

[0090] For example, basic image attributes may include, but are not limited to, width, height, color space, bit depth, number of channels, etc.

[0091] For example, encoding parameters may include, but are not limited to, quantization parameters, change types, prediction modes, etc.

[0092] For example, the block partitioning structure may include the coding block size, quadtree partitioning structure, etc.

[0093] For example, entropy coding configurations may include, but are not limited to, Huffman tables, CABAC (Context-Based Adaptive Binary Arithmetic Coding) context models, etc.

[0094] For example, metadata may include, but is limited to, the time the image to be processed was captured, the device model, color profile, etc.

[0095] For example, an encoding syntax parsing operation can be performed on the encoded bitstream of the image to be processed to obtain second encoding information. For example, the parsing operation includes, but is not limited to, parsing the bitstream type, entropy decoding, and parsing the basic specifications.

[0096] For example, the bitstream type refers to the image encoding standard type, such as JPEG (Joint Photographic Experts Group) or JPEG XL (JPEG Extended Range). Entropy decoding is a crucial step in the parsing of the encoded bitstream, used to restore the compressed variable-length codewords to the original symbol sequence, achieving lossless restoration using entropy coding principles from information theory (such as Huffman coding and arithmetic coding). Entropy decoding does not involve inverse quantization or IDCT (Inverse Discrete Cosine Transform); it only restores the symbol stream output during encoding, which is still compressed domain data, not pixels.

[0097] For example, taking JPEG as an example, the quantized DCT (Discrete Cosine Transform) coefficient matrix can be obtained through entropy decoding.

[0098] For example, the basic specification can refer to the bitstream syntax structure specification defined in the image coding standard. The basic specification stipulates the position, length, and meaning of each field in the bitstream. By parsing the frame header syntax elements in the bitstream that conform to the preset coding standard, the width, height, color format information, etc. of the image can be extracted.

[0099] Taking JPEG as an example, the specific steps for performing encoding syntax parsing on the encoded bitstream can include: first identifying the start identifier of the encoded bitstream, then parsing the segment structure, performing entropy decoding, and finally extracting metadata.

[0100] In this embodiment, the encoded bitstream of the image to be processed is first parsed at the syntax level to obtain the second encoding information used to describe the overall encoding structure. Then, based on this high-level semantic guidance, the first encoding information of each encoding block is selectively and accurately extracted from the original bitstream. This two-level mechanism of "parsing the structure first and then extracting in a targeted manner" can avoid blindly traversing or completely decoding the entire bitstream, and can significantly improve the efficiency and accuracy of encoding information extraction from encoding blocks.

[0101] Meanwhile, because the second encoded information provides contextual constraints and structural priors, the extraction process of the first encoded information possesses semantic consistency and spatial integrity, laying a high-fidelity compressed domain foundation for subsequent region importance assessment and attention modulation. Thus, without reconstructing pixels, efficient, structure-aware parsing of the internal structure of the compressed bitstream is achieved, balancing computational overhead and information fidelity, effectively supporting collaborative optimization between the compressed domain and the visual model.

[0102] Step 403: Determine the regional importance weights of the embedding vectors of each image block of the image to be processed based on the first encoding information.

[0103] In this application, step 403 can be implemented in any of the embodiments of this application, so it will not be described in detail here.

[0104] Step 404: The encoder performs multi-layer attention calculation on the image embedding sequence obtained by mapping multiple image regions, and adjusts the attention distribution based on the region importance weight in each layer to obtain the visual feature representation of the image to be processed.

[0105] In this application, the visual model may include an encoder and a decoder. The encoder may be used to perform multi-layer attention calculation on the image embedding sequence to obtain the visual feature representation of the image to be processed.

[0106] In some embodiments, an encoder may be used to perform multi-layer feature extraction on the image embedding sequence, wherein in each layer of the encoder, an attention network performs attention calculation on the image embedding sequence based on region importance weights to obtain a visual feature representation of the image to be processed.

[0107] For example, in each layer of attention computation, attention is calculated on the feature representation output by the previous layer based on the region importance weight, so that the obtained feature representation is input into the next layer for attention computation. For example, the way to introduce region importance weight into attention computation may include, but is not limited to: as a bias term of attention weight, as a scaling factor of attention score, as a gating parameter of attention computation, etc.

[0108] In one embodiment, region importance weights can be introduced into each layer of the ViT encoder's MHSA (Multi-Head Self-Attention) module as a scaling factor or bias term for the attention score, thereby enhancing the model's attention to high-information image regions. The image embedding sequence can be formed by segmenting, linearly projecting, and positionally encoding the image to be processed, and includes multiple image block embedding vectors and optional class tokens.

[0109] Step 405: Use a decoder to decode the visual feature representation to obtain the task processing result.

[0110] In this application, a decoder can be used to decode the visual feature representation to obtain a task processing result that matches the task type to be executed.

[0111] In this embodiment, by utilizing the encoding information of each encoding block extracted from the encoded bitstream of the image to be processed, an importance weight of the image region is constructed, and this weight is introduced into the attention calculation of the visual model, thereby guiding the model to focus on high information density regions in the image. This not only enhances the model's ability to learn salient features, but also makes the generated visual feature representation richer in semantic information and structural awareness. Thus, by combining the synergistic optimization of compression domain prior knowledge and pixel domain attention mechanism, the sensitivity of the visual model to key content can be improved, while the efficiency and accuracy of downstream tasks can be enhanced.

[0112] To implement the above embodiments, this application also proposes an image processing apparatus. Figure 5 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application.

[0113] like Figure 5 As shown, the image processing apparatus 500 includes: The acquisition module 510 is used to acquire the task to be processed; wherein, the task to be processed includes the encoded bitstream of the image to be processed; Extraction module 520 is used to extract the first encoding information of each first encoding block of the image to be processed from the encoded bitstream; The determining module 530 is used to determine the regional importance weights of each image patch embedding vector of the image to be processed based on the first encoding information; wherein, the image patch embedding vector is obtained by segmenting the image to be processed into multiple image regions using a visual model and embedding the image regions; the regional importance weight is used to characterize the importance of the image region corresponding to the image patch embedding vector. The task processing module 540 is used to perform processing tasks on the image to be processed based on the region importance weight using a visual model, and obtain the task processing results.

[0114] Optionally, module 530 is defined for: Based on the first encoding information, determine the set of encoding blocks corresponding to the image block embedding vector from each first encoding block; The regional importance weights are determined based on the first coding information of the second coding block in the coding block set.

[0115] Optionally, module 530 is defined for: Based on the first encoded information of the second encoded block, determine the amount of information in the second encoded block; The information content of the second coding block in the coding block set is aggregated to obtain the regional importance weight.

[0116] Optionally, module 530 is defined for: The information weight of the second coding block is determined based on the coverage of the image region corresponding to the image block embedding vector to the second coding block. Based on the information weights, the information quantities are weighted to obtain the regional importance weights.

[0117] Optionally, module 530 is defined for: Based on the first encoding information, the encoding block space mapping information of the image to be processed is constructed; wherein, the encoding block space mapping information is used to describe the region division of the image to be processed in the encoding space; Based on the coding block space mapping information, the coding block set is determined from each first coding block.

[0118] Optionally, module 530 is defined for: Based on the spatial mapping information of the coding block, the first coding block in each first coding block that overlaps with the image region corresponding to the image block embedding vector is determined; The set of coded blocks is determined based on the first coded block that overlaps with the image region corresponding to the image block embedding vector.

[0119] Optionally, module 530 is defined for: Based on the position of the image region in the pixel space of the image to be processed, determine the coverage of the image block embedding vector in the pixel space; Based on the spatial mapping information of the coding block, the first coding block that overlaps with the coverage area is determined in each first coding block; The first coding block that overlaps with the coverage area is determined as the first coding block that overlaps with the image region corresponding to the image block embedding vector.

[0120] Optionally, the extraction module 520 is used for: The encoded bitstream is parsed to obtain the second encoded information of the image to be processed; Based on the second encoding information, the first encoding information is extracted from the encoded bitstream.

[0121] Optionally, the visual model includes an encoder and a decoder, and the task processing module 540 is used for: An encoder is used to perform multi-layer attention calculation on the image embedding sequence obtained by mapping multiple image regions, and the attention distribution is adjusted based on the region importance weight in each layer to obtain the visual feature representation of the image to be processed. A decoder is used to decode the visual feature representation to obtain the task processing result.

[0122] It should be noted that the explanation of the foregoing image processing method embodiments also applies to the image processing apparatus of this embodiment, and therefore will not be repeated here.

[0123] In this embodiment, by extracting the encoding information of each first encoding block from the compressed domain encoded bitstream of the image to be processed, and using the encoding information naturally generated during the encoding process of the image to be processed, the regional importance weights of the embedding vectors of each image block of the image to be processed are determined. Thus, without introducing additional network structures or increasing the number of model parameters, the visual model can be guided to focus on high-information regions that are allocated more resources during the compression encoding process. This not only reduces computational overhead and cost, but also improves the model's ability to perceive key semantic regions and enhances the overall execution efficiency of image processing tasks.

[0124] According to embodiments of this application, this application also provides an electronic device, a readable storage medium, and a computer program product.

[0125] Figure 6A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0126] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in ROM (Read-Only Memory) 602 or a computer program loaded from storage unit 608 into RAM (Random Access Memory) 603. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 702, and RAM 603 are interconnected via bus 604. I / O (Input / Output) interface 605 is also connected to bus 604.

[0127] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0128] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as image processing methods. For example, in some embodiments, the image processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the image processing methods described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform image processing methods by any other suitable means (e.g., by means of firmware).

[0129] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0130] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0131] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0132] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0133] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0134] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0135] According to an embodiment of this application, this application also provides a computer program product that, when an instruction processor in the computer program product is executed, performs the image processing method proposed in the above embodiments of this application.

[0136] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0137] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An image processing method, comprising: Obtain the task to be processed; wherein, the task to be processed includes the encoded bitstream of the image to be processed; Extract the first encoding information of each first encoding block of the image to be processed from the encoded bitstream; Based on the first encoding information, the region importance weights of each image patch embedding vector of the image to be processed are determined; wherein, the image patch embedding vectors are obtained by segmenting the image to be processed into multiple image regions using a visual model and embedding the image regions; the region importance weights are used to characterize the importance of the image region corresponding to the image patch embedding vector. The visual model is used to perform the processing task on the image to be processed based on the region importance weights, and the task processing result is obtained.

2. The method as described in claim 1, wherein, The step of determining the region importance weights of each image patch embedding vector of the image to be processed based on the first encoding information includes: Based on the first encoding information, determine the set of encoding blocks corresponding to the image block embedding vector from each of the first encoding blocks; The importance weight of the region is determined based on the first coding information of the second coding block in the set of coding blocks.

3. The method as described in claim 2, wherein, Determining the regional importance weight based on the first coding information of the second coding block in the coding block set includes: Based on the first encoding information of the second encoding block, the information content of the second encoding block is determined; The information content of the second coding block in the coding block set is aggregated to obtain the region importance weight.

4. The method of claim 3, wherein, The aggregation of information from the second coded block in the coded block set to obtain the region importance weight includes: The information weight of the second coding block is determined based on the degree of coverage of the image region corresponding to the image block embedding vector to the second coding block; Based on the information weights, the information quantities are weighted to obtain the regional importance weights.

5. The method of claim 2, wherein, The step of determining the set of coding blocks corresponding to the image block embedding vector from each of the first coding blocks based on the first coding information includes: Based on the first encoding information, the encoding block space mapping information of the image to be processed is constructed; wherein, the encoding block space mapping information is used to describe the region division of the image to be processed in the encoding space; Based on the coded block space mapping information, the set of coded blocks is determined from each of the first coded blocks.

6. The method of claim 5, wherein, The step of determining the set of coded blocks from each of the first coded blocks based on the coded block space mapping information includes: Based on the coding block spatial mapping information, determine the first coding block in each first coding block that overlaps with the image region corresponding to the image block embedding vector; The set of coded blocks is determined based on the first coded block that overlaps with the image region corresponding to the image block embedding vector.

7. The method of claim 6, wherein, The step of determining the first coding block in each of the first coding blocks that overlaps with the image region corresponding to the image block embedding vector based on the coding block spatial mapping information includes: Based on the position of the image region in the pixel space of the image to be processed, the coverage range of the image block embedding vector in the pixel space is determined; Based on the coding block space mapping information, determine the first coding block in each of the first coding blocks that overlaps with the coverage area; The first coding block that overlaps with the coverage area is determined as the first coding block that overlaps with the image region corresponding to the image block embedding vector.

8. The method according to any one of claims 1-7, wherein, The first encoding information of each encoding block of the image to be processed extracted from the encoded bitstream includes: The encoded bitstream is parsed to obtain the second encoded information of the image to be processed; Based on the second encoding information, the first encoding information is extracted from the encoded bitstream.

9. The method according to any one of claims 1-7, wherein, The visual model includes an encoder and a decoder. The visual model, based on the region importance weights, is used to perform the processing task on the image to be processed, obtaining the task processing result, including: The encoder is used to perform multi-layer attention calculation on the image embedding sequence obtained by mapping the multiple image regions, and the attention distribution is adjusted in each layer based on the importance weight of the region to obtain the visual feature representation of the image to be processed; The visual feature representation is decoded using the decoder to obtain the task processing result.

10. An image processing apparatus, comprising: An acquisition module is used to acquire a task to be processed; wherein, the task to be processed includes the encoded bitstream of the image to be processed; An extraction module is used to extract the first encoding information of each first encoding block of the image to be processed from the encoded bitstream; The determining module is configured to determine the regional importance weights of each image patch embedding vector of the image to be processed based on the first encoding information; wherein, the image patch embedding vectors are obtained by segmenting the image to be processed into multiple image regions using a visual model and embedding the image regions; the regional importance weights are used to characterize the importance of the image region corresponding to the image patch embedding vector. The task processing module is used to perform the task to be processed on the image to be processed based on the region importance weight using the visual model, and obtain the task processing result.

11. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-9.

13. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-9.

Images