An adaptive decoding method and system for compressed image files

Abstract

The application provides an adaptive decoding method and system for compressed image files, which comprises the following steps: performing a decoding availability determination on a received compressed image file to obtain a decodability mark; if the decodability mark indicates that the compressed image file is decodable, obtaining decoded pixel domain image data, and serializing an original code stream of the compressed image file into a first string; inputting the pixel domain image data and the first string as multi-modal input into a multi-modal large language model to output a first structured text; if the decodability mark indicates that the compressed image file is not decodable, serializing the original code stream of the compressed image file into a second string; inputting the second string as single-modal input into the multi-modal large language model to output a second structured text. The application can adaptively select an input strategy between two understanding paths of decodability and non-decodability, and still realize robust image understanding of the file content in the case of damage of the compressed image file.

Description

Technical Field

[0001] This invention relates to the field of image recognition and understanding technology, specifically to an adaptive decoding method and system for compressed image files. Background Technology

[0002] In the fields of digital image processing and communication, images are commonly stored and transmitted in compressed formats to save storage space and bandwidth. However, under complex operating conditions such as network packet loss, aging storage media, read / write errors, or electromagnetic interference, the binary bitstream of compressed image files is highly susceptible to structural damage. This damage directly violates the syntax structure defined by the compression standard, leading to decoding failures or severe artifacts and distortions in the reconstructed image.

[0003] For corrupted files, existing technologies mainly focus on two directions: First, fault-tolerant decoding technology, aiming to recover as much pixel information as possible from the corrupted bitstream. However, its recovery capability is limited and it heavily relies on prior parsing of specific compression format syntax, making it difficult to apply when the format is unknown or the header is severely damaged. Second, deep learning-based image inpainting or super-resolution technology. For example, the existing published patent CN114298937A proposes using thumbnails in JPEG files for super-resolution and fusing them with the restored original image. However, deep learning-based image inpainting or super-resolution technology is essentially still pixel-domain restoration and enhancement, and its successful implementation requires at least partial decoding of valid pixel data (such as thumbnails). For severely corrupted files that are completely undecodeable, existing technologies lack effective solutions, creating a technological gap.

[0004] Therefore, there is an urgent need to provide an adaptive decoding method and system for compressed image files that can overcome the pre-decoding dependency of image understanding methods, and maintain stable and robust content understanding capabilities when faced with compressed image files of varying degrees of damage, especially when normal decoding is not possible. Summary of the Invention

[0005] In view of this, it is necessary to provide an adaptive decoding method and system for compressed image files to solve the technical problem in the prior art that relies on normal decoding before the compressed image file can be understood, which leads to the inability to understand the image under the premise of normal decoding.

[0006] In a first aspect, to solve the above-mentioned technical problems, the present invention provides an adaptive decoding method for compressed image files, comprising:

[0007] The received compressed image file is decoded and its decodeability is determined to obtain a decodeability tag;

[0008] If the decodeability flag indicates that it is decodeable, the decoded pixel domain image data is obtained, and the original bitstream of the compressed image file is serialized into a first string; the pixel domain image data and the first string are used as multimodal inputs and input into a multimodal large language model to output the first structured text.

[0009] If the decodeability flag indicates that it is not decodeable, the original bitstream of the compressed image file is serialized into a second string; the second string is used as a unimodal input and input into the multimodal large language model to output the second structured text.

[0010] The multimodal large language model shares the same set of model weights to process the multimodal input and the unimodal input.

[0011] In one possible implementation, serializing the original bitstream of the compressed image file into a first string includes:

[0012] The compressed image file is read in binary mode to obtain a byte array;

[0013] Each byte in the byte array is converted into two hexadecimal characters to form the initial string;

[0014] The initial string is normalized to obtain the first string.

[0015] In one possible implementation, the step of determining the decodeability of the received compressed image file to obtain a decodeability tag includes:

[0016] The compressed image file is decoded using a standard image decoder. If decoding is successful and a pixel image with a valid size is obtained, the decodeability flag indicates that it is decodeable. If decoding fails or a pixel image with a valid size is not obtained, the decodeability flag indicates that it is not decodeable.

[0017] In one possible implementation, the method further includes:

[0018] Obtain the original image file and perform damage processing on the original image file based on a preset damage model to obtain a damaged image file. Construct training samples based on the sample labels of the original image file and the damaged image file.

[0019] Based on the training samples, the pre-trained multimodal large language model is fine-tuned to obtain the multimodal large language model.

[0020] In one possible implementation, the corruption model simulates bitstream corruption based on bit-flipping corruption and byte-deletion corruption.

[0021] In one possible implementation, when the corruption model performs corruption processing on the original image file, the corruption processing location is uniformly and randomly selected within the entire byte sequence range of the original image file.

[0022] In one possible implementation, the structure of the multimodal large language model includes a visual encoder, a text encoder, a cross-modal connection module, and a language backbone network;

[0023] The visual encoder is used to extract features from the pixel domain image data to obtain visual features;

[0024] The text encoder is used to extract features from the first string or the second string to obtain sequence features;

[0025] The cross-modal connection module is used to receive and fuse the visual features and the sequence features to generate a cross-modal representation;

[0026] The language backbone network module is used to generate the first structured text based on the cross-modal representation, or to generate the second structured text based on the sequence features.

[0027] In one possible implementation, during the parameter fine-tuning training of the multimodal large language model, the parameters of the visual encoder are frozen, while the parameters of the text encoder, the cross-modal connection module, and the language backbone network module can be trained, and trainable continuous cue vectors are added to the input sequence of the multimodal large language model.

[0028] In one possible implementation, the parameter fine-tuning training process of the multimodal large language model further includes: introducing an adaptation mechanism into the self-attention layer and feedforward connection layer of the language backbone network module and the cross-modal connection module to form a trainable parameter update path, wherein the adaptation mechanism is:

[0029] At least one of the following methods is employed: introducing a low-rank matrix into the weight matrix of the self-attention layer or the feedforward connection layer; inserting an adapter module with a bottleneck structure into the computation path of the self-attention layer or the feedforward connection layer; or adding a trainable prefix vector before the key or value vector of the self-attention layer.

[0030] Secondly, the present invention also provides an adaptive decoding system for compressed image files, comprising:

[0031] The decoding availability determination unit is used to determine the decoding availability of the received compressed image file and obtain a decodeability tag;

[0032] The first decoding unit is configured to, when the decodeability flag indicates that it is decodeable, acquire the decoded pixel domain image data, serialize the original bitstream of the compressed image file into a first string, and input the pixel domain image data and the first string as multimodal inputs into a multimodal large language model, and output the first structured text.

[0033] The second decoding unit is used to serialize the original bitstream of the compressed image file into a second string when the decodeability flag indicates that it is undecodeable; and to input the second string as a single-modal input into the multimodal large language model to output the second structured text.

[0034] The multimodal large language model shares the same set of model weights to process the multimodal input and the unimodal input.

[0035] The beneficial effects of this invention are as follows: The adaptive decoding method for compressed image files provided by this invention comprehensively utilizes pixel-domain image data and original bitstream sequence information for decodeable compressed image files; for undecodeable compressed image files, it directly performs image understanding based on the original bitstream sequence. Compared with existing technologies that cannot perform image understanding on undecodeable compressed image files, this invention can decode / understand compressed image files under all conditions, from complete and slightly damaged to severely damaged and even undecodeable, significantly improving the robustness of image understanding.

[0036] Furthermore, this invention serializes the original bitstream of the compressed image file into a string, leveraging the powerful sequence modeling and understanding capabilities of a multimodal large language model. This enables the multimodal large language model to directly learn the semantic representation of the image content at the bitstream level. Even when the compressed image file is undecodeable, content inference can still be performed by analyzing the remaining bitstream patterns, improving the accurate understanding of compressed image files in undecodeable situations.

[0037] Furthermore, by setting the multimodal large language model to share the same set of model weights to process the multimodal input and the unimodal input, a unified multimodal large language model is realized that can perform image understanding on compressed image files in both decodeable and undecodeable cases, without the need to maintain two independent models for different cases. This simplifies the complexity and overhead of the overall architecture, while ensuring the consistency of the output format, which is convenient for subsequent processing. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 A schematic flowchart of an embodiment of the adaptive decoding method for compressed image files provided by the present invention;

[0040] Figure 2 A schematic flowchart illustrating an embodiment of how to generate the first string provided by the present invention;

[0041] Figure 3 A schematic diagram illustrating an embodiment of the training process for obtaining a multimodal large language model provided by the present invention;

[0042] Figure 4 A schematic diagram illustrating an embodiment of the bit flipping and byte deletion operations provided by the present invention;

[0043] Figure 5 A schematic diagram of the structure of an embodiment of the multimodal large language model provided by the present invention;

[0044] Figure 6 A schematic diagram illustrating an embodiment of the complete process of the adaptive decoding method for compressed image files provided by the present invention;

[0045] Figure 7 This is a schematic diagram of an embodiment of the adaptive decoding system for compressed image files provided by the present invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0047] It should be understood that the illustrative drawings are not drawn to scale. The flowcharts used in this invention illustrate operations implemented according to some embodiments of the invention. It should be understood that the operations in the flowcharts may be implemented out of order, and steps without logical contextual relationships may be reversed or performed simultaneously. Furthermore, those skilled in the art, guided by the content of this invention, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor systems and / or microcontroller systems.

[0048] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0049] This invention provides an adaptive decoding method and system for compressed image files, which will be described below.

[0050] Figure 1 A schematic flowchart of an embodiment of the adaptive decoding method for compressed image files provided by the present invention is shown below. Figure 1 As shown, the adaptive decoding method for compressed image files includes:

[0051] S101. Decode the received compressed image file to determine its decodeability and obtain a decodeability tag.

[0052] Specifically, the decodeability flag can be 0 or 1. When it is 0, it means that it is not decodeable, and when it is 1, it means that it is decodeable.

[0053] S102. If the decodability flag indicates that it is decodable, then obtain the decoded pixel domain image data and serialize the original bitstream of the compressed image file into the first string; use the pixel domain image data and the first string as multimodal inputs to input into the multimodal large language model and output the first structured text.

[0054] Specifically, multimodal input It can be represented as:

[0055]

[0056] In the formula, For pixel and image data; The first string; It is a cross-modal fusion operator.

[0057] S103. If the decodability flag indicates that it is not decodable, the original bitstream of the compressed image file is serialized into a second string; the second string is used as a single-modal input and input into the multimodal large language model to output the second structured text.

[0058] Specifically, the single-modal input is:

[0059] .

[0060] Among them, the multimodal large language model shares the same set of model weights to handle multimodal input and unimodal input.

[0061] It should be understood that the adaptive decoding method for compressed image files in this embodiment of the invention can be implemented in any device based on the adaptive decoding method for compressed image files, such as an image understanding device or an image analysis device. Specifically, the adaptive decoding method for compressed image files is stored in the aforementioned device as a pre-programmed program. When the device starts, the program is invoked, and the adaptive decoding method for compressed image files is implemented.

[0062] Compared with existing technologies, the adaptive decoding method for compressed image files provided in this invention comprehensively utilizes pixel-domain image data and original bitstream sequence information for decodeable compressed image files; for undecodeable compressed image files, it directly performs image understanding based on the original bitstream sequence. Compared with existing technical solutions that cannot perform image understanding on undecodeable compressed image files, this invention can decode / understand compressed image files under all conditions, from complete to slightly damaged to severely damaged and even undecodeable, significantly improving the robustness of image understanding.

[0063] Furthermore, in this embodiment of the invention, the original bitstream of the compressed image file is serialized into a string. Leveraging the powerful sequence modeling and understanding capabilities of the multimodal large language model, the model can directly learn the semantic representation of the image content at the bitstream level. Even when the compressed image file is undecodeable, content inference can still be performed by analyzing the remaining bitstream patterns, improving the accurate understanding of compressed image files in undecodeable situations.

[0064] Furthermore, by setting the multimodal large language model to share the same set of model weights to process the multimodal input and the unimodal input, a unified multimodal large language model is realized that can perform image understanding on compressed image files in both decodeable and undecodeable cases, without the need to maintain two independent models for different cases. This simplifies the complexity and overhead of the overall architecture, while ensuring the consistency of the output format, which is convenient for subsequent processing.

[0065] To facilitate the processing of multimodal large language models, in some embodiments of the present invention, such as Figure 2 As shown, the original bitstream of the compressed image file is serialized into a first string, including:

[0066] S201. Read the compressed image file in binary mode to obtain a byte array;

[0067] S202. Convert each byte in the byte array into two hexadecimal characters to form the initial string;

[0068] S203. Perform length normalization on the initial string to obtain the first string.

[0069] Specifically, the length normalization process is as follows: if the length of the initial string is less than the normalized length, then "0" characters are padded to the right of the initial string until the length of the initial string equals the normalized length. If the length of the initial string is greater than the normalized length, then it is truncated according to the rules to ensure that the length of the initial string is equal to the normalized length.

[0070] The truncation rule can be: keep the first L characters of the initial string and discard all characters after L, where L is the normalized length.

[0071] It should be understood that the principle and process of serializing the original bitstream of a compressed image file into a second string are exactly the same as those of serializing the original bitstream of a compressed image file into a first string. Please refer to steps S201-S203, which will not be elaborated here.

[0072] In some embodiments of the present invention, step S101 includes:

[0073] The compressed image file is decoded based on a standard image decoder. If the decoding is successful and a pixel image with a valid size is obtained, the decodeability flag indicates that it is decodeable. If the decoding fails or a pixel image with a valid size is not obtained, the decodeability flag indicates that it is not decodeable.

[0074] Specifically, successful decoding means that the image can be obtained through decoding, while decoding failure means that the image cannot be obtained through decoding.

[0075] The effective size can be determined based on pre-set judgment rules, and no specific limitations are made here.

[0076] This invention determines whether a decodability flag is decodable or not based solely on whether decoding was successful. Instead, it considers both successful decoding and the acquisition of a pixel image with a valid size. This avoids situations where an image is obtained but is completely black or where decoding errors occur, further ensuring the accuracy of the decodability flag. This, in turn, ensures the accuracy of subsequent multimodal, multilingual model image understanding path selection, improving the decoding / understanding accuracy of compressed image files.

[0077] It should be understood that, in order to improve the accuracy of multimodal large language models in image decoding / understanding scenarios, in some embodiments of the present invention, such as... Figure 3 As shown, before step S102, the adaptive decoding method for compressed image files further includes:

[0078] S301. Obtain the original image file and perform damage processing on the original image file based on the preset damage model to obtain a damaged image file. Construct training samples based on the sample labels of the original image file and the damaged image file.

[0079] The original file images can be selected from media data and their compressed representations obtained from multiple channels, including publicly released, commercially licensed, self-collected, and synthesized / simulated data.

[0080] S302. Based on the training samples, fine-tune the parameters of the pre-trained multimodal large language model to obtain the multimodal large language model.

[0081] Specifically, the loss function used during the training of the multimodal large language model is the autoregressive supervised loss function. Its training optimization objective is to maximize the likelihood of the target response token by token, without the need to introduce an additional multitasking head.

[0082] It should be noted that by changing the input instructions, the same model can complete different tasks, and the output layer is always a projection of the language model's vocabulary, eliminating the need for multiple task heads. For example, when the task type is different, such as image classification or image description, different text will be input, rather than adding different detection heads as in traditional methods. In other words, this embodiment of the invention classifies all tasks as text generation, improving its robustness to understanding.

[0083] Among them, the pre-trained multimodal large language model refers to a large language model trained on a general dataset, which is a general and mature large language model.

[0084] The embodiments of the present invention train a pre-trained multimodal large language model based on the constructed training samples, which makes the trained multimodal large language model more suitable for the field of compressed file decoding / understanding, thereby ensuring the accuracy of image decoding / understanding of compressed image files.

[0085] Furthermore, in this embodiment of the invention, only the parameters of the pre-trained multimodal large language model are fine-tuned, and only a portion of the parameters are fine-tuned, rather than all the parameters of the pre-trained multimodal large language model. This eliminates the technical problems of large computational and storage overhead, catastrophic forgetting risk, and difficulty in multi-task adaptation caused by fine-tuning all the parameters. It ensures that the multimodal large language model does not lose its original general knowledge while acquiring new capabilities, which facilitates rapid deployment and iteration.

[0086] To simulate random and unpredictable file corruption scenarios that occur in real transmission and storage links of compressed image files, in some embodiments of the present invention, the corruption model simulates bitstream corruption based on bit-flipping corruption and byte-deletion corruption methods.

[0087] It should be noted that, to ensure reproducibility, the random seed and damage injection parameters during the generation process of the damaged image file are recorded.

[0088] Specifically, bit-flipping corruption refers to a model that randomly inverts a portion of the bits in the original image file's bitstream, while scale-flipping is the finest-grained form of perturbation. Let the byte array of the original image file's bitstream be:

[0089]

[0090] In the formula, N is the length in bytes. If the damage ratio is... ρ The expected number of flipped bits for:

[0091]

[0092] In the formula, This is the floor operator.

[0093] For each flip operation, first sample byte position i uniformly in the interval [1, N], then sample bit position k uniformly in the interval [0, 7], and finally execute:

[0094]

[0095] In the formula, This is the original image file bitstream; This is a corrupted image file after bit flipping; For bitwise XOR; To generate a mask at the k-th bit, repeat the process. This will produce a corrupted image file.

[0096] In specific embodiments of the present invention, such as Figure 4 As shown, Figure 4 The left side shows the bit-flipping operation. Bit-flipping corruption can effectively simulate damage caused by hardware storage errors, random noise in the transmission channel, and other forms of corruption.

[0097] In a specific implementation of this invention, the byte deletion corruption method refers to randomly deleting a continuous segment of bytes from the original image file bitstream, which is a medium-granularity structural corruption. Let the byte array of the original image file bitstream be:

[0098]

[0099] If the damage ratio is ρ Then delete the number of bytes for:

[0100]

[0101] Then in the interval [1, N] n drop The sampling starts at position s and the deletion range is [s, s+n]. drop 1). The corrupted image file is:

[0102]

[0103] In the formula, This is a corrupted image file after byte deletion.

[0104] In specific embodiments of the present invention, such as Figure 4 As shown, Figure 4 The image on the right shows a byte deletion operation. Byte deletion corruption can effectively simulate damage forms such as bad blocks in storage media and sudden packet loss during transmission.

[0105] This invention improves the adaptability of corrupted image files to real-world scenarios by setting two different types of corruption methods: bit flip corruption and byte deletion corruption. This enhances the accuracy of compressed image understanding in real-world scenarios.

[0106] Since the location of damage is completely random in real-world applications, in order to be realistic, in a specific embodiment of the present invention, when the damage model performs damage processing on the original image file, the damage processing location is uniformly and randomly selected within the entire byte sequence of the original image file, without distinguishing between the header, parameter area, or entropy coding segment.

[0107] This invention simulates unpredictable and indiscriminate damage patterns in the real world by uniformly and randomly selecting damaged locations throughout the entire byte sequence of the original image file. This avoids the problem of learning a model that is not realistic due to injecting damage only into non-critical areas. This allows the trained multimodal large language model to learn more fundamental and robust bitstream features, rather than relying on assumptions about the integrity of a specific file structure, thus exhibiting stronger generalization ability in practical applications.

[0108] In some embodiments of the present invention, such as Figure 5 As shown, the structure of a multimodal large language model includes a visual encoder, a text encoder, a cross-modal connectivity module, and a language backbone network.

[0109] Visual encoders are used to extract features from pixel-domain image data to obtain visual features;

[0110] A text encoder is used to extract features from a first or second string to obtain sequence features;

[0111] The cross-modal connectivity module is used to receive and fuse visual features and sequence features to generate cross-modal representations;

[0112] The language backbone network module is used to generate the first structured text based on cross-modal representations, or to generate the second structured text based on sequence features.

[0113] Based on the above-mentioned multimodal large language model, in order to further reduce its training load, in the specific implementation of this invention, during the parameter fine-tuning training of the multimodal large language model, the parameters of the visual encoder are frozen, while the parameters of the text encoder, cross-modal connection module and language backbone network module can be trained, and trainable continuous cue vectors are added to the input sequence of the multimodal large language model.

[0114] Since the visual encoder has been pre-trained on billions of high-quality image-text pairs, it is already an extremely powerful general-purpose visual feature extractor. The text encoder, cross-modal connectivity module, and language backbone network module are structures more closely related to the specific domain of this invention; therefore, the training load can be reduced by freezing the parameters of the text encoder.

[0115] Furthermore, this embodiment of the invention provides precise task guidance and adaptive input conditions by adding trainable continuous cue vectors to the input sequence of the multimodal large language model, enabling the multimodal large language model to efficiently process inputs from different modalities with a unified architecture. This design makes the multimodal large language model in this embodiment easier to train, deploy, and scale, and possesses high industrial application value.

[0116] In addition to freezing some parameters as described above, to further reduce the number of training parameters for the multimodal large language model, in a specific embodiment of the present invention, the parameter fine-tuning training process of the multimodal large language model also includes: introducing an adaptation mechanism in the self-attention layer and feedforward connection layer of the language backbone network module and the cross-modal connection module to form a trainable parameter update path. The adaptation mechanism is as follows:

[0117] Introducing a low-rank matrix into the weight matrix of the self-attention layer or feedforward connection layer, inserting an adapter module with a bottleneck structure into the computation path of the self-attention layer or feedforward connection layer, or adding a trainable prefix vector before the key or value vector of the self-attention layer are at least one of the following:

[0118] In other words, by using efficient parameter fine-tuning techniques such as LoRA, Adapter, and Prefix Tuning, the number of training parameters can be further reduced, significantly reducing training and storage overhead while ensuring performance.

[0119] In a specific embodiment of the present invention, the complete process of the adaptive decoding method for compressed image files proposed in the present invention is as follows: Figure 6 As shown, the original image file is first corrupted based on the corruption model to obtain a corrupted image file. Then, the corrupted image file is decoded based on a standard decoder. If it is determined to be decodable, a fixed-length file segment is extracted from the file while obtaining a normal-sized pixel image. Both modalities are input into a multimodal large language model for joint modeling and inference, and the image content discrimination result is output. If it is determined to be undecodable, only the file segment is used as input to complete the single-modal sequence analysis and output the understanding result.

[0120] In summary, the adaptive decoding method for compressed image files proposed in this invention can adaptively select an input strategy between two understanding paths: decodable and undecodable. Even when compressed image files are corrupted (including bitstream-level corruption), robust image understanding of the file content is still achieved. Furthermore, this invention constructs a corrupted file image dataset by designing a corruption model. This dataset simulates image files in real-world corruption scenarios. Vertical domain optimization techniques are used to fine-tune the multimodal large language model, significantly enhancing its understanding capability in corrupted image scenarios while minimizing changes to the original multimodal large model structure and parameters.

[0121] To better implement the adaptive decoding method for compressed image files in this embodiment of the invention, based on the adaptive decoding method for compressed image files, this embodiment of the invention also provides an adaptive decoding system for compressed image files, such as... Figure 7 As shown, the adaptive decoding system 700 for compressed image files includes:

[0122] Decoding availability determination unit 701 is used to determine the decoding availability of the received compressed image file and obtain a decodeability tag;

[0123] The first decoding unit 702 is used to acquire the decoded pixel domain image data when the decodeability flag indicates that it is decodeable, and serialize the original bitstream of the compressed image file into a first string; and input the pixel domain image data and the first string as multimodal inputs into the multimodal large language model, and output the first structured text.

[0124] The second decoding unit 703 is used to serialize the original bitstream of the compressed image file into a second string when the decodeability flag indicates that it is undecodeable; and to input the second string as a single-modal input to a multimodal large language model and output the second structured text.

[0125] Among them, the multimodal large language model shares the same set of model weights to handle multimodal input and unimodal input.

[0126] The adaptive decoding system 700 for compressed image files provided in the above embodiments can implement the technical solutions described in the above embodiments of the adaptive decoding method for compressed image files. The specific implementation principles of each module or unit can be found in the corresponding content in the above embodiments of the adaptive decoding method for compressed image files, and will not be repeated here.

[0127] The above provides a detailed description of the adaptive decoding method and system for compressed image files provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will know that there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. An adaptive decoding method of a compressed image file, characterized by, The method comprises the following steps: decoding availability determination is performed on the received compressed image file to obtain a decodability mark; if the decodability mark indicates decodable, obtaining decoded pixel domain image data, and serializing the original code stream of the compressed image file into a first string; inputting the pixel domain image data and the first string as multi-modal input into a multi-modal large language model to output a first structured text; if the decodability mark indicates undecodable, serializing the original code stream of the compressed image file into a second string; inputting the second string as single-modal input into the multi-modal large language model to output a second structured text; wherein the multi-modal large language model shares the same set of model weights to process the multi-modal input and the single-modal input.

2. The adaptive decoding method of a compressed image file according to claim 1, characterized by, The method for serializing the original code stream of the compressed image file into the first string comprises the following steps: reading the compressed image file in binary mode to obtain a byte array; converting each byte in the byte array into two hexadecimal characters to form an initial string; performing length normalization processing on the initial string to obtain the first string.

3. The adaptive decoding method of a compressed image file according to claim 1, characterized by, The method for performing decoding availability determination on the received compressed image file to obtain a decodability mark comprises the following steps: decoding the compressed image file based on a standard image decoder; if the decoding is successful and a pixel image with a valid size is obtained, the decodability mark indicates decodable; if the decoding fails or a pixel image with a valid size is not obtained, the decodability mark indicates undecodable.

4. The adaptive decoding method of a compressed image file according to claim 1, characterized by, The method further comprises the following steps: obtaining an original image file, and performing damage processing on the original image file based on a preset damage model to obtain a damaged image file, and constructing a training sample based on a sample label of the original image file and the damaged image file; performing parameter fine-tuning training on a pre-trained multi-modal large language model based on the training sample to obtain the multi-modal large language model.

5. The adaptive decoding method of a compressed image file according to claim 4, characterized by, The damage model simulates code stream damage based on a bit flip damage mode and a byte deletion damage mode.

6. The adaptive decoding method of a compressed image file according to claim 4, characterized by, When performing damage processing on the original image file, the damage processing position is uniformly and randomly selected within the entire byte sequence range of the original image file.

7. The adaptive decoding method of a compressed image file according to claim 4, characterized by, The structure of the multi-modal large language model comprises a visual encoder, a text encoder, a cross-modal connection module, and a language backbone network; the visual encoder is used for feature extraction on the pixel domain image data to obtain visual features; the text encoder is used for feature extraction on the first string or the second string to obtain sequence features; the cross-modal connection module is used for receiving and fusing the visual features and the sequence features to generate cross-modal representations; the language backbone network module is used for generating the first structured text based on the cross-modal representations, or generating the second structured text based on the sequence features.

8. The adaptive decoding method of a compressed image file according to claim 7, characterized by, In the parameter fine-tuning training process of the multi-modal large language model, the parameters of the visual encoder are frozen, and the parameters of the text encoder, the cross-modal connection module, and the language backbone network module can be trained, and a trainable continuous prompt vector is added in the input sequence of the multi-modal large language model.

9. The adaptive decoding method of a compressed image file according to claim 8, characterized by, In the parameter fine-tuning training process of the multi-modal large language model, an adaptive mechanism is introduced in the self-attention layer and the feedforward connection layer of the language backbone network module and the cross-modal connection module to form a trainable parameter update path, and the adaptive mechanism is at least one of the following: At least one of the following is introduced in the weight matrix of the self-attention layer or the feedforward connection layer: a low-rank matrix, an adapter module with a bottleneck structure inserted in the calculation path of the self-attention layer or the feedforward connection layer, and a trainable prefix vector added before the key or value vector of the self-attention layer.

10. An adaptive decoding system of a compressed image file, characterized by, Comprise: A decoding availability determination unit for performing decoding availability determination on the received compressed image file to obtain a decodability flag; A first decoding unit for obtaining decoded pixel domain image data when the decodability flag indicates that the compressed image file is decodable, and serializing the original code stream of the compressed image file into a first string; inputting the pixel domain image data and the first string as multi-modal input into a multi-modal large language model to output a first structured text; A second decoding unit for serializing the original code stream of the compressed image file into a second string when the decodability flag indicates that the compressed image file is not decodable; inputting the second string as single-modal input into the multi-modal large language model to output a second structured text; Wherein, the multi-modal large language model shares the same set of model weights to process the multi-modal input and the single-modal input.

Images