Image semantic communication method and system based on multi-modal large language model

Abstract

The invention provides an image semantic communication method and system based on a multi-modal large language model (MLLMs), and aims to improve the semantic accuracy and system robustness of image transmission. The method comprises the following steps: extracting semantic features from an image source, detecting through an expert AI model, processing an OOD problem in combination with a general AI model, optimizing recognition precision by using a Bayesian method, encoding the features and transmitting the encoded features through a wireless channel. And after decoding by a receiving end, reconstructing the image by using a generation-evaluation framework based on the MLLMs and performing iterative optimization so as to ensure accurate reconstruction of semantic information of the image. According to the method, the OOD data processing capability is particularly enhanced, the image reconstruction quality is improved through the generation-evaluation framework, the limitation of a traditional model on new data distribution is solved, and the reliability of the system and the accuracy of image reconstruction are enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to semantic communication technology, in particular to an image semantic communication method and system based on a multi-modal large language model. BACKGROUND

[0002] With the development of 5G and even future 6G communication technology, semantic communication systems are becoming a research hotspot as an important part of the next generation of wireless communication technology. Traditional communication systems mainly focus on the reliable transmission of signals, while semantic communication systems go further. They not only focus on the accurate transmission of information, but also pay attention to the semantic integrity of information. This means not only ensuring the lossless arrival of data packets, but also ensuring that the receiving end can correctly understand the intention of the sending end. In the field of image semantic communication, current technologies are mainly based on deep learning models, especially convolutional neural networks (CNN) and transformer architectures. These models extract features from images for compression and decompression at the receiving end to achieve efficient transmission of images while preserving the semantic information of the images as much as possible. These technologies have made significant progress, enabling efficient transmission of image data over wireless networks while preserving the semantic information of the images as much as possible.

[0003] Despite these advances, there are still several key issues:

[0004] 1. Out-of-Distribution (OOD) problem: Machine learning models perform poorly when faced with data they have never seen before. This is because models are usually trained based on specific data sets, and once new data distributions appear, the generalization ability of the model is limited. For example, in image recognition for autonomous vehicles, if the model has not encountered a specific type of obstacle, it may not be able to correctly identify it.

[0005] 2. Unstable image reconstruction performance: For some image semantic communication systems, the receiving end uses a generative model to reconstruct images. Due to the inherent randomness and instability of generative models, this will result in unstable image reconstruction quality.

[0006] It should be noted that the information disclosed in the above background section is only for understanding the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0007] The main purpose of the present application is to solve the problems existing in the background art, and to provide an image semantic communication method and system based on a multi-modal large language model.

[0008] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0009] In a first aspect of the present application, a multimodal large language model-based image semantic communication method comprises the following steps:

[0010] Step 1, input image processing: obtaining image data from an image source and performing preliminary semantic feature extraction;

[0011] Step 2, OOD detection and processing: identifying image features and calculating confidence through an expert AI model, if the confidence is lower than a preset threshold, triggering a general AI model to re-evaluate the features to identify and handle OOD situations;

[0012] Step 3, Bayesian optimization: using Bayesian methods to adjust the probability distribution output by the general AI model based on context information to improve the identification accuracy of the target class;

[0013] Step 4, semantic feature encoding: based on the results of OOD detection and processing and Bayesian optimization, the obtained semantic feature vector is encoded into a form suitable for wireless channel transmission;

[0014] Step 5, signal transmission: transmitting the encoded signal through a wireless channel;

[0015] Step 6, signal reception and decoding: the receiving end receives the signal and decodes it to recover the semantic feature vector;

[0016] Step 7, image reconstruction and generation-evaluation framework iterative optimization: based on the decoded semantic feature vector, reconstruct the image through the generation-evaluation framework based on the multimodal large language model MLLMs and perform iterative optimization to achieve accurate image reconstruction;

[0017] Step 8: output the final image.

[0018] Further:

[0019] In step 2, the triggering of the general AI model to re-evaluate the features to identify and handle OOD situations specifically includes:

[0020] Detecting features with confidence lower than the threshold, triggering the general AI model to intervene;

[0021] Crop the low-confidence object area framed by the expert AI model to form a sub-image;

[0022] Submit the sub-image to the general AI model with a hint for classification;

[0023] From the class and its confidence list returned from the general AI model, select the highest confidence class as the identification result;

[0024] Correct the feature to the identification result of the general AI;

[0025] Integrate the final recognition results of all objects to form a complete image semantic information vector.

[0026] In step 3, the probability distribution output by the general AI model is adjusted according to the context information using the Bayesian method, specifically including:

[0027] For image features with low confidence, a preliminary recognition result is generated by the general AI model;

[0028] Based on the image context information, the preliminary recognition result is adjusted by applying the Bayesian theorem to adjust the probability distribution of the preliminary recognition result;

[0029] The recognition result of the general AI model is corrected through the posterior probability optimized by the Bayesian method to improve the accuracy of the recognition.

[0030] In step 4, the semantic feature vector is encoded into a form suitable for wireless channel transmission, specifically including:

[0031] Obtain the semantic feature vector corresponding to the whole image;

[0032] The channel encoder converts the semantic feature vector into a form suitable for wireless channel transmission by using an appropriate encoding scheme.

[0033] In step 7, under the generation-evaluation framework, collaboration is carried out between the multi-modal large language models MLLMs, one model is responsible for generating images, and the other model plays the role of evaluator, evaluating the quality of the generated images and providing feedback in order to continuously optimize the generated results.

[0034] In step 7, the image is reconstructed and iteratively optimized through the generation-evaluation framework based on the multi-modal large language models MLLMs, specifically including:

[0035] The generated image is compared with the original semantic description by using the evaluation model to identify semantic errors in the image;

[0036] The evaluation model provides evaluation feedback according to the semantic errors;

[0037] The generation model regenerates the image based on the evaluation feedback;

[0038] In this way, the reconstructed image is iteratively optimized until the generated image meets the accuracy requirements of the semantic description or reaches the preset maximum number of iterations.

[0039] In the second aspect of the present application, an image semantic communication system based on multi-modal large language models includes a sending end and a receiving end,

[0040] The sending end includes:

[0041] An input image processor is configured to acquire image data from an image source and perform preliminary semantic feature extraction;

[0042] An OOD detector is configured to identify image features by an expert AI model and calculate a confidence level, and if the confidence level is lower than a preset threshold, trigger a general AI model to re-evaluate the features to identify and handle OOD cases;

[0043] A Bayesian optimizer is configured to adjust the probability distribution output by the general AI model according to the context information to improve the identification accuracy of the target class;

[0044] A semantic feature encoder is configured to encode the obtained semantic feature vector into a form suitable for wireless channel transmission based on the results of OOD detection and processing and Bayesian optimization;

[0045] A signal transmission unit is configured to transmit the encoded signal through a wireless channel;

[0046] The receiving end comprises:

[0047] A signal receiver is configured to receive the signal transmitted through the wireless channel;

[0048] A signal decoder is configured to decode the received signal to recover the semantic feature vector;

[0049] An image reconstruction and generation-evaluation framework iterative optimizer is configured to reconstruct the image and perform iterative optimization through a generation-evaluation framework based on multi-modal large language models (MLLMs) according to the decoded semantic feature vector, to achieve accurate image reconstruction;

[0050] An output unit is configured to output the final reconstructed image.

[0051] In a third aspect of the present application, a computer-readable storage medium stores a computer program, which, when executed by a processor, implements the multi-modal large language model-based image semantic communication method.

[0052] The present application has the following advantages:

[0053] The present application provides a multi-modal large language model-based image semantic communication method and system, which innovatively introduces multi-modal large language models (MLLMs) to significantly improve the performance of the image semantic communication system. Through the method of the present application, the system not only ensures accurate transmission of information when processing image data, but also pays attention to the semantic integrity of the information, so that the receiving end can correctly understand the intention of the sending end. Especially when facing OOD (out-of-distribution) data, the powerful generalization ability of MLLMs effectively solves the recognition problem of traditional deep learning models on new data distribution, enhancing the robustness of the system.

[0054] Meanwhile, the present application optimizes the image reconstruction process and improves the quality and accuracy of image reconstruction by introducing the "generation-evaluation" framework. Under this framework, one MLLM is responsible for generating images, while the other serves as an evaluator to assess the quality of the generated images and provide feedback. This iterative optimization process continuously improves the semantic accuracy of the images until the generated images fully meet the semantic description.

[0055] In addition, the present application improves the accuracy and robustness of semantic encoding by combining the advantages of expert AI models and general AI models. The use of Bayesian optimization further enhances the performance of the system, which optimizes the reasoning process of MLLMs by considering contextual information. The present application makes the system more accurate in expressing semantic information of images.

[0056] The combination of the above innovative means in the present application not only overcomes the limitations of existing technologies, but also provides a more advanced and reliable solution for future semantic communication technology, laying a solid foundation for the development of wireless communication technology.

[0057] Other benefits of the embodiments of the present application will be further described below. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 The overall workflow diagram of the image semantic communication method based on a multi-modal large language model of the embodiments of the present application.

[0059] Figure 2 The workflow diagram of the OOD detection and processing and Bayesian optimization steps of the embodiments of the present application. DETAILED DESCRIPTION

[0060] The embodiments of the present application will be described in detail below. It should be emphasized that the following description is merely exemplary and is not intended to limit the scope of the present application and its applications.

[0061] In addition, the terms "first", "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly specified.

[0062] The application provides a kind of image semantic communication method and system based on multimodal large language model, to solve the problems existing in the image semantic communication technology mentioned above, Specifically: improve the robustness of system: in the face of OOD input, by introducing multimodal large language model (MLLMs) to improve the performance of system, through the powerful generalization ability of MLLMs, system can maintain high performance. MLLMs can better understand and process diverse inputs through the learning of multimodal data, even if these inputs are different from training data. Improve the quality of image reconstruction: by introducing "generation-evaluation" framework, the quality of image reconstruction is improved by cooperation between multiple MLLMs. Under this framework, one model is responsible for generating images, while another model plays the role of evaluator, evaluating the quality of generated images and providing feedback to continuously optimize the results of generation.

[0063] Referring to Figure 1 and Figure 2 , the application embodiment provides a kind of image semantic communication method based on multimodal large language model, including the following steps:

[0064] Step 1, input image processing: obtain image data from image source, and perform preliminary semantic feature extraction;

[0065] Step 2, OOD detection and processing: identify image features and calculate confidence through expert AI model, if the confidence is lower than the preset threshold, trigger general AI model to re-evaluate the features to identify and handle OOD situation;

[0066] Step 3, Bayesian optimization: use Bayesian method to adjust the probability distribution output by general AI model according to context information to improve the identification accuracy of target category;

[0067] Step 4, semantic feature encoding: based on the results of OOD detection and processing and Bayesian optimization, the obtained semantic feature vector is encoded into a form suitable for wireless channel transmission;

[0068] Step 5, signal transmission: transmit the encoded signal through wireless channel;

[0069] Step 6, signal reception and decoding: the receiving end receives the signal and decodes to recover the semantic feature vector;

[0070] Step 7, image reconstruction and generation-evaluation framework iterative optimization: according to the semantic feature vector obtained by decoding, the image is reconstructed and iteratively optimized through the generation-evaluation framework based on multimodal large language model MLLMs, to realize accurate image reconstruction;

[0071] Step 8: output the final image.

[0072] In a preferred embodiment, step 2, which involves triggering the general AI model to re-evaluate features to identify and handle OOD situations, specifically includes: detecting features with confidence levels below a threshold and triggering the intervention of the general AI model; cropping the low-confidence object region defined by the expert AI model to form a sub-image; submitting the sub-image to the general AI model with a prompt for classification; selecting the highest-confidence category from the list of categories and their confidence levels returned by the general AI model as the recognition result; correcting the feature to the recognition result of the general AI; and integrating the final recognition results of all objects to form a complete image semantic information vector.

[0073] In a preferred embodiment, step 3, which involves adjusting the probability distribution of the general AI model output based on contextual information using the Bayesian method, specifically includes: generating preliminary recognition results for low-confidence image features using the general AI model; adjusting the probability distribution of the preliminary recognition results based on the image context information using Bayes' theorem; and correcting the recognition results of the general AI model using the Bayesian-optimized posterior probability to improve the accuracy of recognition.

[0074] In a preferred embodiment, in step 7, within the generation-evaluation framework, multimodal large language models (MLLMs) collaborate. One model generates the image, while the other acts as an evaluator, assessing the quality of the generated image and providing feedback to continuously optimize the generated result. The process of reconstructing the image and iteratively optimizing it using the generation-evaluation framework based on MLLMs specifically includes: comparing the generated image with the original semantic description using the evaluation model to identify semantic errors in the image; the evaluation model providing evaluation feedback based on the semantic errors; the generation model regenerating the image based on the evaluation feedback; and thus, iteratively optimizing the reconstructed image until the generated image meets the accuracy requirements of the semantic description or reaches a preset maximum number of iterations.

[0075] This invention also provides an image semantic communication system based on a multimodal large language model, including a sender and a receiver.

[0076] The sending end includes:

[0077] An input image processor is used to acquire image data from an image source and perform preliminary semantic feature extraction.

[0078] The OOD detector is used to identify image features and calculate confidence scores through an expert AI model. If the confidence score is lower than a preset threshold, it triggers a general AI model to re-evaluate the features in order to identify and process OOD situations.

[0079] A Bayesian optimizer is used to adjust the probability distribution of the output of a general AI model based on contextual information in order to improve the accuracy of target category recognition.

[0080] A semantic feature encoder is used to encode the semantic feature vectors obtained based on OOD detection and processing and Bayesian optimization into a form suitable for wireless channel transmission.

[0081] A signal transmission unit, used to transmit encoded signals via a wireless channel;

[0082] The receiving end includes:

[0083] A signal receiver, used to receive signals transmitted via a wireless channel;

[0084] A signal decoder is used to decode received signals and recover semantic feature vectors.

[0085] An image reconstruction and generation-evaluation framework iterative optimizer is used to reconstruct images and iteratively optimize them based on the semantic feature vectors obtained from decoding, using a generation-evaluation framework based on multimodal large language models (MLLMs) to achieve accurate image reconstruction.

[0086] The output unit is used to output the final reconstructed image.

[0087] The following describes specific embodiments of the present invention.

[0088] System Framework

[0089] Semantic encoder: Responsible for extracting semantic features from input images.

[0090] MLLM module: Provides auxiliary semantic information when encountering OOD input.

[0091] Bayesian optimization module: Optimizes the output probability distribution of MLLM based on image context information.

[0092] Channel encoder: Encodes semantic features into signals suitable for transmission over a wireless channel.

[0093] Receiver decoder: Decodes the received signal to recover its semantic features.

[0094] Image reconstruction module: Reconstructs the image based on the semantic features obtained from decoding.

[0095] Generate-evaluate framework: Improve the accuracy of image reconstruction by leveraging the collaboration of multiple MLLMs.

[0096] Workflow

[0097] Step 1: Input Image Processing

[0098] Step 1.1: The system obtains the image data to be transmitted from the image source.

[0099] Step 1.2: The semantic encoder receives image data and begins to perform preliminary semantic feature extraction on the image, including but not limited to object detection and scene understanding.

[0100] Step 2: OOD Detection and Processing

[0101] - Step 2.1: For each extracted semantic feature, an expert AI model (e.g., YOLOv8) identifies and calculates its confidence level.

[0102] Step 2.2: If the confidence level of the model is lower than the pre-set threshold ρ, then the feature is considered to be an OOD (Out-of-Distribution) situation.

[0103] Step 2.2.1: For each semantic feature extracted by the semantic encoder, the expert AI model (YOLOv8) will identify each feature object according to its internal mechanism and assign a confidence score C to each identification result. i C i This represents the confidence level of the i-th feature.

[0104] Step 2.2.2: The system sets a confidence threshold ρ, which is an empirical value used to distinguish whether the model has enough confidence in its recognition results.

[0105] Step 2.2.3: For each identified object, the system checks the confidence level C given by the expert AI model. i Is it below this preset threshold ρ?

[0106] Step 2.2.4: If the confidence score C i If the value is less than ρ, then the expert AI model's recognition result for this object is considered unreliable, and there may be an OOD situation.

[0107] Step 2.2.5: Mark the object as a potential OOD situation and prepare to trigger the general AI model for further evaluation.

[0108] Step 2.3: At this point, the general AI model InstructBLIP is triggered to re-evaluate the low-confidence features.

[0109] Step 2.3.1: Once an object with a confidence level below the threshold is detected, the system will trigger the intervention of the general AI model (InstructBLIP).

[0110] Step 2.3.2: The system crops out the object region defined by the expert AI model from the original image to form a new sub-image.

[0111] Step 2.3.3: Submit this sub-image to a general AI model for classification, along with a simple prompt, such as: "Please tell me what's in this image?"

[0112] Step 2.3.4: The general AI model identifies the content of the sub-image and returns a list containing possible categories and their confidence scores. Assume the probability distribution generated by the general AI model is p(s). i ), where s i This represents the i-th possible category.

[0113] Step 2.3.5: The system selects the category with the highest confidence from the results returned by the general AI model as the final recognition result.

[0114] Step 2.3.6: Correct the feature to the recognition result of general AI;

[0115] Step 2.3.7: After re-evaluating all low-confidence feature objects, the system integrates the final recognition results of all objects to form a complete image semantic information vector.

[0116] S i,A This refers to the results identified by expert AI models, S i,B This refers to the results recognized by a general AI model. Through this framework, the system can effectively combine the advantages of both models, improving the accuracy and robustness of the recognition.

[0117] Step 3: Bayesian Optimization

[0118] Step 3.1: The general AI model identifies each low-confidence object sub-image and generates a preliminary identification result.

[0119] When the expert AI model (YOLOv8) exhibits low confidence when processing a certain image feature—that is, the model's confidence in recognizing a particular feature object is below a predetermined threshold (e.g., less than 0.7)—it is considered that there is uncertainty in the recognition of this object. In this case, the image region of the feature object is passed to the general AI model (InstructBLIP). The general AI model receives this image region and generates a preliminary recognition result.

[0120] For example, a general AI model might output a list containing multiple highly probable categories and their corresponding probabilities. Suppose the output is (Bear, 0.5), (Plate, 0.4), (Other, 0.1), then the initial identification result would be the output category with the highest probability, which is Bear. However, it can be seen that in this example, the confidence difference between "Bear" and "Plate" is not significant.

[0121] Step 3.2: Based on the context information, use the Bayesian method to adjust the probability distribution of the general AI model output, thereby improving the model's accuracy in identifying target categories.

[0122] After obtaining the initial recognition results from the general AI model, the probability distribution of the model's output is adjusted using known image context information. Context information refers to other clearly identifiable objects or scene features in the image besides the target object.

[0123] For example, in the example above, if the image background contains items such as a dining table, cake, teacup, and cutlery, then information about these items can be used to help determine the identity of the target object. Specifically, Bayes' theorem is used to calculate the new posterior probability distribution. First, the prior probabilities are defined as the probability distribution output by the general AI model: P(Bear) = 0.5, P(Plate) = 0.4, P(Others) = 0.1. Then, based on the contextual information (dining table, cake, etc.), the relevance score between each candidate category and the background information is calculated. For example, "plate" is strongly correlated with dining table, cake, etc., while "bear" is not. Next, using the normalized relevance scores as known information, combined with the prior probabilities, the updated posterior probability distributions P'(Bear), P'(Plate), and P'(Others) are calculated using Bayes' theorem.

[0124] Step 3.3: The adjusted probability distribution is used to correct the initial identification results.

[0125] After Bayesian optimization, the new posterior probability distribution may change, for example, (bear, 0.1), (plate, 0.9), (other, 0.0). This means that after considering contextual information, the model's confidence in the "plate" category increases significantly, while its confidence in the "bear" category decreases significantly. Therefore, the final recognition result is corrected to "plate". This step ensures that even when the initial model's recognition is uncertain, by introducing contextual information, the semantic information of the image can be expressed more accurately, thus improving the accuracy of recognition.

[0126] In this way, the Bayesian optimization scheme not only enhances the model's performance when faced with OOD input, but also gives more opportunities to consider alternative categories in complex situations, thereby improving the accuracy and robustness of the image semantic compression process.

[0127] Step 4: Semantic Feature Encoding

[0128] Step 4.1: Obtain the semantic feature vector corresponding to the whole image.

[0129] Step 4.2: The channel encoder uses an appropriate coding scheme (such as LDPC or Turbo code) to convert the semantic feature vector into a form suitable for wireless channel transmission.

[0130] Step 5: Signal Transmission

[0131] Step 5.1: Transmit the encoded signal to the receiving end via a wireless channel. During this process, the signal may be affected by various noises.

[0132] Step 6: Signal Reception and Decoding

[0133] Step 6.1: After receiving the signal, the receiving end uses the corresponding decoding algorithm to recover the semantic feature vector.

[0134] Step 6.2: The decoded semantic tags will be passed to the next step for image reconstruction.

[0135] The innovation of the "generate-evaluate framework" lies in its utilization of cooperation among multimodal large language models to enhance the reliability and quality of image reconstruction. Specifically, this framework includes the following detailed steps in the receiver's workflow:

[0136] Step 7: Image Reconstruction and Generation - Evaluation Framework Iterative Optimization

[0137] Step 7.1: Generating Part

[0138] First, a complete textual semantic description of the image is obtained based on the received semantic feature vector. Then, the image generation model generates an image based on this description. The semantic feature vector contains semantic features of the desired image, such as the type and quantity of objects and their relative positions in the image.

[0139] The task of generative models is to reconstruct images as accurately as possible from a complete semantic description of the image. For example, if the description mentions a cake and forks on a table, the generative model needs to generate an image that includes these elements.

[0140] Step 7.2: Evaluation Section

[0141] The role of the evaluation model is to assess the generated image. It compares the generated image with the original semantic description to check whether the generated image matches the expected description.

[0142] If there are any unexpected features in the generated image, the evaluation model will point out these errors. For example, if the description mentions three forks, but the generated image only shows two, the evaluation model will point this out.

[0143] The evaluation model can also identify other semantic errors in the generated images, such as errors in the number, location, or other attributes of objects.

[0144] Step 7.3: Iterative optimization of the generation-evaluation framework

[0145] Step 7.3.1: Adjustments based on evaluation feedback

[0146] When the evaluation model detects problems with the generated image, it provides feedback to the generating model. This feedback can include the specific type and location of the error.

[0147] Step 7.3.2: Repeat generation

[0148] The generative model regenerates a new image based on the evaluation feedback. This process continues iteratively until the generated image fully matches the semantic description of the image or the preset maximum number of iterations is reached.

[0149] This iterative process helps to gradually improve the quality of the generated image, making the final output image more accurately reflect the original semantic description.

[0150] Through the above steps, the "generative-evaluation framework" can significantly improve the reliability and quality of generative models in image reconstruction.

[0151] Step 8: Output the final image

[0152] Step 8.1: Once the image reconstruction module confirms that the reconstructed image matches the semantic description, it outputs the final image.

[0153] In summary, this invention provides an advanced image semantic communication solution that not only overcomes some limitations of existing technologies but also enhances the system's robustness and image reconstruction quality by introducing a multimodal large language model, opening up new possibilities for future semantic communication technologies. Compared with traditional technologies, the significant advantages of this invention are:

[0154] 1. Enhanced robustness: By integrating MLLMs, this invention exhibits greater robustness when facing unknown data. Even in OOD scenarios, the system can leverage the model's generalization ability to process this data, thereby ensuring accurate image recognition and reconstruction.

[0155] 2. Enhanced Image Reconstruction Quality: Within the "generate-evaluate" framework, multiple MLLMs work collaboratively to improve the quality of image reconstruction. This approach is similar to the iterative improvement process humans use when creating works of art, continuously trying and refining to ultimately obtain high-quality image reconstruction results.

[0156] This invention solves the OOD problem by combining the advantages of expert AI models and general AI models, improving the accuracy and robustness of semantic encoding. By incorporating a Bayesian optimization scheme and introducing contextual information to optimize the inference process of MLLMs, this invention can more accurately represent the semantic information of images. Through the construction of a generator-evaluation framework, at the receiving end, the invention significantly improves the reliability and quality of image reconstruction through cooperation among multiple MLLMs. These innovations provide a more advanced and reliable image semantic communication system, laying a solid foundation for future wireless communication technologies.

[0157] This invention also provides a storage medium for storing a computer program, which, when executed, performs at least the methods described above.

[0158] This invention also provides a control device, including a processor and a storage medium for storing a computer program; wherein the processor executes the computer program by performing at least the method described above.

[0159] This invention also provides a processor that executes a computer program, at least performing the methods described above.

[0160] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk drive or magnetic tape drive. The storage media described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0161] In the several embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0162] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0163] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0164] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0165] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0166] The methods disclosed in the several method embodiments provided by this invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0167] The features disclosed in the several product embodiments provided by this invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0168] The features disclosed in the several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0169] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or application, should be considered within the scope of protection of the present invention.

Claims

1. An image semantic communication method based on a multimodal large language model, characterized in that, Includes the following steps: Step 1: Input Image Processing: Obtain image data from the image source and perform preliminary semantic feature extraction; Step 2, OOD Detection and Processing: Image features are identified and confidence scores are calculated using an expert AI model. If the confidence score is lower than a preset threshold, a general AI model is triggered to re-evaluate the features in order to identify and process OOD situations. Step 3, Bayesian optimization: Use the Bayesian method to adjust the probability distribution of the general AI model output based on contextual information to improve the accuracy of target category recognition; Step 4, Semantic Feature Encoding: Based on the results of OOD detection and processing and Bayesian optimization, the obtained semantic feature vector is encoded into a form suitable for wireless channel transmission; Step 5, Signal Transmission: Transmit the encoded signal through the wireless channel; Step 6, Signal Reception and Decoding: The receiver receives and decodes the signal to recover the semantic feature vector; Step 7, Image Reconstruction and Iterative Optimization of the Generation-Evaluation Framework: Based on the semantic feature vector obtained from decoding, the image is reconstructed and iteratively optimized using a generation-evaluation framework based on multimodal large language models (MLLMs) to achieve accurate image reconstruction. Step 8: Output the final image.

2. The image semantic communication method based on a multimodal large language model as described in claim 1, characterized in that, In step 2, the triggering of the general AI model to re-evaluate features in order to identify and handle OOD situations specifically includes: Detecting features with confidence levels below a threshold triggers the intervention of a general AI model. Cropping low-confidence object regions defined by the expert AI model creates sub-images; Submit sub-images to a general AI model with hints for classification; From the list of categories and their confidence scores returned by the general AI model, select the category with the highest confidence score as the recognition result; The feature is then corrected to reflect the results of general AI recognition. The final recognition results of all objects are integrated to form a complete image semantic information vector.

3. The image semantic communication method based on a multimodal large language model as described in claim 1 or 2, characterized in that, Step 3, which involves adjusting the probability distribution of the general AI model output using Bayesian methods based on contextual information, specifically includes: For low-confidence image features, preliminary recognition results are generated by a general AI model; Based on image context information, Bayes' theorem is applied to adjust the probability distribution of the preliminary recognition results; By using the Bayesian-optimized posterior probability, the recognition results of the general AI model are corrected to improve the accuracy of recognition.

4. The image semantic communication method based on a multimodal large language model as described in any one of claims 1 to 3, characterized in that, Step 4, specifically encoding the semantic feature vector into a form suitable for wireless channel transmission, includes: Obtain the semantic feature vector corresponding to the entire image; The channel encoder uses an appropriate coding scheme to convert semantic feature vectors into a form suitable for wireless channel transmission.

5. The image semantic communication method based on a multimodal large language model as described in any one of claims 1 to 4, characterized in that, In step 7, under the generation-evaluation framework, multimodal large language models (MLLMs) collaborate, with one model responsible for generating images and the other acting as an evaluator to assess the quality of the generated images and provide feedback in order to continuously optimize the generated results.

6. The image semantic communication method based on a multimodal large language model as described in claim 5, characterized in that, Step 7, which involves reconstructing the image and iteratively optimizing it using a generative-evaluative framework based on multimodal large language models (MLLMs), specifically includes: The generated image is compared with the original semantic description using an evaluation model to identify semantic errors in the image; The evaluation model provides evaluation feedback based on semantic errors; The generative model regenerates the image based on the evaluation feedback; Therefore, the reconstructed image is iteratively optimized until the generated image meets the accuracy requirements of semantic description or reaches the preset maximum number of iterations.

7. An image semantic communication system based on a multimodal large language model, characterized in that, Including the sender and receiver, The sending end includes: An input image processor is used to acquire image data from an image source and perform preliminary semantic feature extraction. The OOD detector is used to identify image features and calculate confidence scores through an expert AI model. If the confidence score is lower than a preset threshold, it triggers a general AI model to re-evaluate the features in order to identify and process OOD situations. A Bayesian optimizer is used to adjust the probability distribution of the output of a general AI model based on contextual information in order to improve the accuracy of target category recognition. A semantic feature encoder is used to encode the semantic feature vectors obtained based on OOD detection and processing and Bayesian optimization into a form suitable for wireless channel transmission. A signal transmission unit, used to transmit encoded signals via a wireless channel; The receiving end includes: A signal receiver, used to receive signals transmitted via a wireless channel; A signal decoder is used to decode received signals and recover semantic feature vectors. An image reconstruction and generation-evaluation framework iterative optimizer is used to reconstruct images and iteratively optimize them based on the semantic feature vectors obtained from decoding, using a generation-evaluation framework based on multimodal large language models (MLLMs) to achieve accurate image reconstruction. The output unit is used to output the final reconstructed image.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the image semantic communication method based on a multimodal large language model as described in any one of claims 1 to 7.

Images