Text generation and information recovery method and system based on large model

Abstract

The invention provides a large model-based text generation and information recovery method and system. The method comprises the following steps of: converting complex situation information into simple and accurate text description by utilizing a text generation model; converting the text description of the situation information into a semantic vector by using a text encoder of a pre-trained CLIP model, and capturing semantic information of the text; encoding the noise image into a hidden space representation by using an encoder of a variational auto-encoder, and learning a compressed representation of the image; fusing the semantic vector generated by the pre-trained CLIP model with the image hidden space representation generated by the encoder of the variational auto-encoder, and optimizing the fused hidden space representation by using the trained de-noising Unet network; the optimized hidden space representation is decoded into a high quality situational information image using a decoder of a variational auto-encoder. By applying the technical scheme of the invention, the technical problem of incomplete received information caused by inevitable information loss in the data compression process in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a large model-based text generation and information recovery method and system. Background Art

[0002] Affected by the harsh natural environment and human interference, the communication infrastructure in some special scenarios is weak and cannot guarantee stable and reliable communication quality, which has greatly affected the information transmission between communication nodes in the communication network. At the same time, the demand for real-time and high-speed information transmission is increasing day by day, especially the transmission of real-time images, which can facilitate people to understand the on-site status in a timely and accurate manner. How to ensure the timely transmission of information between communication units in special environments with limited communication conditions has become a technical problem that needs to be solved urgently.

[0003] In the prior art, data compression methods are generally used to cope with limited communication bandwidth, such as removing redundant information in the data, or quantizing and encoding highly repetitive data, or using standardization and normalization methods to reduce the amount of data. In terms of specific compression methods, the prior art includes block compression, incremental compression, dictionary coding compression and other technologies.

[0004] Although the existing technology can provide protection for information exchange between communication units under limited communication conditions to a certain extent, there is inevitably information loss in the data compression process, resulting in incomplete received information. Summary of the invention

[0005] The present invention provides a large model-based text generation and information recovery method and system, which can solve the technical problem in the prior art that information loss is inevitable in the data compression process, resulting in incomplete received information.

[0006] According to one aspect of the present invention, a large-model-based text generation and information recovery method is provided, which includes: using a text generation model to convert complex situation information into a concise and accurate text description; using an information recovery model to reconstruct a high-quality, semantically accurate situation image from the received text description, and in the information recovery stage, using a pre-trained CLIP model's text encoder to convert the situation information text description into a semantic vector to capture the semantic information of the text; using a variational autoencoder (VAE) encoder to encode a noisy image into a latent space representation and learn a compressed representation of the image; fusing the semantic vector generated by the pre-trained CLIP model with the image latent space representation generated by the variational autoencoder (VAE) encoder, and optimizing the fused latent space representation using a trained denoising Unet network; and using a variational autoencoder (VAE) decoder to decode the optimized latent space representation into a high-quality situation information image.

[0007] Furthermore, the text generation model uses the latest multimodal version of the MiniCPM series, MiniCPM-V2.0. The model is built based on MiniCPM 2.4B and SigLip-400M, with a total of 2.8B parameters. MiniCPM-V 2.0 has leading optical character recognition (OCR) and multimodal understanding capabilities.

[0008] Furthermore, the information recovery model is constructed with reference to the Stable Diffusion image generation model. The information recovery model includes a semantic extractor (Text Extraction), an image denoiser (Unet) and an image encoder / decoder (VAE). The semantic extractor (Text Extraction) is responsible for processing the input text information and converting it into corresponding Token Embeddings. The ClipText model developed and implemented by OpenAI is used as the main tool for semantic extraction. The CLIP Encoder is pre-trained on a large-scale image-text pair dataset. In the information recovery model, the CLIP Encoder receives the input situation text description and converts it into a fixed-length vector representation.

[0009] Furthermore, a variational autoencoder (VAE) encoder is used to encode the noisy image into a latent space representation, and learning the compressed representation of the image specifically includes: using an image compression algorithm to reduce the resolution of the original information image collected by the data sender to obtain a compressed low-resolution image; using a variational autoencoder (VAE) encoder to encode the low-resolution image into a latent space representation, and learning the compressed representation of the image.

[0010] Furthermore, the semantic vector generated by the pre-trained CLIP model is fused with the image latent space representation generated by the encoder of the variational autoencoder (VAE), and the fused latent space representation is optimized using the trained denoising Unet network. Specifically, U-Net receives two parts of input, one is the latent space representation generated by the encoder of the variational autoencoder (VAE), and the other is the semantic vector generated by the CLIP Encoder; in the encoder (CNN) part of U-Net, the latent space representation generated by the encoder of the variational autoencoder (VAE) is first decoded and upsampled to reconstruct a preliminary estimate of the original image; the preliminary estimate of the original image is fused with the semantic vector generated by the CLIP Encoder to form a fused latent space; the decoder (deconvolution) part of U-Net gradually upsamples and restores the fused latent space to generate a high-quality situation image with the same resolution as the original image.

[0011] Furthermore, the initial estimate of the original image is fused with the semantic vector generated by CLIP Encoder through concatenation, addition or attention.

[0012] Furthermore, the fused latent space contains the joint representation of image content and situation description, guiding the subsequent recovery process.

[0013] Furthermore, in the process of restoring the fused latent space, the U-Net decoder uses the multi-scale features extracted by the U-Net encoder and the information transmitted by the jump connection to gradually restore the details and texture of the image. The semantic guidance provided by the conditional vector helps the network generate images that conform to the description content, ensuring that the restoration results are consistent with the given situation information.

[0014] Furthermore, using the VAE decoder to decode the optimized latent space representation into a high-quality situation information image specifically includes: using the VAE decoder to extract and interpret the key features and attributes of the image from the latent space through deconvolution layers and upsampling operations, and decoding the optimized latent space representation into a high-quality situation information image.

[0015] According to another aspect of the present invention, a large model-based text generation and information recovery system is provided, which uses the large model-based text generation and information recovery method as described above to perform text generation and information recovery.

[0016] The technical solution of the present invention is applied to provide a text generation and information recovery method based on a large model. The method uses large model technology to achieve low bandwidth and high real-time communication requirements for the situation image transmission problem between communication units in special environments with limited communication conditions. The method consists of two key components: a text generation model and an information recovery model; the text generation model aims to convert complex situation information into concise and accurate text descriptions. By utilizing natural language processing technology and the semantic understanding ability of the visual large model, the module can automatically extract key information in the situation image, such as target category, motion state, etc., and generate corresponding text descriptions. This text description is not only semantically rich, but also has a small amount of data, which is very suitable for transmission in a low-bandwidth environment; the information recovery model is responsible for reconstructing a high-quality, semantically accurate situation image from the received text description. The module adopts image compression and generation technology, such as variational autoencoder (VAE), and realizes the transformation from compact text description to realistic situation image by learning low-dimensional representation and generation model of the image. The generated image not only has high visual quality, but also is consistent with the original situation information at the semantic level, ensuring the accuracy of situation perception. Therefore, compared with the prior art, the text generation and information recovery method based on the large model provided by the present invention can, on the one hand, adaptively adjust the compression rate according to network conditions, adapt to the elastic network conditions in special environments, and effectively ensure the real-time communication; on the other hand, due to the use of the information recovery model, the consistency of the sent and received information is guaranteed to the maximum extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The included drawings are used to provide a further understanding of the embodiments of the present invention, which constitute a part of the specification, are used to illustrate the embodiments of the present invention, and together with the text description, explain the principles of the present invention. Obviously, the drawings in the following description are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 A schematic diagram of the structure of a text generation model provided according to a specific embodiment of the present invention is shown, wherein its input is an uncompressed image and its output is a text which accurately describes the image;

[0019] Figure 2 A schematic diagram of an information recovery model provided according to a specific embodiment of the present invention is shown, wherein the input of the model is file information and image information, and the image is a compressed ultra-low resolution image with an extremely small file size;

[0020] Figure 3 A schematic diagram of a CLIP model architecture provided according to a specific embodiment of the present invention is shown;

[0021] Figure 4 A schematic diagram of a Unet network architecture provided according to a specific embodiment of the present invention is shown;

[0022] Figure 5 A schematic diagram of the VAE model structure provided according to a specific embodiment of the present invention is shown. DETAILED DESCRIPTION

[0023] It should be noted that, in the absence of conflict, the embodiments in this application and the features in the embodiments can be combined with each other. The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0025] Unless otherwise specifically stated, the relative arrangement of the parts and steps described in these embodiments, numerical expressions and numerical values ​​do not limit the scope of the present invention. At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the accompanying drawings are not drawn according to the actual proportional relationship. The technology, method and equipment known to ordinary technicians in the relevant field may not be discussed in detail, but in appropriate cases, the technology, method and equipment should be regarded as a part of the authorization specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once a certain item is defined in an accompanying drawing, it does not need to be further discussed in subsequent drawings.

[0026] like Figures 1 to 5As shown, according to a specific embodiment of the present invention, a large model-based text generation and information recovery method is provided, and the large model-based text generation and information recovery method includes: using a text generation model to convert complex situation information into a concise and accurate text description; using an information recovery model to reconstruct a high-quality, semantically accurate situation image from the received text description, and in the information recovery stage, using a pre-trained CLIP model text encoder to convert the text description of the situation information into a semantic vector to capture the semantic information of the text; using a variational autoencoder (VAE) encoder to encode the noisy image into a latent space representation, and learn the compressed representation of the image; fusing the semantic vector generated by the pre-trained CLIP model with the image latent space representation generated by the variational autoencoder (VAE) encoder, and using a trained denoising Unet network to optimize the fused latent space representation; using a variational autoencoder (VAE) decoder to decode the optimized latent space representation into a high-quality situation information image.

[0027] By using this configuration, a text generation and information recovery method based on a large model is provided. This method uses large model technology to achieve low-bandwidth and high-real-time communication requirements for the problem of situation image transmission between communication units in special environments with limited communication conditions. The method consists of two key components: a text generation model and an information recovery model; the text generation model aims to convert complex situation information into concise and accurate text descriptions. By utilizing natural language processing technology and the semantic understanding ability of the visual large model, the module can automatically extract key information from the situation image, such as target category, motion state, etc., and generate corresponding text descriptions. This text description is not only semantically rich, but also has a small amount of data, which is very suitable for transmission in low-bandwidth environments; the information recovery model is responsible for reconstructing high-quality and semantically accurate situation images from the received text description. This module uses image compression and generation technology, such as variational autoencoder (VAE), and realizes the transformation from compact text description to realistic situation image by learning low-dimensional representation and generation model of images. The generated image is not only of high visual quality, but also consistent with the original situation information at the semantic level, ensuring the accuracy of situation awareness. Therefore, compared with the prior art, the text generation and information recovery method based on the large model provided by the present invention can, on the one hand, adaptively adjust the compression rate according to network conditions, adapt to the elastic network conditions in special environments, and effectively ensure the real-time communication; on the other hand, due to the use of the information recovery model, the consistency of the sent and received information is guaranteed to the maximum extent.

[0028] Specifically, in the present invention, the text generation model uses the latest multimodal version of the MiniCPM series, MiniCPM-V 2.0, which is built based on MiniCPM 2.4B and SigLip-400M and has a total of 2.8B parameters. MiniCPM-V 2.0 has leading optical character recognition (OCR) and multimodal understanding capabilities.

[0029] The information recovery model is constructed with reference to the Stable Diffusion image generation model. The information recovery model includes a semantic extractor (Text Extraction), an image denoiser (Unet) and an image encoder / decoder (VAE). The semantic extractor (Text Extraction) is responsible for processing the input text information and converting it into corresponding TokenEmbeddings (token embeddings). The ClipText model developed and implemented by OpenAI is used as the main tool for semantic extraction. The CLIP Encoder is pre-trained on a large-scale image-text pair dataset. In the information recovery model, the CLIP Encoder receives the input situation text description and converts it into a fixed-length vector representation.

[0030] Among them, in the present invention, in order to enable the model to understand and associate the semantic information between text and image, the encoder of variational autoencoder (VAE) is used to encode the noise image into latent space representation, and the compressed representation of the learning image specifically includes: using the image compression algorithm to reduce the resolution of the original information image collected by the data sender to obtain a compressed low-resolution image; using the encoder of variational autoencoder (VAE) to encode this low-resolution image into latent space representation, and learning the compressed representation of the image. In this configuration, the text description of the situation information image is combined with the compressed image representation, so that the model can understand and associate the semantic information between the text and image.

[0031] Further, in the present invention, the semantic vector generated by the pre-trained CLIP model is fused with the image latent space representation generated by the encoder of the variational autoencoder (VAE), and the fused latent space representation is optimized using the trained denoising Unet network. Specifically, the U-Net receives two parts of input, one is the latent space representation generated by the encoder of the variational autoencoder (VAE), and the other is the semantic vector generated by the CLIP Encoder. The CLIP Encoder converts the situation description in text form into a semantically rich vector representation to provide additional semantic guidance for the image restoration process. In the encoder (CNN) part of the U-Net, the latent space representation generated by the encoder of the variational autoencoder (VAE) is first decoded and upsampled to reconstruct a preliminary estimate of the original image; the preliminary estimate of the original image is fused with the semantic vector generated by the CLIP Encoder to form a fused latent space; the decoder (deconvolution) part of the U-Net gradually upsamples and restores the fused latent space to generate a high-quality situation image with the same resolution as the original image.

[0032] The initial estimate of the original image is fused with the semantic vector generated by CLIP Encoder through concatenation, addition or attention, with the aim of aligning and interacting the semantic information with the visual features. The fused latent space contains the joint representation of the image content and the situation description, guiding the subsequent restoration process.

[0033] Furthermore, in the process of restoring the fused latent space, the U-Net decoder uses the multi-scale features extracted by the U-Net encoder and the information transmitted by the jump connection to gradually restore the details and texture of the image. The semantic guidance provided by the conditional vector helps the network generate images that conform to the description content, ensuring that the restoration results are consistent with the given situation information.

[0034] Through this architectural design, VAE Encoder and U-Net form a collaborative system. VAE Encoder is responsible for encoding low-resolution images into compact latent vectors that capture the main features and semantic information of the image. U-Net uses this latent vector and the semantic vector provided by CLIP Encoder to generate high-quality, semantically accurate high-resolution situation images through conditional image super-resolution reconstruction.

[0035] Furthermore, in the present invention, using a VAE decoder to decode the optimized latent space representation into a high-quality situation information image specifically includes: using the VAE decoder to extract and interpret the key features and attributes of the image from the latent space through a deconvolution layer and upsampling operations, and decoding the optimized latent space representation into a high-quality situation information image.

[0036] According to another aspect of the present invention, a large model-based text generation and information recovery system is provided, which uses the large model-based text generation and information recovery method as described above to perform text generation and information recovery.

[0037] By using this configuration, a text generation and information recovery system based on a large model is provided. The system uses large model technology to achieve low-bandwidth and high-real-time communication requirements for the problem of situation image transmission between communication units in special environments with limited communication conditions. The method consists of two key components: a text generation model and an information recovery model; the text generation model aims to convert complex situation information into concise and accurate text descriptions. By utilizing natural language processing technology and the semantic understanding ability of the visual large model, the module can automatically extract key information from the situation image, such as target category, motion state, etc., and generate corresponding text descriptions. This text description is not only semantically rich, but also has a small amount of data, which is very suitable for transmission in low-bandwidth environments; the information recovery model is responsible for reconstructing high-quality and semantically accurate situation images from the received text description. The module uses image compression and generation technology, such as variational autoencoder (VAE), and realizes the transformation from compact text description to realistic situation image by learning low-dimensional representation and generation model of images. The generated image is not only of high visual quality, but also consistent with the original situation information at the semantic level, ensuring the accuracy of situation awareness. Therefore, compared with the prior art, the large model-based text generation and information recovery system provided by the present invention can solve the problem of situation image transmission between communication units under limited communication conditions.

[0038] In order to further understand the present invention, the following Figures 1 to 5 The large model-based text generation and information recovery method and system provided by the present invention are described in detail.

[0039] like Figures 1 to 5 As shown, according to a specific embodiment of the present invention, a text generation and information recovery method based on a large model is provided, and the method consists of two key components: a text generation model and an information recovery model.

[0040] 1) Text Generation Model

[0041] The text generation model uses the latest multimodal version of the MiniCPM series, MiniCPM-V 2.0. The model is built on MiniCPM 2.4B and SigLip-400M, with a total of 2.8B parameters. MiniCPM-V 2.0 has leading optical character recognition (OCR) and multimodal understanding capabilities. The model has reached the best level of the open source community on the comprehensive OCR capability evaluation benchmark OCR Bench, and even achieved performance close to Gemini Pro in scene text understanding.

[0042] 2) Information recovery model

[0043] In the information recovery stage, an architecture and workflow similar to the Stable Diffusion model are adopted. First, the text encoder of the pre-trained CLIP (Contrastive Language-Image Pre-training) model is used to convert the text description of the situation information into a semantic vector to capture the semantic information of the text. Then, the encoder of the variational autoencoder (VAE) is used to encode the noisy image into a latent space representation and learn the compressed representation of the image. Next, the semantic vector generated by the CLIP text encoder is fused with the image latent space representation generated by the VAE encoder, and the fused latent space representation is optimized using the trained denoising Unet network. Unet gradually removes noise and adds details in the image through multi-scale cross-layer connections. Finally, the decoder of VAE is used to decode the optimized latent space representation into a high-quality situation information image. The decoder generates a realistic image based on the optimized latent space representation, realizing the recovery and generation from the text description of the situation information to the image. By adopting the key steps of Stable Diffusion such as text encoding, image encoding, latent space optimization and image decoding, an efficient and accurate information recovery process is established, which can generate high-quality situation information images based on text descriptions and complete the transmission and recovery of situation information.

[0044] Compared with the existing technology, on the one hand, this method can adaptively adjust the compression rate according to network conditions, adapt to the elastic network conditions in special environments, and effectively ensure the real-time communication; on the other hand, due to the use of the information recovery model, the maximum program guarantees the consistency of sending and receiving information.

[0045] As a specific embodiment of the present invention, the implementation method includes two parts: a text generation model and an information recovery model.

[0046] The text generation model aims to transform complex situation information into concise and accurate text descriptions. By leveraging natural language processing technology and the semantic understanding capabilities of the large visual model, the module can automatically extract key information from situation images, such as target category, motion status, etc., and generate corresponding text descriptions. This text description is not only semantically rich, but also has a small amount of data, making it very suitable for transmission in low-bandwidth environments.

[0047] The information recovery model is responsible for reconstructing high-quality, semantically accurate situation images from the received text descriptions. This module uses image compression and generation techniques, such as variational autoencoders (VAEs), to achieve the transformation from compact text descriptions to realistic situation images by learning low-dimensional representations and generative models of images. The generated images are not only of high visual quality, but also consistent with the original situation information at the semantic level, ensuring the accuracy of situation awareness.

[0048] 1) Text Generation Model

[0049] Build a text generation model based on MiniCPM-V 2.0 to achieve the following functions.

[0050] a. Leading OCR and multimodal understanding capabilities

[0051] MiniCPM-V 2.0 significantly improves OCR and multimodal understanding capabilities, and its scene text understanding capabilities are close to GeminiPro. Its performance exceeds that of mainstream models with larger parameter scales (e.g. 17-34B) on multiple mainstream evaluation benchmarks. This enables the model to accurately extract and understand key information in images, providing a reliable foundation for subsequent situation information generation.

[0052] b. Trustworthy Behavior

[0053] MiniCPM-V 2.0 is the first large multimodal model on-device that uses multimodal RLHF alignment (with the help of the RLHF-V [CVPR'24] series technology. The model achieves performance similar to GPT-4V on Object HalBench.

[0054] c. Efficient encoding of high-definition images with arbitrary aspect ratios

[0055] MiniCPM-V 2.0 can accept 1.8-megapixel image inputs of arbitrary aspect ratios (based on the latest LLaVA-UHD technology), which enables the model to perceive more fine-grained visual information such as small objects and dense text.

[0056] d. Efficient deployment

[0057] MiniCPM-V 2.0 can be efficiently deployed on most consumer-grade graphics cards, personal computers, mobile phones and other terminal devices. This can be well adapted to individual information processing systems, unmanned vehicles, drones and other terminal equipment with limited computing power.

[0058] e. Bilingual support

[0059] MiniCPM-V 2.0 provides leading Chinese-English bilingual multimodal capability support. This capability is achieved through the cross-language generalization technology of multimodal capabilities proposed in the VisCPM[ICLR'24] paper. This can well support the extraction of target English information and the output of Chinese situation information.

[0060] After completing the construction of the text generation model, define the model interface function to implement the model call. The interface definition is as follows:

[0061]

[0062] 2) Information recovery model

[0063] First, we build an information recovery model with reference to the Stable Diffusion image generation model. Stable Diffusion is a text-to-image (txt2img) latent diffusion model (LDM). The information recovery model uses an architecture similar to Stable Diffusion, but improves on the generation of latent space representation. First, we use an image compression algorithm to reduce the resolution of the original information image collected by the data sender to obtain a compressed low-resolution image. Then, we use the variational autoencoder (VAE) encoder to encode this low-resolution image into a latent space representation and learn the compressed representation of the image.

[0064] Next, the semantic vector generated by the pre-trained CLIP (Contrastive Language-Image Pre-training) model is fused with the latent space representation generated by the VAE encoder. This step combines the text description of the situation information image with the compressed image representation, allowing the model to understand and associate the semantic information between the text and the image.

[0065] Then, the trained denoising Unet network is used to optimize the fused latent space representation. The Unet network gradually removes noise and adds details in the image through multi-scale cross-layer connections. Finally, the VAE decoder is used to decode the optimized latent space representation into a high-quality situation information image to complete the recovery of the situation information. Through the above steps, the information recovery model can effectively combine the information of the text description and the original situation image to restore and generate a high-quality situation information image that conforms to the semantic description.

[0066] The information recovery model includes three core components: semantic extractor (Text Extraction), image denoiser (Unet), and image codec (VAE).

[0067] (1) Text Extraction

[0068] Text Extraction plays a vital role in the information recovery model. It is responsible for processing the input text information and converting it into corresponding Token Embeddings. This process is a key step in mapping natural language text to a continuous vector space, providing semantic guidance for subsequent image generation and enhancement.

[0069] Specifically, the information recovery model uses ClipText developed by OpenAI as the main tool for semantic extraction. ClipText is a pre-trained language model based on the BERT (Bidirectional Encoder Representations from Transformers) architecture, which contains 63 million parameters and has powerful language understanding and representation capabilities.

[0070] The semantic extractor (Text Extraction) uses the open source tool ClipText, and CLIP Encoder is a component of the open source tool ClipText. CLIP Encoder is pre-trained on a large-scale image-text pair dataset. During the pre-training process, the model learns to map images and corresponding text descriptions to a common embedding space, so that in this space, semantically similar images and texts are closer in vector representation. This comparative learning method enables CLIP Encoder to effectively capture the semantic information of text descriptions and learn high-quality text representations.

[0071] In the information recovery model, CLIP Encoder receives the input situation text description and converts it into a fixed-length vector representation. This vector encodes the semantic content of the text, including information such as scenes, objects, attributes, and the relationship between them. By mapping text to a semantic space, CLIP Encoder provides a universal language representation that can guide the subsequent image generation process and ensure that the generated image is semantically consistent with the given text description. The CLIP model architecture is shown in Figure 1. Figure 3 shown.

[0072] CLIP Encoder's powerful semantic understanding and representation capabilities come from its pre-training on large-scale multimodal data. By learning the correspondence between images and text, CLIP Encoder can build a bridge between vision and language, so that text descriptions can effectively guide the generation and editing of images. At the same time, pre-training also enables CLIP Encoder to have good generalization capabilities, able to handle text inputs of various styles and fields, and provide robust and universal semantic representations. As a pre-trained language model, ClipText has been unsupervisedly pre-trained on large-scale text data and has learned rich language knowledge and contextual information. This pre-training enables ClipText to better understand and represent natural language text and capture the semantic relationships and contextual dependencies between words.

[0073] (2) Image Denoiser (Unet)

[0074] The image denoiser (Denosing Unet) component can be viewed as a conditional image super-resolution reconstruction, where the U-Net architecture is used to generate high-quality, high-resolution situation images based on low-resolution original images and corresponding situation description information. The low-resolution image provides the basic visual content, while the situation description information provides additional semantic guidance. The Unet network architecture is shown in Figure 1. Figure 4 shown.

[0075] The input data is first encoded by the VAE Encoder and then passed to the U-Net for processing. Specifically, the encoder of the variational autoencoder (VAE) receives the low-resolution original state image and encodes it into a compact latent vector representation (i.e., latent space representation). This latent vector captures the main features and semantic information of the image, providing the necessary prior knowledge for the subsequent restoration process.

[0076] Next, U-Net receives two parts of input: one is the latent vector generated by VAE Encoder, and the other is the situation description vector (i.e., semantic vector) generated by CLIP Encoder. CLIP Encoder converts the situation description in text form into a semantically rich vector representation, providing additional semantic guidance for the image restoration process.

[0077] In the encoder (convolution) part of U-Net, the latent vector generated by the VAE Encoder is first decoded and upsampled to reconstruct a preliminary estimate of the original image. This preliminary estimate is then fused with the situation description vector generated by the CLIP Encoder to form a joint conditional vector (i.e., the fused latent space). Fusion can be achieved through mechanisms such as concatenation, addition, or attention, with the goal of aligning and interacting semantic information with visual features. The fused conditional vector contains a joint representation of the image content and situation description, guiding the subsequent recovery process.

[0078] Next, the decoder (deconvolution) part of U-Net gradually upsamples and restores the conditional vector to generate a high-quality situation image with the same resolution as the original image. During the restoration process, the decoder of U-Net uses the multi-scale features extracted by the encoder of U-Net and the information transmitted by the jump connection to gradually restore the details and texture of the image. At the same time, the semantic guidance provided by the conditional vector helps the network generate images that conform to the description content, ensuring that the restoration result is consistent with the given situation information.

[0079] Through this architectural design, VAE Encoder and U-Net form a collaborative system. VAE Encoder is responsible for encoding low-resolution images into compact latent vectors that capture the main features and semantic information of the image. U-Net uses this latent vector and the semantic vector provided by CLIP Encoder to generate high-quality, semantically accurate high-resolution situation images through conditional image super-resolution reconstruction.

[0080] The jump connection design of U-Net also plays an important role in the situation recovery task. By directly passing the encoder's different scale features to the decoder, the jump connection helps the network retain the image's detailed information and improves the quality of restoration. At the same time, the jump connection also promotes feature reuse and gradient back propagation, accelerating the training and convergence of the network.

[0081] (3) Image Codec (VAE)

[0082] Variational Autoencoder (VAE) is a powerful generative model consisting of two parts: Encoder and Decoder. It combines the advantages of autoencoders and probabilistic graph models, and can learn compressed representations of data and generate samples similar to the original data. The VAE model structure is as follows: Figure 5 shown.

[0083] VAE consists of two main components: Encoder and Decoder. The encoder receives the low-resolution raw state image as input, extracts the features of the image through a series of convolutional layers and downsampling operations, and gradually reduces the size of the feature map. At the end of the encoding process, the encoder maps the extracted features to a low-dimensional latent space and generates two vectors: the mean vector and the standard deviation vector. These two vectors define a Gaussian distribution that represents the probability distribution of the image in the latent space. A latent vector is randomly sampled from this Gaussian distribution as a compressed representation of the image.

[0084] The goal of the VAE encoder is to map the original image to a latent space and learn a low-dimensional representation of the image. The latent space has some important properties, such as continuity, structure, and decoupling. Continuity means that similar images are close in the latent space, while different images are far away. Structure means that the latent space has a certain organization, and different latent dimensions may correspond to different attributes or features of the image. Decoupling allows specific aspects of the image, such as shape, texture, and color, to be controlled and modified independently. By mapping the image to the probability distribution of the latent space, the VAE encoder not only captures the main features and semantic information of the image, but also learns the intrinsic structure and properties of the image.

[0085] The decoder receives the sampled latent vector and maps it back to the original image space. Through a series of deconvolution layers and upsampling operations, the decoder gradually restores the latent vector to a high-resolution image. The decoder's reconstruction process is guided and constrained by the latent vector, ensuring that the generated image is visually similar to the original image and has consistent semantics and structure. The decoder extracts and interprets the key features and attributes of the image from the latent space, and cleverly combines and restores them to the complete image. By training the decoder to minimize the reconstruction error, VAE learns the mapping relationship from the latent representation to the original image and masters the ability to generate and synthesize images.

[0086] The generation process of the VAE decoder is characterized by diversity, control, and generalization. By sampling different latent vectors from the latent distribution, the decoder can generate diverse images, reflecting the inherent variation and richness of the data. By manipulating the latent vectors, specific properties of the generated image can be controlled and modified, enabling image editing and manipulation. The decoder learns a general representation of the image, can generate novel and reasonable images, and has a certain degree of generalization ability.

[0087] In the information recovery model, VAE works with CLIP encoder and U-Net. CLIP encoder provides a textual description of the situation, capturing the semantic information of the scene. VAE encoder compresses the low-resolution original situation image into a compact latent representation, learning the key features and intrinsic structure of the image. U-Net receives the latent vector generated by VAE encoder and the semantic vector generated by CLIP encoder, and through skip connections and multi-scale feature fusion, restores the compressed latent representation to a high-quality, semantically accurate high-resolution situation image under semantic guidance. Finally, VAE decoder further enhances and optimizes the image generated by U-Net, making it more outstanding in visual quality, detail expression and realism.

[0088] By seamlessly integrating VAE into the information recovery model, the model can effectively learn and utilize the low-dimensional representation of the image, capture the key features and semantic information of the image, and generate high-resolution situation images that are high-quality, diverse, and semantically accurate. This enables the model to handle complex scenes in communication-restricted scenarios, recover and generate detailed situation images, and provide commanders with more accurate and comprehensive situation understanding and decision support.

[0089] After completing the information recovery model construction, define the model interface function to implement the model call. The interface definition is as follows:

[0090]

[0091] In summary, the present invention provides a text generation and information recovery method based on a large model. The method uses large model technology to achieve low-bandwidth and high-real-time communication requirements for the problem of situation image transmission between communication units in special environments with limited communication conditions. The method consists of two key components: a text generation model and an information recovery model; the text generation model aims to convert complex situation information into concise and accurate text descriptions. By utilizing natural language processing technology and the semantic understanding ability of the visual large model, the module can automatically extract key information in the situation image, such as target category, motion state, etc., and generate corresponding text descriptions. This text description is not only semantically rich, but also has a small amount of data, which is very suitable for transmission in a low-bandwidth environment; the information recovery model is responsible for reconstructing high-quality, semantically accurate situation images from the received text description. The module adopts image compression and generation technology, such as variational autoencoder (VAE), and realizes the transformation from compact text description to realistic situation image by learning low-dimensional representation and generation model of images. The generated image not only has high visual quality, but also is consistent with the original situation information at the semantic level, ensuring the accuracy of situation perception. Therefore, compared with the prior art, the text generation and information recovery method based on the large model provided by the present invention can, on the one hand, adaptively adjust the compression rate according to network conditions, adapt to the elastic network conditions in special environments, and effectively ensure the real-time communication; on the other hand, due to the use of the information recovery model, the consistency of the sent and received information is guaranteed to the maximum extent.

[0092] For ease of description, spatially relative terms such as "above", "above", "on the upper surface of", "above", etc. may be used here to describe the spatial positional relationship between a device or feature and other devices or features as shown in the figure. It should be understood that spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation described in the figure. For example, if the device in the accompanying drawings is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be positioned as "below other devices or structures" or "below other devices or structures". Thus, the exemplary term "above" can include both "above" and "below". The device can also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatially relative descriptions used here are interpreted accordingly.

[0093] In addition, it should be noted that the use of terms such as "first" and "second" to limit components is only for the convenience of distinguishing the corresponding components. If not otherwise stated, the above terms have no special meaning and therefore cannot be understood as limiting the scope of protection of the present invention.

[0094] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A text generation and information recovery method based on a large model, characterized in that: The text generation and information recovery method based on the large model includes: Use text generation models to transform complex situation information into concise and accurate text descriptions; The information recovery model is used to reconstruct a high-quality, semantically accurate situation image from the received text description. In the information recovery stage, the text encoder of the pre-trained CLIP model is used to convert the text description of the situation information into a semantic vector to capture the semantic information of the text; the encoder of the variational autoencoder (VAE) is used to encode the noisy image into a latent space representation, and the compressed representation of the image is learned; the semantic vector generated by the pre-trained CLIP model is fused with the image latent space representation generated by the encoder of the variational autoencoder (VAE), and the trained denoising Unet network is used to optimize the fused latent space representation; the decoder of the variational autoencoder (VAE) is used to decode the optimized latent space representation into a high-quality situation information image.

2. The method for text generation and information recovery based on a large model according to claim 1, characterized in that: The text generation model uses the latest multimodal version of the MiniCPM series, MiniCPM-V 2.

0. The model is built based on MiniCPM 2.4B and SigLip-400M and has a total of 2.8B parameters. MiniCPM-V 2.0 has leading optical character recognition (OCR) and multimodal understanding capabilities.

3. The method for text generation and information recovery based on a large model according to claim 1, characterized in that: The information recovery model is constructed by referring to the Stable Diffusion image generation model. The information recovery model includes a semantic extractor (Text Extraction), an image denoiser (Unet) and an image encoder / decoder (VAE). The semantic extractor (Text Extraction) is responsible for processing the input text information and converting it into corresponding TokenEmbeddings (token embeddings). The ClipText model developed and implemented by OpenAI is used as the main tool for semantic extraction. The CLIP Encoder is pre-trained on a large-scale image-text pair dataset. In the information recovery model, the CLIP Encoder receives the input situation text description and converts it into a fixed-length vector representation.

4. The method for text generation and information recovery based on a large model according to claim 3 is characterized in that: The encoder of the variational autoencoder (VAE) is used to encode the noisy image into a latent space representation. The compressed representation of the learning image specifically includes: Using an image compression algorithm to reduce the resolution of the original information image collected by the data sender to obtain a compressed low-resolution image; This low-resolution image is encoded into a latent space representation using the encoder of a variational autoencoder (VAE) to learn a compressed representation of the image.

5. The method for text generation and information recovery based on a large model according to claim 1, characterized in that: The semantic vector generated by the pre-trained CLIP model is fused with the image latent space representation generated by the encoder of the variational autoencoder (VAE), and the fused latent space representation is optimized using the trained denoising Unet network. Specifically, U-Net receives two parts of input, one is the latent space representation generated by the encoder of the variational autoencoder (VAE), and the other is the semantic vector generated by the CLIP Encoder; In the encoder (CNN) part of U-Net, the latent space representation generated by the encoder of the variational autoencoder (VAE) is first decoded and upsampled to reconstruct a preliminary estimate of the original image; Fusing the initial estimate of the original image with the semantic vector generated by CLIP Encoder to form a fused latent space; The decoder (deconvolution) part of U-Net gradually upsamples and restores the fused latent space to generate a high-quality situation image with the same resolution as the original image.

6. The method for text generation and information recovery based on a large model according to claim 5, characterized in that: The initial estimate of the original image is fused with the semantic vector generated by CLIP Encoder through concatenation, addition or attention.

7. The method for text generation and information recovery based on a large model according to claim 6, characterized in that: The fused latent space contains a joint representation of image content and situation description, guiding the subsequent recovery process.

8. The method for text generation and information recovery based on a large model according to claim 7, characterized in that: In the process of restoring the fused latent space, the U-Net decoder uses the multi-scale features extracted by the U-Net encoder and the information transmitted by the jump connection to gradually restore the details and texture of the image. The semantic guidance provided by the conditional vector helps the network generate images that conform to the description content, ensuring that the restoration results are consistent with the given situation information.

9. The method for text generation and information recovery based on a large model according to claim 8, characterized in that: Using the VAE decoder to decode the optimized latent space representation into a high-quality situation information image specifically includes: using the VAE decoder to extract and interpret the key features and attributes of the image from the latent space through deconvolution layers and upsampling operations, and decoding the optimized latent space representation into a high-quality situation information image.

10. A text generation and information recovery system based on a large model, characterized in that: The large model-based text generation and information recovery system uses the large model-based text generation and information recovery method described in any one of claims 1 to 9 to perform text generation and information recovery.

Images