Semantic communication system based on zero sample learning

Abstract

The invention is applicable to the technical field of semantic communication, and particularly relates to a semantic communication system based on zero sample learning, the system comprises a sending end, a receiving end and a database, and by combining data of various modes of communication images, audios and texts and by utilizing the advantages of CLIP and CLAP multi-mode models, the semantic communication system is established. Images and audios are mapped to a shared text coding vector space, so that robust semantic understanding is promoted, the method is suitable for being applied to a complex marine environment with poor communication conditions, data of various sensors on the USV can be processed and analyzed in real time, and the method is suitable for popularization and application. The integration of the dynamically updated knowledge database ensures continuous learning and performance improvement and can explain new and invisible data, reduces the dependence on a large tagged data set, and solves the limitation of a traditional semantic perception system, thereby improving the reliability and efficiency under severe communication conditions.

Description

Technical Field

[0001] The present invention belongs to the technical field of semantic communication, and in particular relates to a semantic communication system based on zero-sample learning. Background Art

[0002] Existing semantic communication technologies usually rely on the processing and fusion of multimodal data to achieve semantic transmission of information. To ensure the complete transmission of semantic information, the system usually combines multimodal data such as text, images and audio. NLP technology is responsible for extracting core semantics from text data, audio processing technology is used to analyze speech features, and computer vision technology obtains the key content of image information, thereby effectively improving the accuracy and flexibility of semantic communication.

[0003] As a core module, multimodal data fusion usually adopts feature-level fusion and decision-level fusion to weight the features of each modality or unify them through deep learning networks (such as CNN and RNN) to improve the robustness of information transmission in complex environments. Existing solutions also achieve adaptability to dynamic scenarios through transfer learning and model fine-tuning, so as to transfer previously learned features in a changing communication environment, thereby ensuring information coherence and accuracy in different communication scenarios.

[0004] To protect privacy and data security, existing systems localize data processing as much as possible and use encrypted transmission to reduce the risk of data leakage. Some solutions further apply distributed learning and data encryption. In the design of semantic transmission protocols, many systems are based on standard network communication protocols and enhance the semantic understanding module, so that the receiving end can restore the semantic content of the sending end, effectively reducing information redundancy and improving bandwidth utilization. However, this protocol usually relies on a fixed semantic tag library and requires a large amount of training resources, and has limited adaptability.

[0005] Current semantic communication systems also rely on deep learning techniques to improve efficiency by training and leveraging a shared understanding or background knowledge database between the sender and receiver. These basic knowledge databases include empirical data and corresponding semantic information, which are essential for accurately interpreting and processing transmitted data. However, due to the dynamic nature of data and the frequent appearance of unfamiliar object types, the training of semantic communication models based on deep learning is heavily dependent on a large amount of labeled data.

[0006] Although existing semantic communication technologies can effectively transmit multimodal information, they still have limitations in practical applications. First, most systems rely on a large amount of labeled data for model training, which makes it difficult to ensure semantic understanding and transmission in areas where data is scarce. In addition, current systems lack zero-sample processing capabilities and are usually unable to effectively cope with tasks in new concepts or unfamiliar areas. When there is no prior data, the system's semantic recognition and communication are limited and lack adaptability.

[0007] Moreover, semantic communication based on traditional supervised learning methods heavily relies on annotated datasets, which are difficult and time-consuming to obtain, especially in large-scale real-time communication scenarios. This dependence poses a huge challenge to the scalability and flexibility of communication systems.

[0008] The data fusion strategies of existing technologies have poor stability in dynamic environmental changes. The weights of multimodal data are usually set under fixed conditions and cannot be flexibly adjusted, resulting in reduced robustness of communication in different environments and difficulty in maintaining efficient semantic transmission. These defects affect the actual performance of semantic communication systems in complex scenarios. Summary of the invention

[0009] The purpose of the present invention is to provide a semantic communication system based on zero-sample learning, aiming to solve the problem that the data fusion strategy in the prior art has poor stability in the dynamic changes of the environment, and the weights of multimodal data are usually set under fixed conditions and cannot be flexibly adjusted, resulting in reduced robustness of communication in different environments and difficulty in maintaining efficient semantic transmission.

[0010] The present invention is implemented as follows: a semantic communication system based on zero-sample learning, the system comprising: a sending end, a receiving end and a database, the sending end is used to collect multimodal data, the multimodal data at least includes an image I and an audio A, the multimodal data is mapped to different vector spaces, the multimodal data is mapped to corresponding text features, and a corresponding text description is generated, and the sending end is used to send the text description to the receiving end;

[0011] The receiving end includes a zero-shot learning module and a semantic decoder. The receiving end extracts feature information from the text description through the semantic decoder and queries the database based on the extracted feature information to generate data transmission results.

[0012] When there is no matching item in the database, the zero-sample learning module is called to generate data transmission results based on the feature information in the text description.

[0013] Preferably, the transmitting end uses the CLAIP multimodal model to preprocess the multimodal data, and uses the image encoding function I1=E I (I) and the audio encoding function A1 = E A (A), the input image I and audio A are processed simultaneously and mapped to different vector spaces I1 and A1.

[0014] Preferably, the sending end is provided with a local knowledge base, and after the image and audio are mapped to the vector space, the local knowledge base is queried, the text in the local knowledge base is mapped to the vector space through a text semantic encoding function, and the similarity between the image and audio and the text in the local knowledge base is calculated.

[0015] Preferably, the similarity is characterized by cosine similarity:

[0016]

[0017] Among them, V T Represents the text features in the vector space, V M represents the modal feature corresponding to mode M, CS T,M Represents the cosine similarity between the text and the modality M.

[0018] Preferably, by setting a distance threshold, when the distance between the input vector corresponding to the multimodal data in the vector space and the existing vector corresponding to the text in the local knowledge base exceeds the distance threshold, the API of the multimodal model is called to perform description or retrieval generation.

[0019] Preferably, when the similarity reaches a predefined threshold, it is determined that a matching text description has been found in the local knowledge base:

[0020] min(CS T,I ,CS T,A )≥threshold;

[0021] Among them, CS T,I Represents the similarity between text information T and image modality I, CS T,A Represents the similarity between text information T and audio modality A.

[0022] Preferably, the sending end includes a semantic encoder and an integrated dynamic knowledge base, and the semantic encoder extracts semantic features from source input data and refines high-level semantic information.

[0023] Preferably, the zero-shot learning module processes the received semantic representation and dynamic data set to generate semantic embeddings, and the semantic decoder reconstructs the semantic content as output.

[0024] Preferably, the sending end includes a first database D s , the receiving end includes a second database D r , the first database D s and the second database D r Both contain semantic image pairs. The first database D s Contains pairs of semantic descriptions and their corresponding images N s Represents the total number of pairs, the second database D r The definition is similar to this, containing pairs of semantic descriptions r j and its corresponding image Where N r Represents the total number of pairs, Ns <N r .

[0025] Preferably, when there is no matching item in the database, the zero-shot learning module is called up, and in the step of generating the data transmission result based on the feature information in the text description, the receiving end decodes through the semantic decoder, processes the transmission of the unseen object or the new data that does not belong to the training set, and uses the generative model G to generate the description D of the unseen object according to the semantic embedding:

[0026] D=G(Z I ,Z A );

[0027] Where Z I and Z A The CLIP and CLAP models are used to convert image and audio data into semantic embeddings in a shared vector space. In this process, the data features are consistent with the semantic features of unseen categories in the shared semantic space, thereby generating reconstructed data D.

[0028] The present invention provides a semantic communication system based on zero-shot learning, which combines the image, audio and text data of multiple modalities of communication, and maps the image and audio to a shared text encoding vector space by utilizing the advantages of CLIP and CLAP multimodal models, thereby promoting robust semantic understanding. The system is suitable for application in complex marine environments with poor communication conditions, and can process and analyze data from various sensors on USVs in real time. The integration of a dynamically updated knowledge database ensures continuous learning and performance improvement and can interpret new and unseen data, reducing dependence on large labeled data sets and addressing the limitations of traditional semantic perception systems, thereby improving reliability and efficiency under harsh communication conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of the principle of a semantic communication system based on zero-sample learning provided by an embodiment of the present invention;

[0030] Figure 2 A workflow diagram of a semantic communication system based on zero-sample learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0032] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first xx script may be referred to as a second xx script, and similarly, a second xx script may be referred to as a first xx script.

[0033] like Figure 1 and Figure 2 As shown, a semantic communication system based on zero-sample learning is provided in an embodiment of the present invention, the system comprising: a sending end, a receiving end and a database, the sending end is used to collect multimodal data, the multimodal data at least includes an image I and an audio A, the multimodal data is mapped to different vector spaces, the multimodal data is mapped to corresponding text features, and a corresponding text description is generated, and the sending end is used to send the text description to the receiving end;

[0034] The receiving end includes a zero-shot learning module and a semantic decoder. The receiving end extracts feature information from the text description through the semantic decoder and queries the database based on the extracted feature information to generate data transmission results.

[0035] When there is no matching item in the database, the zero-sample learning module is called to generate data transmission results based on the feature information in the text description.

[0036] In this embodiment, the working process of the semantic communication system based on zero-shot learning includes several key stages, including preprocessing, input and encoding at the sending end, and decoding, output, retrieval and generation at the receiving end. At the sending end, a multimodal model based on CLAIP (CLIP-CLAP) is used for effective information preprocessing, which greatly enhances the system's ability to manage various input types. Specific application examples include Figure 1 shown.

[0037] At the sending end, an effective information preprocessing method based on the CLAIP multimodal model is used, which greatly improves the system's ability to manage various input types. The CLAIP model combines the CLIP (Contrastive Language-Image Pretraining) and CLAP (Contrastive Language-Audio Pretraining) models to support semantic parsing and efficient transmission of multimodal input. The following will explain in detail the application of CLIP and CLAP in the system.

[0038] The CLIP model is mainly used for semantic association mapping between image I and text. I(I), CLIP can map image data to feature vector space I1. CLIP uses large-scale training data of image-text pairs to capture and associate the semantic information of visual data to the text vector space during the learning process, achieving alignment between image content and text description. In this system, the CLIP model is responsible for converting the original image into a semantic vector representation, so that the image information can convey its meaning in an unsupervised scenario. This mapping is also connected to the local knowledge base of the communicating parties, supporting the system to identify the neighbor relationship between images and texts, thereby accurately matching the relevant semantic descriptions.

[0039] The CLAP model focuses on multimodal comparative pre-training of audio and text. A (A), CLAP maps audio data to the audio vector space A1. This encoding process not only abstracts the sound features in the audio, but also represents the audio content at the semantic level, so that it can be aligned with the corresponding text in the vector space. At the sending end, the CLAP model converts the original audio into a vector representation and matches the closest text description in the local knowledge base through a nearest neighbor search, so that the audio semantic information can be directly used in the communication data.

[0040] In this system, the input image I and audio A are processed simultaneously so that they can be mapped to different vector spaces I1 and A1. This mapping is done by the image encoding function I1 = E I (I) and the audio encoding function A1 = E A (A) is achieved. Here E I and E A represents the encoding function that transforms the original input into the corresponding vector representation. This dual mapping process not only enriches the semantic understanding of the two modalities, but also ensures their effective integration and utilization within the communication framework. The local knowledge base L on both communicating parties plays a key role in this architecture, which helps map the local text information T to the vector space V T This setting supports efficient nearest neighbor search, enabling the system to identify the most appropriate text description corresponding to the multimodal input data. This process can be expressed as V T =E T (T), where E T is a text semantic encoding function that maps text to vector space. Then, the system uses cosine similarity to calculate the similarity between text and multimodal input through nearest neighbor search:

[0041]

[0042] Among them, V T Represents the text features in the vector space, V Mrepresents the modality features corresponding to the modality M (such as image or audio), CS T,M Represents the cosine similarity between the text and the modality M.

[0043] A distance threshold is set in the vector space. When the distance between the input vector and the existing vector in the database exceeds this threshold, the system calls the API of the multimodal model to retrieve or generate descriptions. When the similarity between text-image and text-audio reaches a predefined threshold, the system considers that a matching text description has been found:

[0044] min(CS T,I ,CS T,A )≥threshold;

[0045] Among them, CS T,I Represents the CS value between text information T and image modality I, CS T,A Represents the CS value between the text information T and the audio modality A. This condition ensures that the minimum similarity between the text and any modality reaches or exceeds the predetermined threshold, thereby improving the efficiency of multimodal matching.

[0046] The system integrates the transmitter and receiver structures in the sending and receiving ends. The transmitter in our zero-shot learning-based semantic communication system model includes a semantic encoder and an integrated dynamic knowledge database, which is responsible for processing and sending semantic information. The semantic encoder E takes input data X from the source in Semantic features are extracted from the data and high-level semantic information S is refined to ensure that the transmitted data is semantically rich and adaptable. Subsequently, as new categories are identified by zero-shot learning, the knowledge database will be continuously updated and integrated into the processing of the transmitter, enabling semantic communication to adapt to new data types and scenarios in low-bandwidth communication scenarios.

[0047] The receiver includes a zero-shot learning module and a semantic decoder, which are responsible for interpreting and reconstructing the transmitted semantic information. The zero-shot learning module Z processes the received semantic representation S and the dynamic dataset to generate a semantic embedding T. Then, the semantic decoder reconstructs the semantic content as output. The receiver integrates these processes to ensure accurate interpretation and reconstruction of the transmitted semantic content, including previously unseen data types.

[0048] In the present invention, two databases are used: the sender's (first database) D s and the receiver's (second database) D r , each database contains semantic image pairs. The sender’s database D s Contains pairs of semantic descriptions s i and its corresponding image Where N s Represents the total number of pairs. Receiver database Dr The definition is similar to this, containing pairs of semantic descriptions r j and its corresponding image Where N r Represents the total number of pairs. It is worth noting that the sender’s database is smaller than the receiver’s, as shown by N s <N r . This indicates that the receiver has more extensive background information and reference data to support decoding and rereading. This database design allows the system to interpret and transmit previously unseen semantic information without the need for explicit grammar. Our system leverages this database structure to enable zero-shot learning, enabling the receiver to accurately interpret the missing semantic information in the sender's database through reasoning and mapping mechanisms. This process continuously enriches the receiver's database, enhancing the system's robustness and adaptability in dealing with unknown semantic content.

[0049] When outputting data, two methods are used: cloud retrieval and generation. The cloud retrieval method is to directly retrieve and display the retrieval results if the corresponding feature data previously generated by zero-shot learning exists in the knowledge database.

[0050] The other is to use zero-shot learning capabilities to generate audio or image data. The receiving end extracts the feature information of the original audio or image data x through the semantic encoder, and decodes it with a semantic decoder with zero-shot learning capabilities to handle the transmission of new data that has not been seen or does not belong to the training set. The generative model G is used to generate a meaningful description D of the unseen object based on the semantic embedding:

[0051] D=G(Z I ,Z A );

[0052] Where Z I and Z A The CLIP and CLAP models are used to convert image and audio data into semantic embeddings in a shared vector space. In this process, the data features are consistent with the semantic features of unseen categories in the shared semantic space, thereby generating reconstructed data (i.e., meaningful description) D. This generation method enables the system to adapt to new inputs without a lot of retraining, and is suitable for dynamic narrowband communication environments where new objects or new situations may often appear.

[0053] In actual communication scenarios, the system may encounter an unfamiliar entity u. If the sender's knowledge base contains entity u, only the relevant text information T is transmitted, thereby minimizing the amount of data. In the system, the pre-trained classification model Φ is adjusted to adapt to the application of zero-shot learning capabilities in semantic communication systems. The specific adjustment is expressed as:

[0054] Φ=(F,Ws );

[0055] Where F is the feature extractor, Φ is the pre-trained classification model, and is the weight matrix of the classifier. With this adjustment, the system is able to classify unseen images without training images, ensuring accurate understanding of semantics in dynamic environments. At the same time, by integrating zero-shot learning and multimodal data acquisition capabilities, the system is able to accurately convey semantic information in low-bandwidth scenarios that require the combination of auditory and visual data, even in unfamiliar low-bandwidth communication entities. This integration greatly improves communication efficiency and accuracy in complex and diverse communication environments.

[0056] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A semantic communication system based on zero-shot learning, characterized in that: The system includes: a sending end, a receiving end and a database, the sending end is used to collect multimodal data, the multimodal data at least includes an image I and an audio A, maps the multimodal data to different vector spaces, maps the multimodal data to corresponding text features, generates corresponding text descriptions, and the sending end is used to send the text descriptions to the receiving end; The receiving end includes a zero-shot learning module and a semantic decoder. The receiving end extracts feature information from the text description through the semantic decoder and queries the database based on the extracted feature information to generate data transmission results. When there is no matching item in the database, the zero-sample learning module is called to generate data transmission results based on the feature information in the text description.

2. The semantic communication system based on zero-shot learning according to claim 1, characterized in that: The sending end uses the CLAIP multimodal model to preprocess the multimodal data, and uses the image encoding function I1=E I (I) and the audio encoding function A1 = E A (A), the input image I and audio A are processed simultaneously and mapped to different vector spaces I1 and A1.

3. The semantic communication system based on zero-shot learning according to claim 2, characterized in that: The sending end is provided with a local knowledge base. After the image and audio are mapped to the vector space, the local knowledge base is queried, the text in the local knowledge base is mapped to the vector space through a text semantic encoding function, and the similarity between the image and audio and the text in the local knowledge base is calculated.

4. The semantic communication system based on zero-shot learning according to claim 3, characterized in that: The similarity is characterized by cosine similarity: Among them, V T Represents the text features in the vector space, V M represents the modal feature corresponding to mode M, CS T,M Represents the cosine similarity between the text and the modality M.

5. The semantic communication system based on zero-shot learning according to claim 4, characterized in that: By setting a distance threshold, when the distance between the input vector corresponding to the multimodal data in the vector space and the existing vector corresponding to the text in the local knowledge base exceeds the distance threshold, the API of the multimodal model is called for description or retrieval generation.

6. The semantic communication system based on zero-shot learning according to claim 4, characterized in that: When the similarity reaches a predefined threshold, it is determined that a matching text description has been found in the local knowledge base. min(CS T,I ,CS T,A )≥threshold; Among them, CS T,I Represents the similarity between text information T and image modality I, CS T,A Represents the similarity between text information T and audio modality A.

7. The semantic communication system based on zero-shot learning according to claim 1, characterized in that: The sending end includes a semantic encoder and an integrated dynamic knowledge base. The semantic encoder extracts semantic features from source input data and refines high-level semantic information.

8. The semantic communication system based on zero-shot learning according to claim 1, characterized in that: The zero-shot learning module processes the received semantic representation and dynamic dataset to generate semantic embeddings, and the semantic decoder reconstructs the semantic content as output.

9. The semantic communication system based on zero-shot learning according to claim 1, characterized in that: The sending end includes a first database D s , the receiving end includes a second database D r , the first database D s and the second database D r Both contain semantic image pairs. The first database D s Contains pairs of semantic descriptions and their corresponding images N s Represents the total number of pairs, the second database D r The definition is similar to this, containing pairs of semantic descriptions r j and its corresponding image Where N r Represents the total number of pairs, N s <N r .

10. The semantic communication system based on zero-shot learning according to claim 1, characterized in that: When there is no match in the database, the zero-shot learning module is called to generate the data transmission result based on the feature information in the text description. The receiving end decodes it through the semantic decoder to process the transmission of new data that has not been seen or does not belong to the training set. The generative model G is used to generate the description D of the unseen object based on the semantic embedding: D=G(Z I ,Z A ); Where Z I and Z A The CLIP and CLAP models are used to convert image and audio data into semantic embeddings in a shared vector space. In this process, the data features are consistent with the semantic features of unseen categories in the shared semantic space, thereby generating reconstructed data D.

Images