Satellite-based multi-modal semantic communication method and device and storage medium

Abstract

The invention discloses a multi-mode semantic communication method and device based on a satellite and a storage medium, and the method comprises the steps: a first gateway station receives a plurality of pieces of first data information, carries out the semantic feature extraction, and generates a plurality of corresponding first semantic features; the first gateway station generates corresponding first semantic data based on the plurality of first semantic features and according to a preset first knowledge base; the first gateway station generates corresponding second semantic data and generates corresponding channel transmission data; the second gateway station receives the channel transmission data sent by the satellite, performs channel decoding to generate corresponding third semantic data, and performs semantic decoding to generate corresponding fourth semantic data; the second gateway station generates a plurality of corresponding second semantic features by using the second knowledge base and based on the fourth semantic data; and the second gateway station performs semantic information reconstruction on the plurality of second semantic features so as to generate second data information.

Description

Technical Field

[0001] The present application relates to the field of satellite communication technology, and in particular to a satellite-based multimodal semantic communication method, device and storage medium. Background Art

[0002] With the continuous development of communication technology and the pursuit of seamless coverage communication, a communication method that integrates satellite and 5G has been proposed. The transmitter transmits data information to the gateway through the ground network, and the gateway transmits the data information to the satellite through the feed uplink. Then the satellite transmits the data information to the gateway through the feed downlink, and finally the gateway transmits the data information to the receiver.

[0003] In the entire communication process mentioned above, the communication method used is transmission. Among them, the traditional communication method is a bit-level error-free transmission method, that is, all data information from the sender is transmitted to the receiver. However, since there is often redundant information in the data information, and this redundant information is not necessary for the user's understanding and needs, the traditional communication method not only wastes the satellite's channel resources, but also affects the data transmission rate.

[0004] With regard to the technical problem existing in the above-mentioned prior art that redundant information often exists in data information, using traditional communication methods to transmit data information not only wastes satellite channel resources but also affects the data transmission rate, no effective solution has been proposed so far. Summary of the invention

[0005] The embodiments of the present disclosure provide a satellite-based multimodal semantic communication method and device, namely, a storage medium, to at least solve the technical problem existing in the prior art that redundant information often exists in data information, and therefore, using traditional communication methods to transmit data information not only wastes satellite channel resources, but also affects the data transmission rate.

[0006] According to one aspect of an embodiment of the present disclosure, a satellite-based multimodal semantic communication method is provided, including: a first gateway receives a plurality of first data information corresponding to a transmitting end, extracts semantic features from the plurality of first data information, and generates a plurality of corresponding first semantic features, wherein the plurality of first data information corresponds to different modalities; the first gateway generates corresponding first semantic data based on the plurality of first semantic features and according to a pre-set first knowledge base; the first gateway semantically encodes the first semantic data to generate corresponding second semantic data, and channel encodes the second semantic data to generate corresponding channel transmission data; a second gateway receives the channel transmission data sent by the satellite, channel-decodes the channel transmission data to generate corresponding third semantic data, and semantically decodes the third semantic data to generate corresponding fourth semantic data; the second gateway generates a plurality of corresponding second semantic features based on the pre-set second knowledge base and based on the fourth semantic data; and the second gateway reconstructs semantic information on the plurality of second semantic features to generate second data information, wherein the second data information corresponds to the first data information.

[0007] According to another aspect of an embodiment of the present disclosure, a storage medium is further provided, the storage medium including a stored program, wherein when the program is running, a processor executes any one of the methods described above.

[0008] According to another aspect of the embodiment of the present disclosure, a satellite-based multimodal semantic communication device is also provided, including: a semantic feature extraction module, used for a first gateway to receive a plurality of first data information corresponding to a transmitting end, extract semantic features from the plurality of first data information, and generate a plurality of corresponding first semantic features, wherein the plurality of first data information corresponds to different modalities; a first semantic data generation module, used for the first gateway to generate corresponding first semantic data based on the plurality of first semantic features and according to a pre-set first knowledge base; a semantic encoding module, used for the first gateway to semantically encode the first semantic data, generate corresponding second semantic data, and encode the first semantic data into corresponding second semantic data; The invention relates to a method for transmitting the first data packet to the second gateway station through a satellite, performing channel coding on the second semantic data to generate corresponding channel transmission data; a semantic decoding module, which is used for the second gateway station to receive the channel transmission data sent by the satellite, channel decode the channel transmission data to generate corresponding third semantic data, and semantically decode the third semantic data to generate corresponding fourth semantic data; a semantic feature generation module, which is used for the second gateway station to generate corresponding multiple second semantic features based on the fourth semantic data by using a pre-set second knowledge base; and a semantic information reconstruction module, which is used for the second gateway station to reconstruct semantic information on the multiple second semantic features, thereby generating second data information, wherein the second data information corresponds to the first data information.

[0009] According to another aspect of an embodiment of the present disclosure, a satellite-based multimodal semantic communication device is also provided, including: a processor; and a memory, connected to the processor, for providing the processor with instructions for processing the following processing steps: a first gateway receives a plurality of first data information corresponding to a transmitting end, extracts semantic features from the plurality of first data information, and generates a plurality of corresponding first semantic features, wherein the plurality of first data information corresponds to different modalities; the first gateway generates corresponding first semantic data based on the plurality of first semantic features and according to a first knowledge base set in advance; the first gateway performs semantic encoding on the first semantic data to generate corresponding second semantic data, and performs channel encoding on the second semantic data to generate corresponding channel transmission data; the second gateway receives the channel transmission data sent by the satellite, performs channel decoding on the channel transmission data to generate corresponding third semantic data, and performs semantic decoding on the third semantic data to generate corresponding fourth semantic data; the second gateway generates a plurality of corresponding second semantic features based on the fourth semantic data using a second knowledge base set in advance; and the second gateway performs semantic information reconstruction on the plurality of second semantic features to generate second data information, wherein the second data information corresponds to the first data information.

[0010] The present application discloses a satellite-based multimodal semantic communication method. First, the first gateway receives multiple first data information corresponding to the transmitter, extracts semantic features from the multiple first data information, and generates corresponding multiple first semantic features. Then, the first gateway generates corresponding first semantic data based on the multiple first semantic features and according to a pre-set first knowledge base. Further, the first gateway semantically encodes the first semantic data to generate corresponding second semantic data, and performs channel encoding on the second semantic data to generate corresponding channel transmission data. Afterwards, the second gateway receives the channel transmission data sent by the satellite, performs channel decoding on the channel transmission data to generate corresponding third semantic data, and performs semantic decoding on the third semantic data to generate corresponding fourth semantic data. Then, the second gateway generates corresponding multiple second semantic features based on the pre-set second knowledge base and the fourth semantic data. Finally, the second gateway reconstructs the semantic information of the multiple second semantic features to generate second data information.

[0011] Referring to the above content, it can be seen that the first gateway in the present application does not transmit all the first data information to the satellite based on the traditional communication method, but first extracts semantic features from the first data information, and uses a pre-set first knowledge base to abstract the first semantic features into first semantic data, thereby transmitting the first semantic data using the satellite.

[0012] Similarly, after receiving the third semantic data (corresponding to the first semantic data), the second gateway in the present application uses a pre-set second knowledge base to restore the third semantic data into a second semantic feature, and reconstructs the semantic information of the second semantic feature, thereby ultimately generating second data information corresponding to the first data information.

[0013] That is, unlike the prior art, the present application does not transmit redundant information in the first data information during the communication transmission process, but only transmits semantic information that is easy for the receiving end to understand, thereby greatly reducing the waste of satellite channel resources and improving data transmission efficiency.

[0014] In addition, in actual application scenarios, the sender often sends not only data information of a single modality, but data information of different modalities. Therefore, after the first gateway extracts semantic features for data in different modalities, it can use a cross-modal encoder to map the data information of different modalities to a unified representation space, thereby facilitating subsequent data processing. Similar to the above, after receiving the second semantic features of the unified representation space, the second gateway can use a generative decoder to generate second data information in different modalities based on the second semantic features, thereby facilitating the receiving end to view the second data information in different modalities.

[0015] Thus, the waste of satellite channel resources is reduced and the data transmission rate is increased. This solves the technical problem in the prior art that redundant information often exists in data information, so using traditional communication methods to transmit data information not only wastes satellite channel resources but also affects the data transmission rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present application. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure. In the drawings:

[0017] Figure 1 is a schematic diagram of a satellite communication system according to Embodiment 1 of the present application;

[0018] Figure 2A is a schematic diagram of the hardware architecture of the satellite according to Embodiment 1 of the present application;

[0019] Figure 2B is a schematic diagram of the hardware architecture of the first gateway and the second gateway according to Embodiment 1 of the present application;

[0020] Figure 3 This is a system module diagram of satellite-based multimodal semantic communication according to Example 1 of the present application;

[0021] Figure 4 is a flowchart of the satellite-based multimodal semantic communication method according to Example 1 of the present application;

[0022] Figure 5 is a schematic diagram of a device for satellite-based multimodal semantic communication according to Example 2 of the present application;

[0023] Figure 6 It is a schematic diagram of a device for satellite-based multimodal semantic communication according to Example 3 of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only embodiments of a part of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present disclosure.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0026] Example 1

[0027] According to this embodiment, a method embodiment of satellite-based multimodal semantic communication is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0028] Figure 1The schematic diagram of the satellite communication system according to the present embodiment is shown. The system includes: a transmitting end 10, a first gateway 20, a satellite 30, a second gateway 40 and a receiving end 50. The transmitting end 10 may be, for example, a terminal device corresponding to a first user. The first gateway 20 is connected to the transmitting end 10 for communication and is used to receive the first data information sent by the transmitting end 10. The first data information may be data information in a single mode or in different modes, which is not specifically limited here.

[0029] The first gateway 20 is in communication connection with the second gateway 40 via the satellite 30 , and is used to send the second semantic data (ie, semantic information corresponding to the first data information) to the second gateway 40 .

[0030] The second gateway 40 is communicatively connected to the receiving end 50, and when receiving the third semantic data (corresponding to the second semantic data), the second gateway 40 converts the third semantic data into second data information corresponding to the first data information, and sends the second data to the receiving end 40. The receiving end 40 may be, for example, a terminal device corresponding to the second user.

[0031] Figure 2A It further shows Figure 1 Schematic diagram of the hardware architecture of China Satellite 30. Figure 2A As shown, satellite 30 includes an integrated electronic system, which includes: a processor, a memory, a bus management module and a communication interface. The memory is connected to the processor, so that the processor can access the memory, read the program instructions stored in the memory, read data from the memory or write data to the memory. The bus management module is connected to the processor and is also connected to a bus such as a CAN bus. Therefore, the processor can communicate with the satellite-borne peripherals connected to the bus through the bus managed by the bus management module. In addition, the processor is also connected to communication devices such as cameras, star sensors, measurement and control transponders, and data transmission equipment via the communication interface. It can be understood by those skilled in the art that Figure 2A The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 2A More or fewer components as shown, or with Figure 2A Different configurations are shown.

[0032] Figure 2B It further shows Figure 1 FIG. 4 is a schematic diagram of the hardware architecture of the first gateway station 20 and the second gateway station 40. Figure 2BAs shown, the first gateway 20 and the second gateway 40 may include one or more processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory for storing data, a transmission device for communication functions, and an input / output interface. The memory, the transmission device, and the input / output interface are connected to the processor via a bus. In addition, a display, a keyboard, and a cursor control device connected to the input / output interface may also be included. A person skilled in the art can understand that Figure 2B The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 2B More or fewer components as shown, or with Figure 2B Different configurations are shown.

[0033] It should be noted that Figure 2A and Figure 2B The one or more processors and / or other data processing circuits shown in the figure may generally be referred to as "data processing circuitry" herein. The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other elements in the computing device. As involved in the embodiments of the present disclosure, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0034] Figure 2A and Figure 2B The memory shown in the figure can be used to store software programs and modules of application software, such as the program instruction / data storage device corresponding to the satellite-based multimodal semantic communication method in the embodiment of the present disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, the satellite-based multimodal semantic communication method of the above-mentioned application program is realized. The memory may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory.

[0035] It should be noted that, in some optional embodiments, the above Figure 2A and Figure 2B The devices shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware elements and software elements. Figure 2A and Figure 2B This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the above described apparatus.

[0036] Figure 3 is a system diagram of a satellite-based multimodal semantic communication method according to an embodiment of the present application. Figure 3 As shown, the first gateway 20 includes a semantic feature extraction module, a semantic encoder and a channel encoder. The semantic feature extraction module is connected to the semantic encoder for performing semantic feature extraction on the first data information in different modes sent by the transmitter 10 and generating corresponding first semantic data. The semantic encoder is connected to the channel encoder for performing semantic encoding on the first semantic data and generating second semantic data. The channel encoder is used to perform channel encoding on the second semantic data and generate corresponding channel transmission data.

[0037] In addition, the semantic feature extraction module includes a cross-modal encoder and a first knowledge base. The semantic feature extraction module uses the cross-modal encoder to map multiple third data information generated by semantic feature extraction to a unified representation space, and generates multiple first semantic features. The semantic feature extraction module is also used to search in the first knowledge base based on the multiple first semantic features, and determine the first background information corresponding to the multiple first semantic features.

[0038] The second gateway 40 includes a semantic information reconstruction module, a semantic decoder and a channel decoder. The channel decoder is connected to the channel encoder via satellite communication, and is used to perform channel decoding on the channel transmission data to generate third semantic data. The semantic decoder is connected to the channel decoder for semantic decoding on the third semantic data to generate fourth semantic data. The semantic information reconstruction module is connected to the semantic decoder for semantic information reconstruction on the fourth semantic data in the unified representation space and generating second data information. The second data information corresponds to the first data information.

[0039] In addition, the semantic information reconstruction module includes a generative decoder and a second knowledge base. The semantic information reconstruction module is used to search in the second knowledge base based on the plurality of fourth semantic data and determine the second background information corresponding to the plurality of fourth semantic data. The semantic information reconstruction module generates second data information in different modalities based on the plurality of second semantic features that have been reconstructed using the generative decoder.

[0040] Under the above operating environment, according to the first aspect of this embodiment, a satellite-based multimodal semantic communication method is provided. The method comprises: Figure 3 The multimodal semantic communication system shown in is implemented. Figure 4 A schematic diagram showing the process of the method is shown in FIG. Figure 4 As shown, the method includes:

[0041] S402: The first gateway receives a plurality of first data information corresponding to the sender, extracts semantic features from the plurality of first data information, and generates a plurality of corresponding first semantic features, wherein the plurality of first data information corresponds to different modalities;

[0042] S404: The first gateway generates corresponding first semantic data based on the plurality of first semantic features and using a preset first knowledge base;

[0043] S406: The first gateway performs semantic encoding on the first semantic data to generate corresponding second semantic data, and performs channel encoding on the second semantic data to generate corresponding channel transmission data;

[0044] S408: The second gateway receives the channel transmission data sent by the satellite, performs channel decoding on the channel transmission data to generate corresponding third semantic data, and performs semantic decoding on the third semantic data to generate corresponding fourth semantic data;

[0045] S410: The second gateway generates a plurality of corresponding second semantic features based on the fourth semantic data by using a preset second knowledge base; and

[0046] S412: The second gateway reconstructs semantic information of the plurality of second semantic features to generate second data information, wherein the second data information corresponds to the first data information.

[0047] Specifically, first, the transmitting end 10 sends the corresponding multiple first data information to the first gateway 20 in the form of a signal. Then, the first gateway 20 extracts semantic features from the multiple first data information in response to the multiple first data information sent by the transmitting end 10, and generates corresponding multiple first semantic features (S402). The multiple first data information corresponds to different modalities, and the multiple first data information includes, for example, first data information corresponding to text, first data information corresponding to a graph structure, first data information corresponding to a picture, and / or first data information corresponding to table data, which is not specifically limited here.

[0048] refer to Figure 3 As shown, after receiving the first data information, the semantic feature extraction module in the first gateway 20 uses a preset semantic feature extraction model to extract semantic features from the first data information and generates a plurality of third data information. Wherein, when the plurality of first data information includes first data information corresponding to a graph structure, the semantic feature extraction model is a graph neural network model; when the plurality of first data information includes first data information corresponding to a text, the semantic feature extraction model is a recurrent neural network model; when the plurality of first data information includes first data information corresponding to a picture, the semantic feature extraction model is a convolutional neural network model.

[0049] Then, the semantic feature extraction model uses a pre-set cross-modal encoder to map each third data information of different modalities to a unified representation space and generate multiple first semantic features. Thus, the semantic features of the first data information under different modalities are first extracted, and then the third data information under different modalities is mapped to a unified representation space. This can ensure that the semantic features are not missing while fully extracting the semantic features of the data information of different modalities, and facilitate the subsequent processing of the first semantic features of the unified representation. The above content will be described in detail later, so it will not be repeated here.

[0050] Then, the first gateway 20 generates corresponding first semantic data based on the plurality of first semantic features and according to a preset first knowledge base (S404). Figure 3 As shown, first, the semantic feature extraction module in the first gateway station 20 searches the first knowledge base based on the first semantic feature and determines the first background knowledge corresponding to the first semantic feature. Then, the semantic feature extraction module generates the first semantic data based on the first background knowledge and the first semantic feature. The above content will be described in detail later, so it will not be repeated here.

[0051] Furthermore, the first gateway 20 uses a semantic encoder to semantically encode the first semantic data, thereby generating corresponding second semantic data. Then, the first gateway 20 uses a channel encoder to channel encode the second semantic data, thereby generating corresponding channel transmission data (S406).

[0052] Thereafter, the first gateway 20 sends the channel transmission data to the satellite 30 through the wireless channel, and after receiving the channel transmission data, the satellite 30 sends the channel transmission data to the second gateway 40 .

[0053] The second gateway 40 receives the channel transmission data sent by the satellite 30, performs channel decoding on the channel transmission data, generates corresponding third semantic data, and performs semantic decoding on the third semantic data, generates corresponding fourth semantic data (S408). Specifically, when the channel decoder in the second gateway 40 receives the channel transmission data, it performs channel decoding on the channel transmission data, and generates the third semantic data. Then, the channel decoder sends the third semantic data to the semantic decoder, so that the semantic decoder performs semantic decoding on the third semantic data, and generates the fourth semantic data.

[0054] Further, the second gateway station 40 sends the fourth semantic data to the semantic information reconstruction module, and the semantic information reconstruction module uses the pre-set second knowledge base and generates a corresponding plurality of second semantic features based on the fourth semantic data (S410). Specifically, the semantic information reconstruction module in the second gateway station 40 searches the second knowledge base based on the fourth semantic data and determines the second background information corresponding to the fourth semantic data. Then, the semantic information reconstruction module generates a plurality of second semantic features based on the fourth semantic data and the second background information. The above content will be described in detail later, so it will not be repeated here.

[0055] Finally, the semantic information reconstruction module in the second gateway 40 reconstructs the semantic information of the plurality of second semantic features, thereby generating second data information (S412). The second data information corresponds to the first data information. Specifically, the semantic information reconstruction module in the second gateway 40 generates second data information in different modalities using a generative decoder and based on the plurality of second semantic features. For example, in the case where the plurality of first data information includes first data information corresponding to a graph structure and first data information corresponding to a text, the second data information also includes second data information corresponding to a graph structure and second data information corresponding to a text.

[0056] As described in the background technology, the traditional communication method is a bit-level error-free transmission method, that is, all data information from the sender is transmitted to the receiver. However, since there is often redundant information in the data information, and this redundant information is not necessary for the user's understanding and needs, the traditional communication method not only wastes the satellite channel resources, but also affects the data transmission rate.

[0057] In view of this, the present application provides a satellite-based multimodal semantic communication method. And because the first gateway in the application does not transmit all the first data information to the satellite based on the traditional communication method, but first extracts the semantic features of the first data information, and uses the pre-set first knowledge base to abstract the first semantic features into the first semantic data, so as to transmit the first semantic data by satellite.

[0058] Similarly, after receiving the third semantic data (corresponding to the first semantic data), the second gateway in the present application uses a pre-set second knowledge base to restore the third semantic data into a second semantic feature, and reconstructs the semantic information of the second semantic feature, thereby ultimately generating second data information corresponding to the first data information.

[0059] That is, unlike the prior art, the present application does not transmit redundant information in the first data information during the communication transmission process, but only transmits semantic information that is easy for the receiving end to understand, thereby greatly reducing the waste of satellite channel resources and improving data transmission efficiency.

[0060] In addition, in actual application scenarios, the sender often sends not only data information of a single modality, but data information of different modalities. Therefore, after the first gateway extracts semantic features for data in different modalities, it can use a cross-modal encoder to map the data information of different modalities to a unified representation space, thereby facilitating subsequent data processing. Similar to the above, after receiving the second semantic features of the unified representation space, the second gateway can use a generative decoder to generate second data information in different modalities based on the second semantic features, thereby facilitating the receiving end to view the second data information in different modalities.

[0061] Thus, the waste of satellite channel resources is reduced and the data transmission rate is increased. This solves the technical problem in the prior art that redundant information often exists in data information, so using traditional communication methods to transmit data information not only wastes satellite channel resources but also affects the data transmission rate.

[0062] Optionally, the first gateway receives a plurality of first data information corresponding to the transmitting end, extracts semantic features from the plurality of first data information, and generates a plurality of corresponding first semantic features, including: the first gateway extracts semantic features from the plurality of first data information using a graph neural network model, a recurrent neural network model, and / or a convolutional neural network model, and generates a plurality of third data information, wherein each first data information of different modalities includes a graph structure, text, and / or an image; and the first gateway maps each third data information of different modalities to a unified representation space using a preset cross-modal encoder, and generates a plurality of first semantic features. Further optionally, wherein the cross-modal encoder includes a first generative large language model, and the first gateway maps each third data information of different modalities to a unified representation space using a preset cross-modal encoder, and generates a plurality of first semantic features, including: the first gateway converts each third data information of different modalities into a plurality of first semantic features using the first generative large language model and based on a preset first prompt template, wherein the first prompt template is used to indicate that each third data information of different modalities is mapped to a unified representation space and encoded.

[0063] Specifically, refer to Figure 3As shown, when multiple first data information include first data information corresponding to a graph structure, the semantic feature extraction model is a graph neural network model; when multiple first data information include first data information corresponding to text, the semantic feature extraction model is a recurrent neural network model; when multiple first data information include first data information corresponding to a picture, the semantic feature extraction model is a convolutional neural network model.

[0064] Thus, the semantic feature extraction module can use the graph neural network model, the recurrent neural network model and / or the convolutional neural network model to extract semantic features from the first data information and generate multiple third data information. Then, the semantic feature extraction module uses a pre-set cross-modal encoder to map each third data information of different modalities to a unified representation space and generate multiple first semantic features.

[0065] Among them, the cross-modal encoder can be, for example, a first generative large language model. And when the semantic feature extraction module sends each third data information to the first generative large language model, the semantic feature extraction module can use a pre-set first prompt template and combine the first prompt template with each third data, thereby converting each third data information of different modalities into multiple first semantic features. For example, the prompt information of the first prompt template can be, for example, "map each third data information received in different modalities to a unified representation space and encode it."

[0066] Therefore, the first generative large language model can map the third data information corresponding to the graph structure, the third data information corresponding to the text and / or the third data information corresponding to the picture to a unified representation space, and then encode them to generate multiple first semantic features of the unified representation.

[0067] It is worth noting that the above example uses a cross-modal encoder as the first generative large model. In fact, the cross-modal encoder can be a combination of multiple first modal neural networks and encoder neural networks. The multiple first modal neural networks are used to map the third data information under different modalities to a unified representation space, and the encoder neural network is used to encode the third data information mapped to the unified representation space, thereby generating multiple first semantic features. In the case where the third data information corresponds to a graph structure, the first modal neural network can be GNN+transformer; in the case where the third data information corresponds to text, the first modal neural network can be embedding+transformer; in the case where the third data information corresponds to an image, the first modal neural network can be patch+transformer.

[0068] Optionally, the operation of the first gateway generating corresponding first semantic data based on the multiple first semantic features and using a pre-set first knowledge base includes: the first gateway searching the first knowledge base based on the multiple first semantic features and determining first background information corresponding to the multiple first semantic features; and the first gateway generating the first semantic data based on the first background information.

[0069] Specifically, refer to Figure 3 As shown, first, the semantic feature extraction module in the first gateway 20 builds a shared codebook based on the first knowledge base. Then, the semantic feature extraction module compares the similarity between the shared codebook and each first semantic feature, and selects the optimal semantic information (i.e., the first background information) by using vector quantization. Finally, the semantic feature extraction module generates the first semantic data based on the optimal semantic information (i.e., the first background information).

[0070] Optionally, the second gateway station uses a pre-set second knowledge base and generates a corresponding plurality of second semantic features based on the fourth semantic data, including: the second gateway station searches the second knowledge base based on the fourth semantic data and determines second background information corresponding to the fourth semantic data; the second gateway station generates a plurality of second semantic features based on the second background information.

[0071] Specifically, refer to Figure 3 As shown, first, the semantic information reconstruction module in the second gateway 40 further constructs a shared code book based on the second knowledge base after receiving the fourth semantic data generated by the semantic decoder after semantic decoding. Then, the semantic information reconstruction module compares the similarity between the shared code book and the fourth semantic data, and selects the optimal semantic feature (i.e., the second background information) by vector quantization. Finally, the semantic information reconstruction module generates multiple second semantic features based on the optimal semantic feature (i.e., the second background information).

[0072] It is worth noting that the first knowledge base in the first gateway and the second knowledge base in the second gateway contain a large amount of data, and the large amount of data includes data information in multiple different modes. In addition, the first knowledge base in the first gateway and the second knowledge base in the second gateway are updated in real time and synchronously. That is, when the first knowledge base in the first gateway is updated, the second knowledge base in the second gateway is also updated accordingly.

[0073] Furthermore, when the semantic feature extraction module constructs a shared codebook based on the first knowledge base, the shared codebook may be, for example, a comparison table including a mapping method between semantic features and semantic information. Thus, the semantic feature extraction module may determine the corresponding first semantic data based on the plurality of first semantic features and the shared codebook.

[0074] Similarly, when the semantic information reconstruction module constructs a shared codebook based on the second knowledge base, the shared codebook may be, for example, a comparison table including a mapping method between semantic information and semantic features. Thus, the semantic information reconstruction module may determine the corresponding plurality of second semantic features based on the fourth semantic data and the shared codebook.

[0075] Optionally, the second gateway reconstructs semantic information of multiple second semantic features to generate an operation of second data information, including: the second gateway uses a preset generative decoder and based on the multiple second semantic features to generate second data information in different modalities, wherein the generative decoder includes a second generative large language model, and the second gateway uses a preset generative decoder and based on the multiple second semantic features to generate second data information in different modalities. The operation includes: the second gateway uses the second generative large language model and based on a preset second prompt template to convert the multiple second semantic features into multiple second data information in different modalities, wherein the second prompt template is used to indicate decoding of the multiple second semantic features and mapping the decoded multiple second semantic features from a unified representation space to different modalities.

[0076] Specifically, refer to Figure 3 As shown, when the semantic information reconstruction module generates multiple second semantic features based on the second knowledge base, the pre-set generative decoder is further used to generate second data information in different modes based on the multiple second semantic features. The generative decoder can be, for example, a second generative large language model.

[0077] And when the generative decoder is a second generative large language model, the semantic information reconstruction module can use a pre-set second prompt template and combine the second prompt template with each second semantic feature, so as to restore the plurality of second semantic features of the unified representation to a plurality of second data information in different modalities. For example, the prompt information of the second prompt template can be, for example, "decode the plurality of second semantic features of the unified representation space and map the decoded second semantic features to different modalities".

[0078] Thus, the second generative large language model can decode each second semantic feature and convert each decoded second semantic feature into second data information corresponding to the graph structure, second data information corresponding to the text and / or second data information corresponding to the picture. Thus, the receiving end 50 can view the second data information in different modes.

[0079] It is worth noting that the above example uses a generative decoder as the second generative model. In fact, the generative decoder can be a combination of a decoder neural network and multiple second modal neural networks. The decoder neural network is used to decode each second semantic feature mapped to the unified representation space. The multiple second modal neural network terms should set different modes for the decoded multiple second semantic features, thereby generating multiple second data information under different modes.

[0080] Optionally, it also includes: the first gateway station uses a preset first verification model to determine the consistency between the first semantic data and multiple first data information, and issues an alarm when the first semantic data is inconsistent with the multiple first data information; and the second gateway station uses a preset second verification model to determine the consistency between the fourth semantic data and multiple second data information, and issues an alarm when the fourth semantic data is inconsistent with the multiple second data information.

[0081] Specifically, when the semantic feature extraction module in the first gateway 20 generates the first semantic data based on the multiple first data information corresponding to the sending end 10, the consistency between the first semantic data and the multiple first data information can also be determined by using a preset first verification model. The first verification model can be, for example, a binary classification model, and the first verification model is used to indicate whether the first semantic data is consistent or inconsistent with the multiple first data information. Therefore, when the first semantic data is inconsistent with the multiple first data information, an alarm is issued to the first gateway 10.

[0082] Similarly, when the semantic information reconstruction module in the second gateway 40 generates multiple second data information based on the fourth semantic data, the consistency between the fourth semantic data and the multiple second data information can also be determined by using a preset second verification model. The second verification model can be, for example, a binary classification model, and the second verification model is used to indicate whether the fourth semantic data is consistent or inconsistent with the multiple second data information. Thus, when the fourth semantic data is inconsistent with the multiple second data information, an alarm is issued to the second gateway 40.

[0083] Therefore, according to the first aspect of this embodiment, the technical effect of reducing the waste of satellite channel resources and improving the data transmission rate is achieved.

[0084] In addition, reference Figure 1 As shown, according to the second aspect of this embodiment, a storage medium is provided, wherein the storage medium includes a stored program, wherein when the program is run, a processor executes any one of the above methods.

[0085] Therefore, according to this embodiment, the technical effect of reducing the waste of satellite channel resources and improving the data transmission rate is achieved.

[0086] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0087] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0088] Example 2

[0089] Figure 5 The satellite-based multimodal semantic communication device 500 according to this embodiment is shown, and the device 500 corresponds to the method according to embodiment 1. Figure 5As shown, the device 500 includes: a semantic feature extraction module 510, which is used for the first gateway to receive a plurality of first data information corresponding to the sending end, extract semantic features from the plurality of first data information, and generate a plurality of corresponding first semantic features, wherein the plurality of first data information corresponds to different modalities; a first semantic data generation module 520, which is used for the first gateway to generate corresponding first semantic data based on the plurality of first semantic features and according to a pre-set first knowledge base; a semantic encoding module 530, which is used for the first gateway to semantically encode the first semantic data to generate corresponding second semantic data, and channel encode the second semantic data to generate into corresponding channel transmission data; a semantic decoding module 540, which is used for the second gateway to receive the channel transmission data sent by the satellite, channel-decode the channel transmission data, generate corresponding third semantic data, and semantically decode the third semantic data to generate corresponding fourth semantic data; a semantic feature generation module 550, which is used for the second gateway to generate corresponding multiple second semantic features based on the fourth semantic data by using a pre-set second knowledge base; and a semantic information reconstruction module 560, which is used for the second gateway to reconstruct semantic information of the multiple second semantic features, thereby generating second data information, wherein the second data information corresponds to the first data information.

[0090] Optionally, the semantic feature extraction module 510 includes: a third data information generation module, which is used for the first gateway to use a graph neural network model, a recurrent neural network model and / or a convolutional neural network model to extract semantic features of multiple first data information respectively, and generate multiple third data information, wherein each first data information of different modalities includes a graph structure, text and / or an image; and a first semantic feature generation module, which is used for the first gateway to use a pre-set cross-modal encoder to map each third data information of different modalities to a unified representation space, and generate multiple first semantic features.

[0091] Optionally, the cross-modal encoder includes a first generative large language model, and the first semantic feature generation module includes: a first generative large language model module, which is used by the first gateway to convert various third data information of different modalities into multiple first semantic features based on a pre-set first prompt template using the first generative large language model, wherein the first prompt template is used to indicate that various third data information of different modalities are mapped to a unified representation space and encoded.

[0092] Optionally, the first semantic data generating module 520 includes: a first retrieval module, used by the first gateway to search in the first knowledge base based on multiple first semantic features and determine first background information corresponding to the multiple first semantic features; and a first semantic data generating submodule, used by the first gateway to generate first semantic data based on the first background information.

[0093] Optionally, the semantic feature generation module 550 includes: a second retrieval module, used by the second gateway to search in the second knowledge base based on the fourth semantic data and determine the second background information corresponding to the fourth semantic data; a second semantic feature generation module, used by the second gateway to generate multiple second semantic features based on the second background information.

[0094] Optionally, the semantic information reconstruction module 560 includes: a second data information generation module, which is used for the second gateway to generate second data information in different modalities using a preset generative decoder and based on multiple second semantic features, wherein the generative decoder includes a second generative large language model, and the second data information generation module includes: a second data information generation sub-module, which is used for the second gateway to convert multiple second semantic features into multiple second data information in different modalities using a second generative large language model and based on a preset second prompt template, wherein the second prompt template is used to indicate decoding of multiple second semantic features and mapping the decoded multiple second semantic features from a unified representation space to different modalities.

[0095] Optionally, the device 500 also includes: a first alarm module, which is used by the first gateway station to determine the consistency between the first semantic data and the multiple first data information using a preset first verification model, and to issue an alarm when the first semantic data is inconsistent with the multiple first data information; and a second alarm module, which is used by the second gateway station to determine the consistency between the fourth semantic data and the multiple second data information using a preset second verification model, and to issue an alarm when the fourth semantic data is inconsistent with the multiple second data information.

[0096] Therefore, according to this embodiment, the technical effect of reducing the waste of satellite channel resources and improving the data transmission rate is achieved.

[0097] Example 3

[0098] Figure 6 The satellite-based multimodal semantic communication device 600 according to this embodiment is shown, and the device 600 corresponds to the method according to embodiment 1. Figure 6As shown, the device 600 includes: a processor 610; and a memory 620, connected to the processor 610, for providing the processor 610 with instructions for processing the following processing steps: the first gateway receives a plurality of first data information corresponding to the transmitting end, extracts semantic features from the plurality of first data information, and generates a plurality of corresponding first semantic features, wherein the plurality of first data information corresponds to different modalities; the first gateway generates corresponding first semantic data based on the plurality of first semantic features and according to a pre-set first knowledge base; the first gateway performs semantic encoding on the first semantic data to generate corresponding second semantic data, and performs channel encoding on the second semantic data to generate corresponding channel transmission data; the second gateway receives the channel transmission data sent by the satellite, performs channel decoding on the channel transmission data to generate corresponding third semantic data, and performs semantic decoding on the third semantic data to generate corresponding fourth semantic data; the second gateway generates a plurality of corresponding second semantic features based on the pre-set second knowledge base and the fourth semantic data; and the second gateway performs semantic information reconstruction on the plurality of second semantic features to generate second data information, wherein the second data information corresponds to the first data information.

[0099] Optionally, the first gateway receives multiple first data information corresponding to the sending end, extracts semantic features from the multiple first data information, and generates corresponding multiple first semantic features, including: the first gateway uses a graph neural network model, a recurrent neural network model and / or a convolutional neural network model to extract semantic features from the multiple first data information respectively, and generates multiple third data information, wherein each first data information of different modalities includes a graph structure, text and / or an image; and the first gateway uses a pre-set cross-modal encoder to map each third data information of different modalities to a unified representation space, and generates multiple first semantic features.

[0100] Optionally, the cross-modal encoder includes a first generative large language model, and the first gateway uses a preset cross-modal encoder to map each third data information of different modalities to a unified representation space, and generates a plurality of first semantic features, including: the first gateway uses the first generative large language model and based on a preset first prompt template, converts each third data information of different modalities into a plurality of first semantic features, wherein the first prompt template is used to indicate that each third data information of different modalities is mapped to a unified representation space and encoded.

[0101] Optionally, the operation of the first gateway generating corresponding first semantic data based on the multiple first semantic features and using a pre-set first knowledge base includes: the first gateway searching the first knowledge base based on the multiple first semantic features and determining first background information corresponding to the multiple first semantic features; and the first gateway generating the first semantic data based on the first background information.

[0102] Optionally, the second gateway station uses a pre-set second knowledge base and generates a corresponding plurality of second semantic features based on the fourth semantic data, including: the second gateway station searches the second knowledge base based on the fourth semantic data and determines second background information corresponding to the fourth semantic data; the second gateway station generates a plurality of second semantic features based on the second background information.

[0103] Optionally, the second gateway reconstructs semantic information of multiple second semantic features to generate an operation of second data information, including: the second gateway uses a preset generative decoder and based on the multiple second semantic features to generate second data information in different modalities, wherein the generative decoder includes a second generative large language model, and the second gateway uses a preset generative decoder and based on the multiple second semantic features to generate second data information in different modalities. The operation includes: the second gateway uses the second generative large language model and based on a preset second prompt template to convert the multiple second semantic features into multiple second data information in different modalities, wherein the second prompt template is used to indicate decoding of the multiple second semantic features and mapping the decoded multiple second semantic features from a unified representation space to different modalities.

[0104] Optionally, the device 600 also includes: the first gateway station uses a preset first verification model to determine the consistency between the first semantic data and multiple first data information, and issues an alarm when the first semantic data is inconsistent with the multiple first data information; and the second gateway station uses a preset second verification model to determine the consistency between the fourth semantic data and multiple second data information, and issues an alarm when the fourth semantic data is inconsistent with the multiple second data information.

[0105] Therefore, according to this embodiment, the technical effect of reducing the waste of satellite channel resources and improving the data transmission rate is achieved.

[0106] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0107] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0108] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0109] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0110] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0111] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0112] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A satellite-based multimodal semantic communication method, characterized in that: include: The first gateway receives a plurality of first data information corresponding to the sending end, extracts semantic features from the plurality of first data information, and generates a plurality of corresponding first semantic features, wherein the plurality of first data information corresponds to different modalities; The first gateway generates corresponding first semantic data based on the plurality of first semantic features and using a preset first knowledge base; The first gateway performs semantic encoding on the first semantic data to generate corresponding second semantic data, and performs channel encoding on the second semantic data to generate corresponding channel transmission data; The second gateway receives the channel transmission data sent by the satellite, performs channel decoding on the channel transmission data to generate corresponding third semantic data, and performs semantic decoding on the third semantic data to generate corresponding fourth semantic data; The second information gateway generates a corresponding plurality of second semantic features based on the fourth semantic data by using a preset second knowledge base; as well as The second gateway reconstructs semantic information of the plurality of second semantic features to generate second data information, wherein the second data information corresponds to the first data information.

2. The method according to claim 1, characterized in that The first gateway receives a plurality of first data information corresponding to the sender, extracts semantic features from the plurality of first data information, and generates a plurality of corresponding first semantic features, including: The first gateway extracts semantic features from the plurality of first data information respectively using a graph neural network model, a recurrent neural network model and / or a convolutional neural network model, and generates the plurality of third data information, wherein each of the first data information of different modalities includes a graph structure, text and / or an image; and The first gateway maps each third data information of different modalities to a unified representation space using a preset cross-modal encoder, and generates the plurality of first semantic features.

3. The method according to claim 2, characterized in that The cross-modal encoder includes a first generative large language model, and the first gateway uses a preset cross-modal encoder to map each third data information of different modalities to a unified representation space, and generates the plurality of first semantic features, including: The first information gateway uses the first generative large language model and based on a pre-set first prompt template to convert each third data information of different modalities into the multiple first semantic features, wherein the first prompt template is used to indicate that each third data information of different modalities is mapped to a unified representation space and encoded.

4. The method according to claim 2, characterized in that: The operation of the first gateway generating corresponding first semantic data based on the plurality of first semantic features and using a preset first knowledge base includes: The first gateway searches the first knowledge base based on the multiple first semantic features and determines first background information corresponding to the multiple first semantic features; and The first gateway generates the first semantic data based on the first background information.

5. The method according to claim 4, characterized in that The operation of the second gateway using a preset second knowledge base and generating a corresponding plurality of second semantic features based on the fourth semantic data includes: The second information gateway searches the second knowledge base based on the fourth semantic data and determines the second background information corresponding to the fourth semantic data; The second gateway generates the plurality of second semantic features based on the second context information.

6. The method according to claim 5, characterized in that The operation of the second gateway reconstructing semantic information of the plurality of second semantic features to generate second data information includes: The second gateway generates second data information in different modalities using a preset generative decoder based on the plurality of second semantic features, wherein The generative decoder includes a second generative large language model, and the second gateway uses a preset generative decoder and generates second data information in different modes based on the plurality of second semantic features, including: The second gateway uses the second generative large language model and based on a pre-set second prompt template to convert the multiple second semantic features into multiple second data information in different modalities, wherein the second prompt template is used to indicate decoding the multiple second semantic features and mapping the decoded multiple second semantic features from a unified representation space to different modalities.

7. The method according to claim 1, characterized in that Also includes: The first information gateway determines the consistency between the first semantic data and the plurality of first data information by using a preset first verification model, and issues an alarm when the first semantic data is inconsistent with the plurality of first data information; as well as The second gateway determines the consistency between the fourth semantic data and the plurality of second data information using a preset second verification model, and issues an alarm when the fourth semantic data is inconsistent with the plurality of second data information.

8. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is run, the processor executes the method according to any one of claims 1 to 7.

9. A satellite-based multimodal semantic communication device, characterized in that: include: A semantic feature extraction module, configured to receive, at the first gateway, a plurality of first data information corresponding to the sending end, extract semantic features from the plurality of first data information, and generate a plurality of corresponding first semantic features, wherein the plurality of first data information corresponds to different modalities; A first semantic data generating module, configured for the first gateway to generate corresponding first semantic data based on the plurality of first semantic features and using a preset first knowledge base; A semantic encoding module, configured to perform semantic encoding on the first semantic data at the first gateway to generate corresponding second semantic data, and to perform channel encoding on the second semantic data to generate corresponding channel transmission data; A semantic decoding module, configured to receive, at the second gateway, the channel transmission data sent by the satellite, perform channel decoding on the channel transmission data to generate corresponding third semantic data, and perform semantic decoding on the third semantic data to generate corresponding fourth semantic data; A semantic feature generation module, configured for the second gateway to generate a plurality of corresponding second semantic features based on the fourth semantic data by using a preset second knowledge base; as well as The semantic information reconstruction module is used for the second gateway to reconstruct the semantic information of the plurality of second semantic features, thereby generating second data information, wherein the second data information corresponds to the first data information.

10. A satellite-based multimodal semantic communication device, characterized in that: include: processor; as well as A memory, connected to the processor, configured to provide the processor with instructions for processing the following processing steps: The first gateway receives a plurality of first data information corresponding to the sending end, extracts semantic features from the plurality of first data information, and generates a plurality of corresponding first semantic features, wherein the plurality of first data information corresponds to different modalities; The first gateway generates corresponding first semantic data based on the plurality of first semantic features and using a preset first knowledge base; The first gateway performs semantic encoding on the first semantic data to generate corresponding second semantic data, and performs channel encoding on the second semantic data to generate corresponding channel transmission data; The second gateway receives the channel transmission data sent by the satellite, performs channel decoding on the channel transmission data to generate corresponding third semantic data, and performs semantic decoding on the third semantic data to generate corresponding fourth semantic data; The second information gateway generates a corresponding plurality of second semantic features based on the fourth semantic data by using a preset second knowledge base; as well as The second gateway reconstructs semantic information of the plurality of second semantic features to generate second data information, wherein the second data information corresponds to the first data information.

Images