Semantic communication systems, methods, devices, and media for adaptive hierarchical transmission based on object-attribute-relation

Abstract

The application discloses a semantic communication system and method based on object-attribute-relation adaptive hierarchical transmission, and relates to the field of semantic communication. The semantic communication system comprises the following steps: in a high-bandwidth mode, a semantic sending end sends an object semantic stream, an attribute semantic stream and a relation semantic stream to a semantic receiving end; the semantic receiving end generates an object-attribute-relation reconstruction graph of a sample scene by using the received object semantic stream, attribute semantic stream and relation semantic stream; in a medium-bandwidth mode, the semantic sending end truncates the attribute semantic stream of the sample scene and sends the object semantic stream and the relation semantic stream to the semantic receiving end; and the semantic receiving end generates an object-relation reconstruction graph of the sample scene by using the received object semantic stream and relation semantic stream, thereby eliminating pixel-level redundancy without image reconstruction and realizing adaptive hierarchical transmission and graceful degradation according to a channel state.

Description

Technical Field

[0001] This invention relates to the field of semantic communication, and in particular to a semantic communication system, method, device, and medium based on adaptive hierarchical transmission of object-attribute-relationship. Background Technology

[0002] With the rapid development of 6G mobile networks and IoT technologies, global multimedia data traffic is experiencing explosive growth. Simultaneously, the core paradigm of communication systems is undergoing a fundamental shift—from traditional human-to-human interaction to machine-to-machine and human-machine collaborative interaction. Especially in emerging embodied intelligence applications such as industrial inspection and disaster relief, the receiver of the communication link is no longer a human observer requiring high visual fidelity, but an intelligent agent that needs to make real-time decisions based on environmental semantics.

[0003] However, in dynamic environments, embodied agents face severe challenges due to limited and volatile channel bandwidth: traditional video coding prioritizes human visual perception and transmits a large amount of redundant texture information, which seriously hinders machine decision-making under bandwidth constraints; and in dynamic channels, this redundancy leads to significant "cliff effect" and unacceptable latency. Summary of the Invention

[0004] In view of the above-mentioned technical problems, the present invention provides a semantic communication system, method, device and medium based on adaptive hierarchical transmission of object-attribute-relationship, which aims to overcome the above problems or at least partially solve the above problems.

[0005] The first aspect of the present invention provides a semantic communication system based on adaptive hierarchical transmission of object-attribute-relationship, the semantic communication system comprising at least: a semantic sending end and a semantic receiving end deployed with an embodied intelligent agent; When the bandwidth mode between the semantic sender and the semantic receiver is high bandwidth mode, the semantic sender uses image data, text data and audio data in the multimodal data of the sample scene to generate object semantic stream, attribute semantic stream and relational semantic stream of the sample scene and sends them to the semantic receiver. The semantic receiver uses the received object semantic stream, attribute semantic stream and relation semantic stream to perform semantic reconstruction, and obtains the reconstructed object semantic representation, reconstructed attribute semantic representation and reconstructed relation semantic representation. Based on the reconstructed object semantic representation, reconstructed attribute semantic representation and reconstructed relation semantic representation, an object-attribute-relation reconstruction graph of the sample scene is generated. The semantic receiver controls the embodied intelligent agent to reconstruct the object-attribute-relationship graph of the sample scene and perform inference tasks corresponding to the high bandwidth mode. When the bandwidth mode between the semantic sender and the semantic receiver changes from high bandwidth mode to medium bandwidth mode, the semantic sender truncates the attribute semantic stream of the sample scene and sends the object semantic stream and relation semantic stream of the sample scene to the semantic receiver; the upper limit of the bandwidth of the medium bandwidth mode is the lower limit of the bandwidth of the high bandwidth mode. The semantic receiver uses the received object semantic stream and relation semantic stream to perform semantic reconstruction, obtain the reconstructed object semantic representation and the reconstructed relation semantic representation, and generate the object-relation reconstruction graph of the sample scene based on the reconstructed object semantic representation and the reconstructed relation semantic representation; The semantic receiver controls the embodied intelligent agent to reconstruct the object-relationship graph of the sample scene and perform the inference task corresponding to the mid-bandwidth pattern.

[0006] A second aspect of the present invention provides a semantic communication method based on adaptive hierarchical transmission of object-attribute-relationship, applied to the semantic communication system based on adaptive hierarchical transmission of object-attribute-relationship described in the first aspect of the present invention, the method comprising: When the bandwidth mode between the semantic sender and the semantic receiver is high bandwidth mode, the semantic sender uses image data, text data and audio data in the multimodal data of the sample scene to generate object semantic stream, attribute semantic stream and relational semantic stream of the sample scene and sends them to the semantic receiver. The semantic receiver uses the received object semantic stream, attribute semantic stream and relation semantic stream to perform semantic reconstruction, and obtains the reconstructed object semantic representation, reconstructed attribute semantic representation and reconstructed relation semantic representation. Based on the reconstructed object semantic representation, reconstructed attribute semantic representation and reconstructed relation semantic representation, an object-attribute-relation reconstruction graph of the sample scene is generated. The semantic receiver controls the embodied intelligent agent to reconstruct the object-attribute-relationship graph of the sample scene and perform inference tasks corresponding to the high bandwidth mode. When the bandwidth mode between the semantic sender and the semantic receiver changes from high bandwidth mode to medium bandwidth mode, the semantic sender truncates the attribute semantic stream of the sample scene and sends the object semantic stream and relation semantic stream of the sample scene to the semantic receiver; the upper limit of the bandwidth of the medium bandwidth mode is the lower limit of the bandwidth of the high bandwidth mode. The semantic receiver uses the received object semantic stream and relation semantic stream to perform semantic reconstruction, obtain the reconstructed object semantic representation and the reconstructed relation semantic representation, and generate the object-relation reconstruction graph of the sample scene based on the reconstructed object semantic representation and the reconstructed relation semantic representation; The semantic receiver controls the embodied intelligent agent to reconstruct the object-relationship graph of the sample scene and perform the inference task corresponding to the mid-bandwidth pattern.

[0007] A third aspect of the present invention provides an electronic device comprising a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the semantic communication method based on object-attribute-relationship adaptive hierarchical transmission as described in the second aspect of the present invention.

[0008] A fourth aspect of the present invention provides a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the semantic communication method based on adaptive hierarchical transmission of object-attribute-relationship as described in the second aspect of the present invention.

[0009] In the semantic communication system based on adaptive hierarchical transmission of object-attribute-relationship proposed in this invention, the traditional visual reconstruction paradigm is broken through. Pixel-level reconstruction is abandoned, and instead, a structured object-attribute-relationship reconstruction graph is directly decoded and reconstructed at the semantic receiver. This eliminates the need for image reconstruction, removes pixel-level redundancy, and completely avoids the huge computational latency problem faced by traditional video frame reconstruction. This "direct semantic" mechanism can maintain the integrity of the core semantics of multimodal data even under extremely low bitrate conditions, thus ensuring real-time decision-making and task execution of embodied intelligent agents under bandwidth-constrained conditions. Simultaneously, to cope with dynamic wireless channel loops… In this context, the system designs a bandwidth-adaptive OAR hierarchical transmission strategy. By decomposing the semantic graph of "object-attribute-relationship" into logical layers with different priorities, and implementing adaptive hierarchical transmission according to the channel state, the system can transmit object information, attribute information, and relationship information when the channel bandwidth is high, and automatically remove higher-level relationship and / or attribute information when the channel quality degrades. This protects core object data, ensures reliable transmission of lower-level object information, and achieves "graceful degradation." As a result, it provides excellent robustness for real-time embodied agents in harsh wireless environments and overcomes the "cliff effect" in traditional methods. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a structural block diagram of a semantic communication system based on adaptive hierarchical transmission of object-attribute-relationship, as shown in an embodiment of the present invention. Figure 2 This is a flowchart of the steps of a semantic communication method based on adaptive hierarchical transmission of objects-attributes-relationships provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] Please refer to Figure 1 , Figure 1 This is a structural block diagram of a semantic communication system based on adaptive hierarchical transmission of object-attribute-relationship, as illustrated in an embodiment of the present invention. Figure 1 As shown, the semantic communication system based on adaptive hierarchical transmission of object-attribute-relationship provided in this embodiment includes at least: a semantic sending end and a semantic receiving end deployed with an embodied intelligent agent. Embodied intelligence is a cutting-edge field at the intersection of artificial intelligence and robotics, emphasizing that intelligent agents achieve autonomous learning and evolution through dynamic interaction between their bodies and the environment. Its core lies in the deep integration of perception, action, and cognition. An embodied intelligent agent is an artificial intelligence system with a physical body that can interact with the environment in real time, and this embodied intelligent agent is deployed in the semantic receiving end.

[0014] In this system, when the bandwidth mode between the semantic sender and the semantic receiver is high bandwidth mode, the semantic sender uses image data, text data and audio data in the multimodal data of the sample scene to generate object semantic stream, attribute semantic stream and relational semantic stream of the sample scene and sends them to the semantic receiver. The semantic receiver uses the received object semantic stream, attribute semantic stream and relation semantic stream to perform semantic reconstruction, and obtains the reconstructed object semantic representation, reconstructed attribute semantic representation and reconstructed relation semantic representation. Based on the reconstructed object semantic representation, reconstructed attribute semantic representation and reconstructed relation semantic representation, an object-attribute-relation reconstruction graph of the sample scene is generated. The semantic receiver controls the embodied intelligent agent to reconstruct the object-attribute-relationship graph of the sample scene and perform inference tasks corresponding to the high-bandwidth mode.

[0015] In this embodiment, the bandwidth mode between the semantic sender and the semantic receiver includes at least a high bandwidth mode and a medium bandwidth mode. The upper limit of the bandwidth of the medium bandwidth mode is the lower limit of the bandwidth of the high bandwidth mode. The available bandwidth between the semantic sender and the semantic receiver represented by the high bandwidth mode is higher than the available bandwidth between the semantic sender and the semantic receiver represented by the medium bandwidth mode.

[0016] In this embodiment, when the bandwidth mode between the semantic sender and the semantic receiver is high bandwidth mode, the semantic sender can perform multimodal feature extraction, aggregation, and semantic decoupling on the image data, text data, and audio data in the multimodal data of the sample scene to generate the object semantic stream, the attribute semantic stream, and the relational semantic stream of the sample scene. The object semantic stream, the attribute semantic stream, and the relational semantic stream of the sample scene are then sent to the semantic receiver via a wireless channel to transmit complete object, attribute, and relational details for fine-grained control.

[0017] In this embodiment, the semantic receiver can receive object semantic streams, attribute semantic streams, and relation semantic streams via a wireless channel, obtaining the received object semantic stream, the received attribute semantic stream, and the received relation semantic stream. Then, the semantic receiver can perform semantic reconstruction based on the received object semantic stream, the received attribute semantic stream, and the received relation semantic stream, respectively, to obtain reconstructed object semantic representations, reconstructed attribute semantic representations, and reconstructed relation semantic representations. Furthermore, based on the generated reconstructed object semantic representation, reconstructed attribute semantic representation, and reconstructed relation semantic representation, the reconstructed object, reconstructed attribute, and reconstructed relation are obtained, respectively. Finally, based on the reconstructed object, reconstructed attribute, and reconstructed relation, an object-attribute-relation reconstruction graph of the sample scene is generated.

[0018] In this embodiment, after the semantic receiver generates an object-attribute-relationship reconstruction graph of the sample scene, it can control the embodied intelligent agent to understand the objects, attributes, and relationships between objects in the sample scene based on the object-attribute-relationship reconstruction graph. Then, based on this understanding, it executes the inference task corresponding to the high-bandwidth mode. The inference task corresponding to the high-bandwidth mode is a fine-grained instruction task, such as supporting high-precision instruction tasks like "grab the broken red cup on the table."

[0019] When the bandwidth mode between the semantic sender and the semantic receiver changes from high bandwidth mode to medium bandwidth mode, the semantic sender truncates the attribute semantic stream of the sample scene and sends the object semantic stream and relation semantic stream of the sample scene to the semantic receiver. The semantic receiver uses the received object semantic stream and relation semantic stream to perform semantic reconstruction, obtain the reconstructed object semantic representation and the reconstructed relation semantic representation, and generate the object-relation reconstruction graph of the sample scene based on the reconstructed object semantic representation and the reconstructed relation semantic representation; The semantic receiver controls the embodied intelligent agent to reconstruct the object-relationship graph of the sample scene and perform the inference task corresponding to the mid-bandwidth pattern.

[0020] In this embodiment, when the bandwidth mode between the semantic sender and the semantic receiver changes from high bandwidth to medium bandwidth, the semantic sender truncates the attribute semantic stream of the sample scene and does not send it to the semantic receiver. Instead, it sends only the object semantic stream and the relational semantic stream of the sample scene to the semantic receiver via a wireless channel. This is because in the agent's operational tasks, relations define space and interaction logic (e.g., a cup on a table), and relations are more important than attributes. Therefore, under bandwidth-constrained conditions, relations are given priority over attributes.

[0021] In this embodiment, the semantic receiver can receive object semantic streams and relation semantic streams via a wireless channel to obtain the received object semantic streams and the received relation semantic streams. Then, the semantic receiver can perform semantic reconstruction based on the received object semantic streams and the received relation semantic streams respectively to obtain reconstructed object semantic representations and reconstructed relation semantic representations. Based on the generated reconstructed object semantic representations and reconstructed relation semantic representations, reconstructed objects and reconstructed relations are obtained respectively. Finally, based on the reconstructed objects and reconstructed relations, an object-relation reconstruction graph of the sample scene is generated.

[0022] In this embodiment, after the semantic receiver generates an object-relationship reconstruction graph of the sample scene, it can control the embodied intelligent agent to understand the objects in the sample scene and the relationships between them based on the object-relationship reconstruction graph. This allows the agent to determine the spatial and interaction logic of the sample scene, and then, based on the understood objects and relationships, execute the inference task corresponding to the medium bandwidth mode. The inference task corresponding to the medium bandwidth mode is a logical reasoning instruction task, such as a path planning task.

[0023] In this embodiment, the traditional visual reconstruction paradigm is broken, and pixel-level reconstruction is abandoned. Instead, the structured object-attribute-relationship reconstruction graph is directly decoded and reconstructed at the semantic receiver. This eliminates the need for image reconstruction, removes pixel-level redundancy, and completely avoids the huge computational latency problem faced by traditional video frame reconstruction. This "direct semantic" mechanism can still maintain the integrity of the core semantics of multimodal data under extremely low bit rate conditions, thereby ensuring real-time decision-making and task execution of embodied agents under bandwidth-constrained conditions. At the same time, in order to cope with dynamic wireless channel environments, this system designs a bandwidth-adaptive OAR layered transmission strategy. By decomposing the semantic graph of "object-attribute-relationship" into logical layers with different priorities, and implementing adaptive layered transmission according to the channel state, the system can realize the transmission of object information, attribute information, and relationship channels when the channel bandwidth is high, and automatically remove higher-level relationship and / or attribute information when the channel quality degrades, thereby protecting the core object data, ensuring the reliable transmission of lower-level object information, and achieving "graceful degradation". This provides excellent robustness for real-time embodied agents in harsh wireless environments and overcomes the "cliff effect" in traditional methods.

[0024] In conjunction with the above embodiments, in one implementation, the present invention also provides a semantic communication system based on adaptive hierarchical transmission of object-attribute-relationship. In this system, when the bandwidth mode between the semantic sender and the semantic receiver changes from a medium bandwidth mode to a low bandwidth mode, the semantic sender truncates the attribute semantic stream and relation semantic stream of the sample scene and sends the object semantic stream of the sample scene to the semantic receiver; the upper limit of the bandwidth in the low bandwidth mode is the lower limit of the bandwidth in the medium bandwidth mode; The semantic receiver uses the received object semantic stream to perform semantic reconstruction, obtains the reconstructed object semantic representation, and generates an object reconstruction graph of the sample scene based on the reconstructed object semantic representation. The semantic receiver controls the embodied intelligent agent to perform inference tasks corresponding to low-bandwidth modes by reconstructing the object graph of the sample scene.

[0025] In this embodiment, when the bandwidth mode between the semantic transmitter and the semantic receiver changes from medium bandwidth mode to low bandwidth mode, or from high bandwidth mode to low bandwidth mode, the semantic transmitter truncates the attribute semantic stream and relation semantic stream of the sample scene. At this time, the channel experiences severe fading, and the attribute semantic stream and relation semantic stream of the sample scene are not sent to the semantic receiver. Only the object semantic stream of the sample scene is sent to the semantic receiver via the wireless channel. This is to ensure that the transmission of the object stream can guarantee the basic entity recognition and obstacle avoidance functions of the intelligent agent under the "extreme generation mode" of the channel. The upper bandwidth limit of the low bandwidth mode is the lower bandwidth limit of the medium bandwidth mode.

[0026] In this embodiment, the semantic receiver can receive the object semantic stream through a wireless channel to obtain the received object semantic stream. Then, the semantic receiver performs semantic reconstruction based on the received object semantic stream to obtain the reconstructed object semantic representation. Then, it obtains the reconstructed object based on the generated reconstructed object semantic representation, and then generates the object reconstruction map of the sample scene based on the reconstructed object.

[0027] In this embodiment, after the semantic receiver generates an object reconstruction map of the sample scene, it can control the embodied intelligent agent to determine the objects in the sample scene based on the object reconstruction map, and then perform the inference task corresponding to the low-bandwidth mode based on the objects in the sample scene. The inference task corresponding to the low-bandwidth mode is a basic perception task, such as a navigation task or an obstacle avoidance task.

[0028] In this embodiment, to address time-varying wireless channel fading and capacity uncertainty, a bandwidth-adaptive OAR hierarchical transmission strategy is designed to dynamically allocate transmission resources based on semantic priority. This strategy involves decomposing the semantic graph of "object-attribute-relationship" into logical layers with different priorities and implementing adaptive hierarchical transmission based on channel conditions. The system can transmit object information, attribute information, and relationship information when the channel has high bandwidth, automatically remove higher-level attribute information when the channel has low bandwidth, and automatically remove higher-level relationship and attribute information when the channel has low bandwidth. This ensures that core object data is always protected and that the underlying object information is reliably transmitted. This allows for intelligent "graceful degradation" when channel resources are limited, providing excellent robustness for real-time embodied agents in harsh wireless environments and overcoming the "cliff effect" in traditional methods.

[0029] Furthermore, in one embodiment, when the bandwidth mode between the semantic sender and the semantic receiver is a medium bandwidth mode or a low bandwidth mode, for data streams (attribute semantic streams, or, attribute semantic streams and relational semantic streams) that are intentionally truncated by the semantic sender, the semantic receiver can use a zero-filling error masking strategy to fill in the missing data streams: missing symbols can be filled in with null vectors. This method can ensure the tensor dimension consistency of the decoder backend and suppress noise illusions.

[0030] In conjunction with any of the above embodiments, the present invention also provides a semantic communication system based on adaptive hierarchical transmission of object-attribute-relationship data, the system further comprising: an adaptive transmission strategy controller. Among them, the adaptive transmission strategy controller determines the semantic utility of each of the object semantic flow, attribute semantic flow and relational semantic flow based on the importance of the object semantic flow, attribute semantic flow and relational semantic flow to the embodied agent; The adaptive transmission strategy controller determines the available bandwidth between the semantic sender and the semantic receiver. The adaptive policy controller assigns transmission masks to object semantic streams, attribute semantic streams, and relational semantic streams with the goal of ensuring that the total length of the semantic streams to be transmitted does not exceed the available bandwidth and maximizing the total semantic utility of the semantic streams to be transmitted. The adaptive policy controller determines whether the bandwidth mode between the semantic sender and the semantic receiver is high-bandwidth, medium-bandwidth, or low-bandwidth based on the transmission masks assigned to the object semantic stream, attribute semantic stream, and relation semantic stream, and then notifies the semantic sender and the semantic receiver.

[0031] In this embodiment, the adaptive transmission strategy controller can determine the semantic utility of each of the object semantic stream, attribute semantic stream, and relational semantic stream based on their importance to the embodied agent in performing downstream decision-making tasks. Semantic utility characterizes the actual effect of semantics on the semantic receiver, the degree to which it meets needs, or its functional value. In this embodiment, the semantic utility of the object semantic stream, relational semantic stream, and attribute semantic stream decreases sequentially, thus establishing a strict priority hierarchy for them: object semantic stream priority > relational semantic stream priority > attribute semantic stream priority.

[0032] In this embodiment, the adaptive transmission strategy controller can determine the available bandwidth between the semantic sender and the semantic receiver in real time, and with the goal of always transmitting the object semantic stream, selectively transmitting the attribute semantic stream and the relational semantic stream, and with the goal of ensuring that the total length of the semantic stream to be transmitted does not exceed the available bandwidth between the semantic sender and the semantic receiver, and maximizing the total semantic utility of the semantic stream to be transmitted, allocate transmission masks for the object semantic stream, the attribute semantic stream and the relational semantic stream respectively.

[0033] In this embodiment, the assigned transmission mask is either 0 or 1, and a transmission mask of 1 for a semantic stream indicates that the semantic stream is to be transmitted, while a transmission mask of 0 for a semantic stream indicates that the semantic stream is not transmitted. That is, in this embodiment, the transmission mask for the object semantic stream is 1, and the transmission masks for the attribute semantic stream and the relational semantic stream are either 0 or 1.

[0034] After assigning transport masks to the object semantic stream, attribute semantic stream, and relation semantic stream, the adaptive policy controller can determine whether the bandwidth mode between the semantic sender and the semantic receiver is high bandwidth mode, medium bandwidth mode, or low bandwidth mode based on the transport masks assigned to the object semantic stream, attribute semantic stream, and relation semantic stream, and then notify the semantic sender and the semantic receiver of the bandwidth mode between the semantic sender and the semantic receiver.

[0035] Specifically, when the transmission mask for the object semantic stream is 1 and the transmission masks for the attribute semantic stream and the relation semantic stream are both 0, the adaptive policy controller determines the bandwidth mode between the semantic sender and the semantic receiver to be low bandwidth mode; when the transmission masks for the object semantic stream and the relation semantic stream are both 1 and the transmission mask for the attribute semantic stream is 0, the adaptive policy controller determines the bandwidth mode between the semantic sender and the semantic receiver to be medium bandwidth mode; and when the transmission masks for the object semantic stream, the relation semantic stream, and the attribute semantic stream are all 1, the adaptive policy controller determines the bandwidth mode between the semantic sender and the semantic receiver to be high bandwidth mode.

[0036] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a semantic communication system based on adaptive hierarchical transmission of object-attribute-relationship. In this system, the semantic transmitter includes: a multimodal feature extraction module to be trained, a multimodal feature aggregation module to be trained, a decoupling module to be trained, a transmitter object gating module, a transmitter relation gating module, and a transmitter attribute gating module.

[0037] The multimodal feature extraction module to be trained is used to extract features from the image data, text data and audio data in the multimodal data of the sample scene, respectively, to obtain the image feature representation, text feature representation and audio feature representation of the sample scene.

[0038] In this embodiment, the semantic sending end can input the image data, text data and audio data in the multimodal data of the sample scene into the multimodal feature extraction module to be trained. Through the multimodal feature extraction module to be trained, the image data, text data and audio data in the multimodal data of the sample scene are extracted respectively to obtain the image feature representation, text feature representation and audio feature representation of the sample scene.

[0039] In a specific example, the multimodal feature extraction module to be trained may include three parallel feature extraction modules (image extraction module, text extraction module, and audio extraction module) to process image data, text data, and audio data in multimodal data respectively, and project the processing results onto a unified bandwidth-compatible dimension (e.g., D=256) to obtain image feature representation, text feature representation, and audio feature representation of the sample scene.

[0040] The multimodal feature aggregation module to be trained is used to obtain a multimodal aggregated representation of the sample scene based on the learnable image modal representation and the image feature representation of the sample scene, the learnable text modal representation and the text feature representation of the sample scene, and the learnable audio modal representation and the audio feature representation of the sample scene.

[0041] In this embodiment, the semantic sending end inputs the image feature representation, text feature representation, and audio feature representation of the sample scene output by the multimodal feature extraction module to be trained into the multimodal feature aggregation module to be trained. Through the multimodal feature aggregation module to be trained, joint source aggregation is performed based on the learnable image modality representation and the image feature representation of the sample scene, the learnable text modality representation and the text feature representation of the sample scene, and the learnable audio modality representation and the audio feature representation of the sample scene to obtain the multimodal aggregated representation of the sample scene.

[0042] In a specific example, the learnable image modality representation can be first concatenated with the image feature representation of the sample scene to obtain image connectivity features using the trainable multimodal feature aggregation module. Then, the learnable text modality representation can be concatenated with the text feature representation of the sample scene to obtain text connectivity features, and the learnable audio modality representation can be concatenated with the audio feature representation of the sample scene to obtain audio connectivity features. Finally, the trainable multimodal feature aggregation module concatenates the image connectivity features, text connectivity features, and audio connectivity features to obtain the multimodal aggregated representation of the sample scene. In an optional example, the learnable image modality representation, learnable text modality representation, and learnable audio modality representation are all learnable embeddings that are updated as the model is trained.

[0043] The decoupling module to be trained is used to decouple the object semantic representation, attribute semantic representation, and relation semantic representation of the sample scene from the multimodal aggregated representation of the sample scene based on the learnable object query representation, learnable attribute query representation, and learnable relation query representation.

[0044] In this embodiment, the semantic sending end inputs the multimodal aggregated representation of the sample scene output by the multimodal feature aggregation module to be trained into the decoupling module to be trained. The decoupling module then performs semantic decoupling on this multimodal aggregated representation: based on the learnable object query representation, the learnable attribute query representation, and the learnable relation query representation, the object semantic representation, attribute semantic representation, and relation semantic representation of the sample scene are decoupled from the multimodal aggregated representation of the sample scene, respectively. The learnable object query representation, learnable attribute query representation, and learnable relation query representation are all learnable semantic queries and are updated as the model is trained.

[0045] In one optional example, the multimodal aggregate representation and the learnable semantic query (learnable object query representation, learnable attribute query representation, or learnable relation query representation) can be jointly source-attention processed by the decoupled module to be trained to obtain the attention processing result. Then, the attention processing result is processed by a linear routing filter for a specific task (object, attribute, or relation) to obtain the object semantic representation, attribute semantic representation, or relation semantic representation.

[0046] The sending-end object gating module is connected to the object semantic representation output of the decoupled module to be trained. It is used to close in low-bandwidth mode, medium-bandwidth mode and high-bandwidth mode to trigger the semantic sending end to compress the object semantic representation of the sample scene into an object semantic stream and send it to the semantic receiving end.

[0047] In this embodiment, the sending-end object gating module is connected to the object semantic representation output end of the decoupling module to be trained. This object semantic representation output end is used to output the object semantic representation generated by the decoupling module to be trained. Specifically, the sending-end object gating module is closed when the bandwidth mode between the semantic sending end and the semantic receiving end is low bandwidth, medium bandwidth, or high bandwidth. When the sending-end object gating module is closed, the semantic sending end is triggered to compress the object semantic representation of the sample scene into an object semantic stream and send it to the semantic receiving end via a wireless channel.

[0048] The sending end relation gating module is connected to the relation semantic representation output end of the decoupled module to be trained. It is used to disconnect in low bandwidth mode and close in high bandwidth mode or medium bandwidth mode to trigger the semantic sending end to compress the relation semantic representation of the sample scene into a relation semantic stream and send it to the semantic receiving end.

[0049] In this embodiment, the sending-end relation gating module is connected to the relation semantic representation output terminal of the decoupling module to be trained. This relation semantic representation output terminal is used to output the relation semantic representation generated by the decoupling module to be trained. Specifically, the sending-end relation gating module is disconnected when the bandwidth mode between the semantic sending end and the semantic receiving end is low bandwidth mode, and closed when the bandwidth mode between the semantic sending end and the semantic receiving end is medium bandwidth mode or high bandwidth mode. When the sending-end relation gating module is closed, the semantic sending end is triggered to compress the relation semantic representation of the sample scene into a relation semantic stream and send it to the semantic receiving end via a wireless channel.

[0050] The sending-end attribute gating module is connected to the attribute semantic representation output of the decoupled module to be trained. It is used to disconnect in low-bandwidth or medium-bandwidth mode and close in high-bandwidth mode to trigger the semantic sending end to compress the attribute semantic representation of the sample scene into an attribute semantic stream and send it to the semantic receiving end.

[0051] In this embodiment, the sending-end attribute gating module is connected to the attribute semantic representation output terminal of the decoupling module to be trained. This output terminal is used to output the attribute semantic representation generated by the decoupling module to be trained. Specifically, the sending-end attribute gating module is disconnected when the bandwidth mode between the semantic sending end and the semantic receiving end is low-bandwidth or medium-bandwidth, and closed when the bandwidth mode is high-bandwidth. When the sending-end attribute gating module is closed, the semantic sending end is triggered to compress the attribute semantic representation of the sample scene into an attribute semantic stream and transmit it to the semantic receiving end via a wireless channel.

[0052] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a semantic communication system based on adaptive hierarchical transmission of object-attribute-relationship. In this system, the semantic receiver includes: a training object decoding module, a training attribute decoding module, a training relation decoding module, and a training graph generation module.

[0053] The training object decoding module is used to decompress and decode the received object semantic stream to obtain the reconstructed object semantic representation; in low bandwidth mode, medium bandwidth mode or high bandwidth mode, the reconstructed object is obtained based on the reconstructed object semantic representation.

[0054] In this embodiment, the object decoding module can sequentially decompress and decode the object semantic stream received by the semantic receiver to obtain the reconstructed object semantic representation. Furthermore, when the bandwidth mode between the semantic sender and receiver is low bandwidth, medium bandwidth, or high bandwidth, the object decoding module can obtain the reconstructed object of the sample scene based on the reconstructed object semantic representation.

[0055] The training attribute decoding module is used to decompress and decode the received attribute semantic stream to obtain the reconstructed attribute semantic representation; in high bandwidth mode, the reconstructed attributes of the reconstructed object are obtained based on the reconstructed attribute semantic representation and the object semantic representation of the reconstructed object.

[0056] In this embodiment, the attribute module to be trained can sequentially decompress and decode the attribute semantic stream received by the semantic receiver to obtain the reconstructed attribute semantic representation. Furthermore, when the bandwidth mode between the semantic sender and receiver is high-bandwidth mode, the attribute decoding module to be trained can obtain the reconstructed attributes of the sample scene based on the reconstructed object semantic representation and the object semantic representation of the reconstructed object. Here, the object semantic representation of the reconstructed object is the entity semantic representation corresponding to the reconstructed object.

[0057] The training relation decoding module decompresses and decodes the received relation semantic stream to obtain the reconstructed relation semantic representation. In medium-bandwidth mode or high-bandwidth mode, based on the reconstructed relation semantic representation and the object semantic representation of two different reconstructed objects, the reconstructed relation of each pair of reconstructed objects is obtained.

[0058] In this embodiment, the relation-to-be-trained module can sequentially decompress and decode the relation semantic stream received by the semantic receiver to obtain a reconstructed relation semantic representation. Furthermore, when the bandwidth mode between the semantic sender and receiver is medium-bandwidth or high-bandwidth, the relation-to-be-trained decoding module can obtain the reconstructed relation between each pair of reconstructed objects based on the reconstructed relation semantic representation and the object semantic representations of two different reconstructed objects. Here, the object semantic representations of the two different reconstructed objects are the entity semantic representations corresponding to the two different reconstructed objects respectively.

[0059] The training graph generation module is used to generate an object reconstruction graph of the sample scene based on the reconstructed objects in low bandwidth mode; to generate an object-relationship reconstruction graph of the sample scene based on the reconstructed objects and the reconstruction relationship between every two reconstructed objects in medium bandwidth mode; and to generate an object-attribute-relationship reconstruction graph of the sample scene based on the reconstructed objects, the reconstruction attributes of the reconstructed objects, and the reconstruction relationship between every two reconstructed objects in high bandwidth mode.

[0060] In this embodiment, when the bandwidth mode between the semantic sender and the semantic receiver is low bandwidth, the training graph generation module can generate an object reconstruction graph of the sample scene based on the reconstructed object; when the bandwidth mode between the semantic sender and the semantic receiver is medium bandwidth, the training graph generation module can generate an object-relationship reconstruction graph of the sample scene based on the reconstructed object and the reconstruction relationship between every two reconstructed objects; when the bandwidth mode between the semantic sender and the semantic receiver is high bandwidth, the training graph generation module can generate an object-attribute-relationship reconstruction graph of the sample scene based on the reconstructed object, the reconstruction attributes of the reconstructed object, and the reconstruction relationship between every two reconstructed objects.

[0061] In this embodiment, a two-stage cascaded architecture is adopted to mitigate the propagation of reconstruction errors: In the first stage, the object decoding module obtains the reconstructed object based on the semantic representation of the reconstructed object, and uses the reconstructed object as the basic anchor point; In the second stage, the object semantic representation of the reconstructed object is injected as context information into the downstream branches (attribute decoding module and / or relation decoding module) to recover the condition-dependent attribute and relation representations, so as to provide strict category constraints for attribute reconstruction and relation reconstruction.

[0062] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a semantic communication system based on adaptive hierarchical transmission of object-attribute-relationship. In this system, the various training modules in the semantic sender and receiver constitute a training model, and the training process of the training model includes at least the following steps S11 to S16: Step S11: Obtain multimodal data and object-attribute-relationship graph of the sample scene.

[0063] In this embodiment, sample data is first acquired. This sample data includes multimodal data of a sample scene and an object-attribute-relationship graph of the sample scene. The object-attribute-relationship graph is an actual object-attribute-relationship graph generated from objects, their attributes, and the relationships between pairs of objects within the sample scene, serving as the label carried by the multimodal data of the sample scene. The sample scene is the scene used by the semantic communication system during model training. It should be noted that the time steps of the image data, text data, and audio data in the multimodal data are aligned. The inherent dependency relationship between objects, attributes, and relationships is as follows: objects act as anchors, attributes refine the state features of objects, and relationships connect appropriate object pairs.

[0064] Step S12: Process the multimodal data of the sample scene using the model to be trained to obtain the reconstructed objects, the reconstruction attributes of the reconstructed objects, and the reconstruction relationship between every two reconstructed objects.

[0065] In this embodiment, multimodal data of the sample scene can be input into the model to be trained. The model then processes the multimodal data to obtain the reconstructed objects, their reconstruction attributes, and the reconstruction relationships between any two objects. The specific processing procedure of the model to be trained on the multimodal data of the sample scene can be found in the processing procedures of the various training modules in the semantic sending and receiving ends of the aforementioned embodiments.

[0066] Step S13: Based on the differences between the object labels in the object-attribute-relationship graph of the reconstructed object and the sample scene, obtain the object loss function value.

[0067] In this embodiment, after obtaining the reconstructed object, the object loss function value can be calculated based on the difference between the reconstructed object and the object labels in the object-attribute-relationship graph of the sample scene. In one example, the object loss function value can be obtained based on focus loss.

[0068] Step S14: Based on the difference between the reconstructed attributes of the reconstructed object and the attribute labels in the object-attribute-relationship graph of the sample scene, obtain the attribute loss function value.

[0069] In this embodiment, after obtaining the reconstructed attributes of the reconstructed object, the attribute loss function value can be calculated based on the difference between the reconstructed attributes of the reconstructed object and the attribute labels in the object-attribute-relationship graph of the sample scene. In one example, the attribute loss function value can be obtained based on the multi-label focus loss function.

[0070] Step S15: Based on the difference between the reconstruction relationship between each pair of reconstructed objects and the relationship label in the object-attribute-relationship graph of the sample scene, obtain the relationship loss function value.

[0071] In this embodiment, after obtaining the reconstruction relationship between every two reconstructed objects, the relationship loss function value can be calculated based on the difference between the reconstruction relationship between every two reconstructed objects and the relationship labels in the object-attribute-relationship graph of the sample scene. In one example, the relationship loss function value can be obtained based on the reweighted cross-entropy loss function.

[0072] Step S16: Update the model parameters of the model to be trained based at least on the object loss function value, attribute loss function value, and relation loss function value. After training is completed, deploy each trained module to the semantic sender and semantic receiver respectively.

[0073] In this embodiment, the total loss function value can be obtained based on at least the object loss function value, the attribute loss function value, and the relation loss function value. Then, the model parameters of each module to be trained in the model to be trained are updated based on the total loss function value until the total loss function value converges and the model training ends. After training is completed, each trained module is obtained and each trained module is deployed on the semantic sending end and the semantic receiving end respectively.

[0074] In one embodiment, each trainable module in the model to be trained includes: a multimodal feature extraction module, a multimodal feature aggregation module, a decoupling module, a sender object gating module, a sender relation gating module, a sender attribute gating module, a trainable object decoding module, a trainable attribute decoding module, a trainable relation decoding module, and a trainable graph generation module. Then, each trained module includes: a trained multimodal feature extraction module, a trained multimodal feature aggregation module, a trained decoupling module, a trained sender object gating module, a trained sender relation gating module, a trained sender attribute gating module, a trained object decoding module, a trained attribute decoding module, a trained relation decoding module, and a trained graph generation module. The trained multimodal feature extraction module, trained multimodal feature aggregation module, trained decoupling module, trained sender object gating module, trained sender relation gating module, and trained sender attribute gating module are deployed at the semantic sender end; the trained object decoding module, trained attribute decoding module, trained relation decoding module, and trained graph generation module are deployed at the semantic receiver end.

[0075] In another embodiment, each trainable module in the model to be trained includes: a multimodal feature extraction module, a multimodal feature aggregation module, a decoupling module, an object decoding module, an attribute decoding module, a relation decoding module, and a graph generation module. In this case, the sending-end object gating module, the sending-end relation gating module, and the sending-end attribute gating module are in the model to be trained but do not need to be trained. Then, each trained module includes: a trained multimodal feature extraction module, a trained multimodal feature aggregation module, a trained decoupling module, a trained object decoding module, a trained attribute decoding module, a trained relation decoding module, and a trained graph generation module. The trained multimodal feature extraction module, the trained multimodal feature aggregation module, the trained decoupling module, the sending-end object gating module, the sending-end relation gating module, and the sending-end attribute gating module are deployed at the semantic sending end; the trained object decoding module, the trained attribute decoding module, the trained relation decoding module, and the trained graph generation module are deployed at the semantic receiving end.

[0076] In this embodiment, after each trained module is deployed at the semantic sender and semantic receiver, the semantic communication system based on adaptive hierarchical transmission of object-attribute-relationship can process the multimodal data of the target scene through the semantic sender. The semantic receiver generates an object-attribute-relationship reconstruction graph of the target scene in high-bandwidth mode, and then controls the embodied agent to use this graph to perform the inference task corresponding to high-bandwidth mode. Alternatively, the semantic receiver generates an object-relationship reconstruction graph of the target scene in medium-bandwidth mode, and then controls the embodied agent to use this graph to perform the inference task corresponding to medium-bandwidth mode. Or, the semantic receiver generates an object reconstruction graph of the target scene in low-bandwidth mode, and then controls the embodied agent to use this graph to perform the inference task corresponding to low-bandwidth mode. Here, the target scene is the scene used by the semantic communication system during model application.

[0077] Specifically, the specific method by which the semantic communication system processes the multimodal data of the target scene and reconstructs the object-attribute-relationship reconstruction graph, object-relationship reconstruction graph, or object reconstruction graph of the target scene under different bandwidth modes is the same as or similar to the specific method by which the semantic communication system processes the multimodal data of the sample scene and reconstructs the object-attribute-relationship reconstruction graph, object-relationship reconstruction graph, or object reconstruction graph of the sample scene under different bandwidth modes in the aforementioned embodiments. For specific details, please refer to the aforementioned embodiments.

[0078] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a semantic communication system based on adaptive hierarchical transmission of object-attribute-relationship. In this system, during the training process of the model to be trained, low-bandwidth mode, medium-bandwidth mode, and high-bandwidth mode are randomly selected as the current bandwidth mode; and the training process of the model to be trained may further include steps S21 to S23, and the "updating the model parameters of the model to be trained based at least on the object loss function value, attribute loss function value, and relation loss function value" in step S16 above may specifically include steps S24 to S26: Step S21: When the current bandwidth mode is low bandwidth mode, medium bandwidth mode or high bandwidth mode, obtain the object alignment loss function value based on the difference between the reconstructed object semantic representation and the object semantic representation of the sample scene.

[0079] In this embodiment, the current bandwidth mode refers to the bandwidth mode between the semantic sender and the semantic receiver during the current training round. When the current bandwidth mode is low bandwidth, medium bandwidth, or high bandwidth, the object alignment loss function value can be calculated based on the difference between the reconstructed object semantic representation and the object semantic representation of the sample scene.

[0080] Step S22: When the current bandwidth mode is medium bandwidth mode or high bandwidth mode, obtain the relation alignment loss function value based on the difference between the reconstructed relation semantic representation and the relation semantic representation of the sample scene.

[0081] In this embodiment, when the current bandwidth mode is medium bandwidth mode or high bandwidth mode, the relation alignment loss function value can be calculated based on the difference between the reconstructed relation semantic representation and the relation semantic representation of the sample scene.

[0082] Step S23: When the current bandwidth mode is high bandwidth mode, obtain the attribute alignment loss function value based on the difference between the reconstructed attribute semantic representation and the attribute semantic representation of the sample scene.

[0083] In this embodiment, when the current bandwidth mode is high bandwidth mode, the attribute alignment loss function value can be calculated based on the difference between the reconstructed attribute semantic representation and the attribute semantic representation of the sample scene.

[0084] In one optional example, the object alignment loss function value, relation alignment loss function value, and attribute alignment loss function value can all be calculated by combining cosine similarity and L1 regularization.

[0085] Step S24: When the current bandwidth mode is low bandwidth mode, update the model parameters of the model to be trained based at least on the object loss function value, attribute loss function value, relation loss function value, and object alignment loss function value.

[0086] In this embodiment, when the current bandwidth mode is low bandwidth mode, the total loss function value can be calculated based on at least the object loss function value, attribute loss function value, relation loss function value, and object alignment loss function value, and then the model parameters of the model to be trained can be updated based on the total loss function value.

[0087] Step S25: When the current bandwidth mode is medium bandwidth mode, update the model parameters of the model to be trained based at least on the object loss function value, attribute loss function value, relation loss function value, object alignment loss function value, and relation alignment loss function value.

[0088] In this embodiment, when the current bandwidth mode is medium bandwidth mode, the total loss function value can be calculated based on at least the object loss function value, attribute loss function value, relation loss function value, object alignment loss function value, and relation alignment loss function value, and then the model parameters of the model to be trained can be updated based on the total loss function value.

[0089] Step S26: When the current bandwidth mode is high bandwidth mode, update the model parameters of the model to be trained based at least on the object loss function value, attribute loss function value, relation loss function value, object alignment loss function value, attribute alignment loss function value, and relation alignment loss function value.

[0090] In this embodiment, when the current bandwidth mode is high bandwidth mode, the total loss function value can be calculated based on at least the object loss function value, attribute loss function value, relation loss function value, object alignment loss function value, attribute alignment loss function value, and relation alignment loss function value, and then the model parameters of the model to be trained can be updated based on the total loss function value.

[0091] Based on the same inventive concept, one embodiment of the present invention provides a semantic communication method based on adaptive hierarchical transmission of object-attribute-relationship, applicable to the semantic communication system based on adaptive hierarchical transmission of object-attribute-relationship described in any of the foregoing embodiments. (Reference) Figure 2 , Figure 2 This is a flowchart illustrating the steps of a semantic communication method based on adaptive hierarchical transmission of object-attribute-relationship according to an embodiment of the present invention. Figure 2 As shown, the method includes the following steps: Step S1-1: When the bandwidth mode between the semantic sender and the semantic receiver is high bandwidth mode, the semantic sender uses the image data, text data and audio data in the multimodal data of the sample scene to generate the object semantic stream, attribute semantic stream and relation semantic stream of the sample scene and sends them to the semantic receiver. Step S1-2: The semantic receiver uses the received object semantic stream, attribute semantic stream and relation semantic stream to perform semantic reconstruction, and obtains the reconstructed object semantic representation, reconstructed attribute semantic representation and reconstructed relation semantic representation. Based on the reconstructed object semantic representation, reconstructed attribute semantic representation and reconstructed relation semantic representation, an object-attribute-relation reconstruction graph of the sample scene is generated. Steps S1-3: The semantic receiver controls the embodied intelligent agent to reconstruct the object-attribute-relationship graph of the sample scene and perform the reasoning task corresponding to the high bandwidth mode. Step S1-4: When the bandwidth mode between the semantic sender and the semantic receiver changes from high bandwidth mode to medium bandwidth mode, the semantic sender truncates the attribute semantic stream of the sample scene and sends the object semantic stream and relation semantic stream of the sample scene to the semantic receiver; the upper limit of the bandwidth of the medium bandwidth mode is the lower limit of the bandwidth of the high bandwidth mode. Steps S1-5: The semantic receiver performs semantic reconstruction using the received object semantic stream and relation semantic stream to obtain the reconstructed object semantic representation and the reconstructed relation semantic representation, and generates an object-relation reconstruction graph of the sample scene based on the reconstructed object semantic representation and the reconstructed relation semantic representation. Steps S1-6: The semantic receiver controls the embodied agent to reconstruct the object-relationship graph of the sample scene and execute the inference task corresponding to the medium bandwidth mode.

[0092] Optionally, the method further includes: When the bandwidth mode between the semantic sender and the semantic receiver changes from medium bandwidth mode to low bandwidth mode, the semantic sender truncates the attribute semantic stream and relation semantic stream of the sample scene and sends the object semantic stream of the sample scene to the semantic receiver; the upper limit of the bandwidth of the low bandwidth mode is the lower limit of the bandwidth of the medium bandwidth mode. The semantic receiver uses the received object semantic stream to perform semantic reconstruction, obtains the reconstructed object semantic representation, and generates an object reconstruction graph of the sample scene based on the reconstructed object semantic representation. The semantic receiver controls the embodied intelligent agent to perform inference tasks corresponding to low-bandwidth modes by reconstructing the object graph of the sample scene.

[0093] Optionally, the method further includes; The adaptive transmission strategy controller determines the semantic utility of each semantic stream—object semantic stream, attribute semantic stream, and relational semantic stream—based on their importance to the embodied agent. The semantic utility of the object semantic stream, relational semantic stream, and attribute semantic stream decreases in that order. The adaptive transmission strategy controller determines the available bandwidth between the semantic sender and the semantic receiver. The adaptive policy controller aims to ensure that the total length of the semantic stream to be transmitted does not exceed the available bandwidth and to maximize the total semantic utility of the semantic stream to be transmitted. It assigns transmission masks to object semantic streams, attribute semantic streams, and relational semantic streams. The transmission mask for object semantic streams is 1, while the transmission masks for attribute semantic streams and relational semantic streams are 0 or 1 respectively. A transmission mask of 1 for a semantic stream indicates that the semantic stream is to be transmitted. The adaptive policy controller determines whether the bandwidth mode between the semantic sender and the semantic receiver is high-bandwidth, medium-bandwidth, or low-bandwidth based on the transmission masks assigned to the object semantic stream, attribute semantic stream, and relation semantic stream, and then notifies the semantic sender and the semantic receiver.

[0094] Optionally, step S1-1 above specifically includes: The multimodal feature extraction module to be trained extracts features from the image data, text data, and audio data in the multimodal data of the sample scene, respectively, to obtain the image feature representation, text feature representation, and audio feature representation of the sample scene; The multimodal feature aggregation module to be trained obtains the multimodal aggregated representation of the sample scene based on the learnable image modal representation and the image feature representation of the sample scene, the learnable text modal representation and the text feature representation of the sample scene, and the learnable audio modal representation and the audio feature representation of the sample scene. The decoupling module to be trained decouples the object semantic representation, attribute semantic representation, and relation semantic representation of the sample scene from the multimodal aggregated representation of the sample scene based on the learnable object query representation, learnable attribute query representation, and learnable relation query representation; The sending end object gating module is connected to the object semantic representation output end of the decoupled module to be trained. It is closed in high bandwidth mode to trigger the semantic sending end to compress the object semantic representation of the sample scene into an object semantic stream and send it to the semantic receiving end. The sending end relation gating module is connected to the relation semantic representation output end of the decoupled module to be trained. It is closed in high bandwidth mode to trigger the semantic sending end to compress the relation semantic representation of the sample scene into a relation semantic stream and send it to the semantic receiving end. The sending-end attribute gating module is connected to the attribute semantic representation output of the decoupled module to be trained. It is closed in high-bandwidth mode to trigger the semantic sending end to compress the attribute semantic representation of the sample scene into an attribute semantic stream and send it to the semantic receiving end.

[0095] Optionally, the step S1-4 above, "the semantic sending end truncates the attribute semantic stream of the sample scene and sends the object semantic stream and relation semantic stream of the sample scene to the semantic receiving end," includes: The sending end object gating module is connected to the object semantic representation output end of the decoupling module to be trained. It is closed in medium bandwidth mode to trigger the semantic sending end to compress the object semantic representation of the sample scene into an object semantic stream and send it to the semantic receiving end. The sending end relation gating module is connected to the relation semantic representation output end of the decoupled module to be trained. It is closed in medium bandwidth mode to trigger the semantic sending end to compress the relation semantic representation of the sample scene into a relation semantic stream and send it to the semantic receiving end. The sending-end attribute gating module is connected to the attribute semantic representation output of the decoupled module to be trained, and is disconnected in medium bandwidth mode.

[0096] Optionally, the semantic sender truncates the attribute semantic stream and relational semantic stream of the sample scene and sends the object semantic stream of the sample scene to the semantic receiver, including: The sending end object gating module is connected to the object semantic representation output end of the decoupling module to be trained. It is closed in the medium-low width mode to trigger the semantic sending end to compress the object semantic representation of the sample scene into an object semantic stream and send it to the semantic receiving end. The sending end relation gating module is connected to the relation semantic representation output of the decoupled module to be trained, and is disconnected in low bandwidth mode. The sending-end attribute gating module is connected to the attribute semantic representation output of the decoupled module to be trained, and is disconnected in low-bandwidth mode.

[0097] Optionally, steps S1-2 above specifically include: The object decoding module decompresses and decodes the received object semantic stream to obtain the reconstructed object semantic representation; in high-bandwidth mode, the reconstructed object is obtained based on the reconstructed object semantic representation. The attribute decoding module to be trained decompresses and decodes the received attribute semantic stream to obtain the reconstructed attribute semantic representation; in high bandwidth mode, the reconstructed attributes of the reconstructed object are obtained based on the reconstructed attribute semantic representation and the object semantic representation of the reconstructed object. The training relation decoding module decompresses and decodes the received relation semantic stream to obtain the reconstructed relation semantic representation; in high bandwidth mode, based on the reconstructed relation semantic representation and the object semantic representation of two different reconstructed objects, the reconstructed relation between each pair of reconstructed objects is obtained; In high-bandwidth mode, the training graph generation module generates an object-attribute-relationship reconstruction graph of the sample scene based on the reconstructed objects, the reconstruction attributes of the reconstructed objects, and the reconstruction relationship between every two reconstructed objects.

[0098] Optionally, steps S1-5 above specifically include: The object decoding module decompresses and decodes the received object semantic stream to obtain the reconstructed object semantic representation; in medium bandwidth mode, the reconstructed object is obtained based on the reconstructed object semantic representation. The training relation decoding module decompresses and decodes the received relation semantic stream to obtain the reconstructed relation semantic representation; in medium-bandwidth mode, based on the reconstructed relation semantic representation and the object semantic representation of two different reconstructed objects, the reconstructed relation between each pair of reconstructed objects is obtained; In medium bandwidth mode, the training graph generation module generates an object-relationship reconstruction graph of the sample scene based on the reconstructed objects and the reconstruction relationship between every two reconstructed objects.

[0099] Optionally, the semantic receiver performs semantic reconstruction using the received object semantic stream to obtain a reconstructed object semantic representation, and generates an object reconstruction graph of the sample scene based on the reconstructed object semantic representation, including: The object decoding module decompresses and decodes the received object semantic stream to obtain the reconstructed object semantic representation; in low bandwidth mode, the reconstructed object is obtained based on the reconstructed object semantic representation. In low-bandwidth mode, the training image generation module generates object reconstruction images of sample scenes based on the reconstructed objects.

[0100] Optionally, the modules to be trained in the semantic sender and the semantic receiver constitute the model to be trained; the training process of the model to be trained includes at least: Obtain multimodal data and object-attribute-relationship graphs for the sample scenario; The multimodal data of the sample scene is processed by the model to be trained to obtain the reconstructed objects, the reconstruction attributes of the reconstructed objects, and the reconstruction relationship between every two reconstructed objects; Based on the differences between the object labels in the object-attribute-relationship graph of the reconstructed object and the sample scene, the object loss function value is obtained; The attribute loss function value is obtained based on the difference between the reconstructed attributes of the reconstructed object and the attribute labels in the object-attribute-relationship graph of the sample scene. The relationship loss function value is obtained based on the difference between the reconstruction relationship between each pair of reconstructed objects and the relationship label in the object-attribute-relationship graph of the sample scene; The model parameters of the model to be trained are updated based on at least the object loss function value, attribute loss function value, and relation loss function value. After training is completed, each trained module is deployed on the semantic sending end and the semantic receiving end respectively.

[0101] Optionally, during the training process of the model to be trained, low-bandwidth mode, medium-bandwidth mode, and high-bandwidth mode are randomly selected as the current bandwidth mode; the training process of the model to be trained also includes: When the current bandwidth mode is low bandwidth mode, medium bandwidth mode or high bandwidth mode, the object alignment loss function value is obtained based on the difference between the reconstructed object semantic representation and the object semantic representation of the sample scene; When the current bandwidth mode is medium bandwidth mode or high bandwidth mode, the relationship alignment loss function value is obtained based on the difference between the reconstructed relation semantic representation and the relation semantic representation of the sample scene; When the current bandwidth mode is high bandwidth mode, the attribute alignment loss function value is obtained based on the difference between the reconstructed attribute semantic representation and the attribute semantic representation of the sample scene; The model parameters of the model to be trained are updated based on at least the object loss function values, attribute loss function values, and relation loss function values, including: When the current bandwidth mode is low bandwidth mode, the model parameters of the model to be trained should be updated based on at least the object loss function value, attribute loss function value, relation loss function value, and object alignment loss function value. Given that the current bandwidth mode is medium bandwidth mode, the model parameters of the model to be trained should be updated based on at least the object loss function value, attribute loss function value, relation loss function value, object alignment loss function value, and relation alignment loss function value. Given that the current bandwidth mode is high bandwidth mode, the model parameters of the model to be trained should be updated based on at least the object loss function value, attribute loss function value, relation loss function value, object alignment loss function value, attribute alignment loss function value, and relation alignment loss function value.

[0102] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0103] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and are not used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0104] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, such as... Figure 3 As shown, Figure 3 This is a schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device includes a memory, a processor, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the semantic communication method based on adaptive hierarchical transmission of object-attribute-relationship according to any of the above embodiments of the present invention.

[0105] Based on the same inventive concept, another embodiment of the present invention provides a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the semantic communication method based on adaptive hierarchical transmission of object-attribute-relationship as described in any of the above embodiments of the present invention. The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0106] As the method embodiments are basically similar to the system embodiments, the description is relatively simple, and relevant parts can be found in the description of the system embodiments.

[0107] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of the present invention is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0108] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0109] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A semantic communication system based on adaptive hierarchical transmission of object-attribute-relationship, characterized in that, The semantic communication system includes at least: a semantic sending end and a semantic receiving end deployed with an embodied intelligent agent; When the bandwidth mode between the semantic sender and the semantic receiver is high bandwidth mode, the semantic sender uses image data, text data and audio data in the multimodal data of the sample scene to generate object semantic stream, attribute semantic stream and relational semantic stream of the sample scene and sends them to the semantic receiver. The semantic receiver uses the received object semantic stream, attribute semantic stream and relation semantic stream to perform semantic reconstruction, and obtains the reconstructed object semantic representation, reconstructed attribute semantic representation and reconstructed relation semantic representation. Based on the reconstructed object semantic representation, reconstructed attribute semantic representation and reconstructed relation semantic representation, an object-attribute-relation reconstruction graph of the sample scene is generated. The semantic receiver controls the embodied intelligent agent to reconstruct the object-attribute-relationship graph of the sample scene and perform inference tasks corresponding to the high bandwidth mode. When the bandwidth mode between the semantic sender and the semantic receiver changes from high bandwidth mode to medium bandwidth mode, the semantic sender truncates the attribute semantic stream of the sample scene and sends the object semantic stream and relation semantic stream of the sample scene to the semantic receiver; the upper limit of the bandwidth of the medium bandwidth mode is the lower limit of the bandwidth of the high bandwidth mode. The semantic receiver uses the received object semantic stream and relation semantic stream to perform semantic reconstruction, obtain the reconstructed object semantic representation and the reconstructed relation semantic representation, and generate the object-relation reconstruction graph of the sample scene based on the reconstructed object semantic representation and the reconstructed relation semantic representation; The semantic receiver controls the embodied intelligent agent to reconstruct the object-relationship graph of the sample scene and perform the inference task corresponding to the mid-bandwidth pattern.

2. The semantic communication system based on adaptive hierarchical transmission of object-attribute-relationship according to claim 1, characterized in that, When the bandwidth mode between the semantic sender and the semantic receiver changes from medium bandwidth mode to low bandwidth mode, the semantic sender truncates the attribute semantic stream and relation semantic stream of the sample scene and sends the object semantic stream of the sample scene to the semantic receiver; the upper limit of the bandwidth of the low bandwidth mode is the lower limit of the bandwidth of the medium bandwidth mode. The semantic receiver uses the received object semantic stream to perform semantic reconstruction, obtains the reconstructed object semantic representation, and generates an object reconstruction graph of the sample scene based on the reconstructed object semantic representation. The semantic receiver controls the embodied intelligent agent to perform inference tasks corresponding to low-bandwidth modes by reconstructing the object graph of the sample scene.

3. The semantic communication system based on adaptive hierarchical transmission of object-attribute-relationship according to claim 1, characterized in that, The semantic communication system further includes: an adaptive transmission strategy controller; The adaptive transmission strategy controller determines the semantic utility of each semantic stream—object semantic stream, attribute semantic stream, and relational semantic stream—based on their importance to the embodied agent. The semantic utility of the object semantic stream, relational semantic stream, and attribute semantic stream decreases in that order. The adaptive transmission strategy controller determines the available bandwidth between the semantic sender and the semantic receiver. The adaptive policy controller aims to ensure that the total length of the semantic stream to be transmitted does not exceed the available bandwidth and to maximize the total semantic utility of the semantic stream to be transmitted. It assigns transmission masks to object semantic streams, attribute semantic streams, and relational semantic streams. The transmission mask for object semantic streams is 1, while the transmission masks for attribute semantic streams and relational semantic streams are 0 or 1 respectively. A transmission mask of 1 for a semantic stream indicates that the semantic stream is to be transmitted. The adaptive policy controller determines whether the bandwidth mode between the semantic sender and the semantic receiver is high-bandwidth, medium-bandwidth, or low-bandwidth based on the transmission masks assigned to the object semantic stream, attribute semantic stream, and relation semantic stream, and then notifies the semantic sender and the semantic receiver.

4. The semantic communication system based on adaptive hierarchical transmission of object-attribute-relationship according to claim 1, characterized in that, The semantic sender includes: The multimodal feature extraction module to be trained extracts features from the image data, text data, and audio data in the multimodal data of the sample scene, respectively, to obtain the image feature representation, text feature representation, and audio feature representation of the sample scene; The multimodal feature aggregation module to be trained obtains the multimodal aggregated representation of the sample scene based on the learnable image modal representation and the image feature representation of the sample scene, the learnable text modal representation and the text feature representation of the sample scene, and the learnable audio modal representation and the audio feature representation of the sample scene. The decoupling module to be trained decouples the object semantic representation, attribute semantic representation, and relation semantic representation of the sample scene from the multimodal aggregated representation of the sample scene based on the learnable object query representation, learnable attribute query representation, and learnable relation query representation; The sending end object gating module is connected to the object semantic representation output end of the decoupling module to be trained. It is closed in low bandwidth mode, medium bandwidth mode and high bandwidth mode to trigger the semantic sending end to compress the object semantic representation of the sample scene into an object semantic stream and send it to the semantic receiving end. The sending end relation gating module is connected to the relation semantic representation output end of the decoupled module to be trained. It is disconnected in low bandwidth mode and closed in high bandwidth mode or medium bandwidth mode to trigger the semantic sending end to compress the relation semantic representation of the sample scene into a relation semantic stream and send it to the semantic receiving end. The sending-end attribute gating module is connected to the attribute semantic representation output of the decoupled module to be trained. It is disconnected in low-bandwidth or medium-bandwidth mode and closed in high-bandwidth mode to trigger the semantic sending end to compress the attribute semantic representation of the sample scene into an attribute semantic stream and send it to the semantic receiving end.

5. The semantic communication system based on adaptive hierarchical transmission of object-attribute-relationship according to claim 4, characterized in that, The semantic receiver includes: The object decoding module decompresses and decodes the received object semantic stream to obtain the reconstructed object semantic representation; in low bandwidth mode, medium bandwidth mode or high bandwidth mode, the reconstructed object is obtained based on the reconstructed object semantic representation; The attribute decoding module to be trained decompresses and decodes the received attribute semantic stream to obtain the reconstructed attribute semantic representation; in high bandwidth mode, the reconstructed attributes of the reconstructed object are obtained based on the reconstructed attribute semantic representation and the object semantic representation of the reconstructed object. The training relation decoding module decompresses and decodes the received relation semantic stream to obtain the reconstructed relation semantic representation; in medium-bandwidth mode or high-bandwidth mode, based on the reconstructed relation semantic representation and the object semantic representation of two different reconstructed objects, the reconstructed relation of each pair of reconstructed objects is obtained; The training graph generation module generates an object reconstruction graph of the sample scene based on the reconstructed objects in low bandwidth mode; in medium bandwidth mode, it generates an object-relationship reconstruction graph of the sample scene based on the reconstructed objects and the reconstruction relationship between every two reconstructed objects; and in high bandwidth mode, it generates an object-attribute-relationship reconstruction graph of the sample scene based on the reconstructed objects, the reconstruction attributes of the reconstructed objects, and the reconstruction relationship between every two reconstructed objects.

6. The semantic communication system based on adaptive hierarchical transmission of object-attribute-relationship according to claim 5, characterized in that, The various training modules in the semantic sender and semantic receiver constitute the training model; The training process of the model to be trained includes at least the following: Obtain multimodal data and object-attribute-relationship graphs for the sample scenario; The multimodal data of the sample scene is processed by the model to be trained to obtain the reconstructed objects, the reconstruction attributes of the reconstructed objects, and the reconstruction relationship between every two reconstructed objects; Based on the differences between the object labels in the object-attribute-relationship graph of the reconstructed object and the sample scene, the object loss function value is obtained; The attribute loss function value is obtained based on the difference between the reconstructed attributes of the reconstructed object and the attribute labels in the object-attribute-relationship graph of the sample scene. The relationship loss function value is obtained based on the difference between the reconstruction relationship between each pair of reconstructed objects and the relationship label in the object-attribute-relationship graph of the sample scene; The model parameters of the model to be trained are updated based on at least the object loss function value, attribute loss function value, and relation loss function value. After training is completed, each trained module is deployed on the semantic sending end and the semantic receiving end respectively.

7. The semantic communication system based on adaptive hierarchical transmission of object-attribute-relationship according to claim 6, characterized in that, During the training process of the model to be trained, low bandwidth mode, medium bandwidth mode and high bandwidth mode are randomly selected as the current bandwidth mode; The training process of the model to be trained also includes: When the current bandwidth mode is low bandwidth mode, medium bandwidth mode or high bandwidth mode, the object alignment loss function value is obtained based on the difference between the reconstructed object semantic representation and the object semantic representation of the sample scene; When the current bandwidth mode is medium bandwidth mode or high bandwidth mode, the relationship alignment loss function value is obtained based on the difference between the reconstructed relation semantic representation and the relation semantic representation of the sample scene; When the current bandwidth mode is high bandwidth mode, the attribute alignment loss function value is obtained based on the difference between the reconstructed attribute semantic representation and the attribute semantic representation of the sample scene; The model parameters of the model to be trained are updated based on at least the object loss function values, attribute loss function values, and relation loss function values, including: When the current bandwidth mode is low bandwidth mode, the model parameters of the model to be trained should be updated based on at least the object loss function value, attribute loss function value, relation loss function value, and object alignment loss function value. Given that the current bandwidth mode is medium bandwidth mode, the model parameters of the model to be trained should be updated based on at least the object loss function value, attribute loss function value, relation loss function value, object alignment loss function value, and relation alignment loss function value. Given that the current bandwidth mode is high bandwidth mode, the model parameters of the model to be trained should be updated based on at least the object loss function value, attribute loss function value, relation loss function value, object alignment loss function value, attribute alignment loss function value, and relation alignment loss function value.

8. A semantic communication method based on adaptive hierarchical transmission of object-attribute-relationship, characterized in that, The method, applied to the semantic communication system based on adaptive hierarchical transmission of object-attribute-relationship as described in any one of claims 1 to 7, comprises: When the bandwidth mode between the semantic sender and the semantic receiver is high bandwidth mode, the semantic sender uses image data, text data and audio data in the multimodal data of the sample scene to generate object semantic stream, attribute semantic stream and relational semantic stream of the sample scene and sends them to the semantic receiver. The semantic receiver uses the received object semantic stream, attribute semantic stream and relation semantic stream to perform semantic reconstruction, and obtains the reconstructed object semantic representation, reconstructed attribute semantic representation and reconstructed relation semantic representation. Based on the reconstructed object semantic representation, reconstructed attribute semantic representation and reconstructed relation semantic representation, an object-attribute-relation reconstruction graph of the sample scene is generated. The semantic receiver controls the embodied intelligent agent to reconstruct the object-attribute-relationship graph of the sample scene and perform inference tasks corresponding to the high bandwidth mode. When the bandwidth mode between the semantic sender and the semantic receiver changes from high bandwidth mode to medium bandwidth mode, the semantic sender truncates the attribute semantic stream of the sample scene and sends the object semantic stream and relation semantic stream of the sample scene to the semantic receiver; the upper limit of the bandwidth of the medium bandwidth mode is the lower limit of the bandwidth of the high bandwidth mode. The semantic receiver uses the received object semantic stream and relation semantic stream to perform semantic reconstruction, obtain the reconstructed object semantic representation and the reconstructed relation semantic representation, and generate the object-relation reconstruction graph of the sample scene based on the reconstructed object semantic representation and the reconstructed relation semantic representation; The semantic receiver controls the embodied intelligent agent to reconstruct the object-relationship graph of the sample scene and perform the inference task corresponding to the mid-bandwidth pattern.

9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein when the program or instructions are executed by the processor, they implement the steps of the semantic communication method based on adaptive hierarchical transmission of objects-attributes-relationships as described in claim 8.

10. A readable storage medium, characterized in that, The program or instructions stored on the readable storage medium implement the steps of the semantic communication method based on adaptive hierarchical transmission of object-attribute-relationship as described in claim 8 when the program or instructions are executed by a processor.

Images