Real-time three-dimensional semantic map construction method based on semantic communication

Abstract

The invention discloses a real-time three-dimensional semantic map construction method based on semantic communication, and belongs to the technical field of semantic communication. The method comprises the following steps: designing a communication system scene model combined with cloud edge collaboration according to real-time three-dimensional semantic mapping task requirements under limited resources, and simplifying the communication system scene model into a basic communication transmission model; then, according to the simplified communication transmission model, designing a coding and decoding structure semantic communication framework with an edge node end as a coding end and an edge server as a decoding end, and meanwhile, adding a quantization module on framework design to realize maximum compression of semantic information, so as to ensure that a real-time semantic mapping task can be completed on the edge node with limited resources; and finally, designing an attention module based on an attention mechanism according to different channel condition requirements of a scene in a communication scene, and embedding the module into the encoding and decoding structure semantic communication framework to ensure that the method can correctly complete a semantic mapping task in real time under various channel conditions.

Description

Technical Field

[0001] The present invention belongs to the technical field of semantic communication, and in particular relates to a real-time three-dimensional semantic map construction method based on semantic communication. Background Art

[0002] Cloud-edge collaborative computing refers to the use of cloud computing, edge computing, and collaborative computing technologies to achieve collaborative work and resource sharing between cloud and edge devices. This technology can effectively combine the powerful computing power of the cloud and the good mobility of the edge to serve applications in various scenarios. This involves the integration and collaboration of edge computing and cloud computing. Edge computing pushes computing and data storage capabilities to the edge of the network to reduce data transmission delays and network loads, and support real-time response. Cloud computing provides large-scale computing and storage resources to support complex data processing and analysis tasks. Cloud-edge collaboration requires the effective combination of the two to achieve dynamic resource allocation and collaborative execution of tasks. By effectively realizing cloud-edge collaboration, the performance and availability of the system can be improved, and various application scenarios based on cloud-edge collaboration, such as smart Internet of Things and smart cities, can be supported.

[0003] Semantic communication technology is a semantic-oriented information exchange technology. It uses a structured and standardized way to send the meaning of the information to be transmitted, so that the computer can clearly transmit the main information based on the transmitted meaning information, which can help people more effectively obtain, analyze, integrate and understand information, improve work efficiency and provide a better service experience. Especially in data compression, semantic communication technology can compress image data more finely and efficiently. Compared with the traditional compression method based on prediction and transformation, which has the problems of distortion and information loss, semantic communication technology can perform semantic analysis on the image content, thereby removing redundant information and compressing using image features, making the compression efficiency higher, the distortion less, and the information richer. A large amount of effective information can be transmitted with a small amount of data.

[0004] Semantic mapping technology refers to creating a structured map for a robot to represent its semantic information, so as to help the robot understand and process information in the environment and support the execution of related tasks. This technology is crucial in the field of robotics, providing a good foundation for the robot's own accurate positioning and service of high-level task requirements. In terms of construction, robot semantic map construction relies on perception and perception fusion technology. It often uses multiple perception data such as vision, lidar, and sound. Perception fusion technology integrates the acquired information to form a comprehensive understanding of the environment. Secondly, semantic map construction requires semantic understanding and knowledge representation. That is, the perceived environmental information is converted into a form that can be understood by the machine, and a semantic representation related to the task is established. Robot semantic map construction often focuses on map construction accuracy and real-time mapping. High-precision and real-time construction is often based on powerful edge computing nodes. In previous studies, issues such as resource consumption have not been considered.

[0005] Applying the cloud-edge collaboration concept to resource-constrained real-time semantic mapping tasks can utilize the powerful computing power of the cloud to assist in the completion of real-time semantic mapping tasks and reduce the cost of completing semantic mapping tasks. Secondly, the integration of semantic communication methods not only improves data transmission efficiency, but also provides support for real-time completion of tasks. In combination with the attention mechanism, effective transmission under different channel conditions is achieved, ensuring efficient communication and correct completion of semantic mapping tasks. Summary of the invention

[0006] Purpose of the invention: The present invention aims to provide a real-time three-dimensional semantic map construction method based on semantic communication, which mainly uses semantic communication to solve the real-time semantic mapping task with high resource requirements on edge devices with limited resources.

[0007] Technical solution: A real-time three-dimensional semantic map construction method based on semantic communication, comprising the following steps:

[0008] (1) Based on the requirements of real-time 3D semantic mapping tasks under limited resources, a communication system scenario model combining cloud-edge collaboration is designed and simplified to the basic communication transmission model;

[0009] The communication system model considers two types of devices: transmitters represented by edge devices and receivers represented by edge servers. The communication tasks performed by the two types of devices include:

[0010] The edge server trains the semantic communication encoder-decoder model and then distributes the semantic encoder model to all edge devices via wireless transmission;

[0011] The edge device fine-tunes the model according to the actual scenario and extracts semantic features, which are then sent to the edge server for semantic recovery;

[0012] The edge server recovers data based on the received semantic information and completes the semantic mapping task;

[0013] (2) According to the communication transmission model, a semantic communication framework with an encoding and decoding structure is proposed, with the edge node as the encoding end and the edge server as the decoding end. At the same time, a vector quantization module is added to the framework design to achieve maximum compression of semantic information, ensuring that real-time semantic mapping tasks can be completed on edge nodes with limited resources.

[0014] Encoding end: including semantic extraction module, channel encoder and knowledge base, the knowledge base is used to store background knowledge shared between the transmitter and the receiver; RGB-D sensor is selected as the data acquisition sensor to capture RGB frame and depth frame. The RGB image will be sent to the semantic extraction module, and the depth image will be directly sent to the channel encoding module;

[0015] Decoding end: including channel decoder, semantic recovery module, frame alignment module and semantic mapping module. The frame alignment module matches RGB frames and depth frames according to preset timestamps. Only the successfully aligned image frame pairs will be sent to the subsequent semantic mapping module. The semantic mapping module includes semantic positioning, semantic extraction and semantic map generation.

[0016] (3) According to the different channel conditions in the communication scenario, an attention module is designed based on the attention mechanism and the perceived signal-to-noise ratio, and the attention module is embedded in the codec structure semantic communication framework described in step (2) to ensure that the method can complete the semantic mapping task in real time and correctly under various channel conditions.

[0017] Furthermore, in the simplified model, the transmitter is responsible for extracting semantic features and transmitting them through the channel. The transmitter includes semantic extraction and channel encoder. Assuming that the image source data is s, the entire transmission process can be expressed as:

[0018] x=P μ (E α (s))

[0019] Among them, E α (·) represents the semantic extraction network with training parameter α, P μ (·) refers to the channel encoder network with training parameters μ, and the output x is later fed into the channel for transmission;

[0020] In order to consider channel fluctuations and achieve channel variability, an adaptive white noise channel is selected as the system channel, and the formula is expressed as follows:

[0021] y=x+n

[0022] Where n represents the channel noise, which follows the normal distribution N(0,σ2 ), y represents the channel output, which will be transmitted to the receiver in the subsequent steps;

[0023] The receiver is responsible for recovering the semantic information and building a semantic map. The process includes channel decoder, semantic recovery and semantic mapping, which is expressed as follows:

[0024]

[0025] Where Q ν (·) represents the channel decoder network with training parameters ν, R β (·) represents the semantic decoder network with training parameter β;

[0026] The semantic restoration process is the inverse process of semantic feature extraction. The output of semantic restoration It will serve as the input for subsequent semantic mapping;

[0027] In terms of semantic mapping, the method includes using semantic synchronous positioning and mapping technology to generate a semantic mapping map M in real time; the simplified model also includes a knowledge base for sharing background knowledge between the transmitter and the receiver, and the knowledge base is represented by the structure and parameters of the neural network, which helps to understand and interpret the exchanged information.

[0028] Furthermore, the semantic extraction module described in step (2) includes a semantic encoder, an attention module and a vector quantization module; the semantic encoder extracts semantic features using a method based on a convolutional neural network to obtain semantic features; the attention module is designed based on the perceptual signal-to-noise ratio and is used to adapt to various channel conditions; the vector quantization module is used to compress semantic information and map continuous information into finite-dimensional discrete vectors, thereby achieving maximum compression without destroying the semantic data.

[0029] Furthermore, the semantic restoration module includes an attention module and a CNN-based semantic decoder, which is used to restore the semantic feature information to the original RGB frame.

[0030] The vector quantization model finds the closest discrete semantic embedding vector for continuous input data, ensuring that the sender only needs to send these semantic vectors instead of the entire semantic features, thereby reducing the amount of data transmission; the vector quantization module will search for the nearest vector in the semantic embedding space, and the mathematical expression is as follows:

[0031] n=arg n min‖f m -e n ||2,

[0032] Among them, f m is the semantic encoding result, e nIt is the semantic embedding space shared by the sender and the receiver;

[0033] After vector quantization, the discrete semantic features are input into the channel encoder together with the depth frame and then sent to the channel for transmission.

[0034] Furthermore, in order to adapt to various channel conditions in the communication process, an attention module is designed based on the attention mechanism and the perceived signal-to-noise ratio. The corresponding attention weights of channels with different signal-to-noise ratios are automatically learned through the training model. The fully connected layer is used with the signal-to-noise ratio as input. The sigmoid function is used in the attention module to obtain the attention weights, and then the attention weights are expanded to match the shape of the input features and then multiplied by the input features. The attention layer is expressed as:

[0035] z f+1 =f FCA (z f ,snr)*z f

[0036] Among them, z f and z f+1 denote the input and output features respectively, and f FCA(·) It consists of a connection layer and a sigmoid function.

[0037] Beneficial effects: The method described in the present invention constructs an encoding-decoding semantic communication framework for completing real-time semantic mapping tasks under limited resources. In addition, considering the impact of different channel conditions on communication, a module based on the attention mechanism is designed to achieve stable data transmission under different channel conditions. In terms of simulation experiments, based on the TUM dataset, it is verified that the system has an error of less than 0.1% in mapping and positioning accuracy compared with the ground truth, and is superior to traditional communication algorithms in terms of real-time performance and channel adaptation. In addition, by implementing a prototype system, the effectiveness of the proposed framework and designed modules in actual indoor scenes is verified. The results show that this method can complete real-time semantic mapping tasks for indoor objects (chairs, computers, people, etc.) under limited resources, and the mapping update time is less than 1 second. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is the system scenario graph;

[0039] Figure 2 To simplify the system framework diagram;

[0040] Figure 3 Building an encoder-decoder framework for semantic communication mapping systems;

[0041] Figure 4 Design for attention module structure;

[0042] Figure 5 Figure 1 is the structure diagram of the transmitter network;

[0043] Figure 6 is the receiver network structure diagram;

[0044] Figure 7 For real-time performance comparison of data transmission;

[0045] Figure 8 The semantic mapping effect using the Tum dataset;

[0046] Fig. 9 This is an ablation experiment to test the transmission quality with and without the attention module at different SNRs;

[0047] Fig.10 This is the result of semantic mapping of real indoor objects in the embodiment. DETAILED DESCRIPTION

[0048] The present invention is described in detail below with reference to the accompanying drawings and specific examples.

[0049] The present invention provides a real-time three-dimensional semantic map construction method based on semantic communication, which introduces the idea of ​​cloud-edge collaborative computing into the real-time semantic mapping task under resource-constrained conditions, and designs a semantic communication framework in combination with the high data transmission efficiency of semantic communication to achieve the real-time mapping task in resource-limited edge devices. The key problem to be solved by the present invention is how to complete the real-time semantic mapping task in a cloud-edge collaborative computing manner and maintain high data transmission efficiency in the process when mobile device resources are limited, and correctly complete the mapping under various channel conditions. Based on semantic communication technology, the semantic features of the image are extracted and then transmitted and restored, which is used as the data source for the semantic mapping task. Figure 1 As shown, the scenario considered is an edge computing node with limited resources and an edge server with powerful computing power. Figure 2 The communication codec structure is formed by simplifying the system model. Figure 3 A codec semantic communication framework designed based on a simplified communication structure. Figure 4 The attention design module implemented by the signal-to-noise ratio-aware attention mechanism enables effective data transmission under various channel conditions. Figure 5 Design a diagram for the transmitter network structure to achieve semantic extraction of data and channel coding transmission. Figure 6 Design a graph for the receiver network structure to achieve semantic recovery of semantic features and semantic mapping. Figure 7 The real-time performance comparison of data transmission proves that the proposed method is superior to the traditional algorithm in real-time performance. Figure 8 The transmission performance under different signal-to-noise ratio conditions is shown, which proves that the transmission performance of this method is better than that of the traditional algorithm under different channel conditions. Fig. 9The mapping effect and positioning comparison achieved by this method using the Tum dataset show that this method can correctly complete the mapping task and perform well in positioning. Fig.10 The specific implementation process of the method of the present invention is as follows:

[0050] Step 1: Design a communication system scenario model

[0051] Two types of devices are considered in the communication system scenario model: transmitters represented by edge devices and receivers represented by edge servers. The former has limited computing resources but good mobility, while the latter has strong computing power and can generate semantic maps in real time. The specific communication process of the two types of devices can be divided into the following steps:

[0052] 1) The edge server trains the semantic communication encoder-decoder model and then distributes the semantic encoder model to all edge devices via wireless transmission.

[0053] 2) The edge device fine-tunes the model according to the actual scenario and extracts semantic features, which are then sent to the edge server for semantic recovery.

[0054] 3) The edge server recovers data based on the received semantic information and completes the semantic mapping task.

[0055] We first simplify the above model and combine Figure 1 and Figure 2 As shown in the simplified model, the transmitter is responsible for extracting semantic features and transmitting them through the channel. This process mainly consists of two parts: semantic extraction and channel encoder. Assuming that the image source data is s, the entire transmission process can be expressed as:

[0056] x=P μ (E α (s)), (1)

[0057] Among them, E α (·) represents the semantic extraction network with training parameter α, P μ (·) refers to the channel encoder network with training parameters μ. The output x is later fed into the channel for transmission.

[0058] In order to consider channel fluctuations and achieve channel variability, the adaptive white noise channel (AWGN) is selected as the system channel, which is expressed by formula (2):

[0059] y=x+n, (2)

[0060] Where n represents the channel noise, which follows the normal distribution N(0,σ 2 ), y represents the channel output, which will be transmitted to the receiver in the subsequent steps.

[0061] The receiver is responsible for recovering the semantic information and building a semantic map. This process mainly includes three parts: channel decoder, semantic recovery and semantic mapping. As shown in formula (3), the semantic recovery process is the inverse process of semantic feature extraction.

[0062]

[0063] Where Q ν (·) represents the channel decoder network with training parameters ν, R β (·) represents the semantic decoder network with training parameters β. The output of semantic restoration It will be used as the input for subsequent semantic mapping. In terms of semantic mapping, the semantic simultaneous localization and mapping (SLAM) technology is used to generate the semantic map M in real time.

[0064] Furthermore, the knowledge base is an essential component of the system model. The background knowledge shared between the sender and the receiver helps to understand and interpret the exchanged information. In this approach, the knowledge base is represented by the structure and parameters of the neural network.

[0065] Step 2: Design the encoding and decoding structure semantic communication framework

[0066] 2.1 System Framework

[0067] The whole system consists of an encoder-decoder structure. On the encoding side, an RGB-D sensor is selected as the data acquisition sensor, which can capture both RGB frames and depth frames. Compared with the RGB sensor, the depth information helps to achieve more effective positioning. The RGB image will be fed into the semantic extraction module, while the depth image will be directly fed into the channel coding module due to the lack of semantic information. The semantic extraction module consists of three parts: the semantic encoder, the attention module, and the vector quantization module. The semantic encoder extracts semantic features using a convolutional neural network (CNN)-based method. After obtaining the semantic features, an attention module is added to adapt to various channel conditions. The attention module is designed based on the perceptual signal-to-noise ratio.

[0068] In order to further compress the semantic information, a vector quantization module is specially designed in the codec structure semantic communication framework. The vector quantization module can map continuous information into finite-dimensional discrete vectors, thereby achieving maximum compression without destroying the semantic data. The quantized and compressed data will be concatenated with the depth frame and then sent to the channel by the channel encoder.

[0069] At the receiving end, channel decoding is first performed after receiving the signal. It helps to restore the signal to the previous semantic information. The recovery process of semantic information is similar to the inverse process of the previous transmission process. Specifically, it consists of two parts: the attention module and the CNN-based semantic decoder. Its ultimate goal is to restore the semantic feature information to the original RGB frame. Compared with the complex processing process of RGB images, depth images can be transmitted directly and faster than RGB images. Therefore, when restoring RGB frames, there will be misalignment between them due to the time difference.

[0070] In order to solve the misalignment problem, a frame alignment module is specially designed to match RGB frames and depth frames according to the preset timestamp. Only the successfully aligned image frame pairs will be sent to the subsequent semantic mapping. For semantic mapping, high map accuracy and good real-time performance are our ultimate goals. This module is divided into three parts: positioning, semantic extraction, and semantic map generation. In order to achieve good real-time performance, these three modules are deployed in three threads for parallel execution.

[0071] In this encoding and decoding architecture, the encoding end only needs to extract semantic information, which requires very little computing power. At the same time, the amount of quantized semantic information is very small, which can ensure good real-time performance and complete the real-time semantic mapping task at the decoding end.

[0072] 2.2 Transmitter Design

[0073] The role of the transmitter is to extract semantic information from the source data, which mainly includes the semantic encoder, attention module and vector quantization. Figure 5 As shown in Figure 5, they are almost all implemented through neural networks: the semantic encoder module is based on CNN, the attention module is based on the perceived signal-to-noise ratio, and the vector quantization module generates a quantization matrix based on the embedded semantic space.

[0074] The semantic encoder is constructed from the RGB frame S∈R via two downsampling modules. H×W×3 Semantic features are extracted from , where H, W, and 3 are the image width, image height, and number of channels, respectively. Specifically, the semantic encoder module includes two convolutional downsampling modules and two residual downsampling modules. Each convolutional downsampling module includes a convolutional layer and a batch normalization layer. Between the two modules, there is an activation function ReLU to improve the nonlinear ability. After the convolutional downsampling operation, there are two residual downsampling modules, which include a residual layer based on ResNet and a batch normalization layer. In this way, the semantic features are extracted.

[0075] After semantic encoding, the semantic features are sent to the attention module and the attention weight is added. The feature tensor output by the attention module enters the vector quantization part to compress the data volume. The principle of vector quantization is that continuous input data will find the closest discrete semantic embedding vector, ensuring that the transmitter only needs to send these semantic vectors instead of the entire semantic features, thereby greatly reducing the amount of data transmission. Specifically, the vector quantization module will search for the nearest vector in the semantic embedding space, as shown in formula (4):

[0076] n=arg n min||f m -e n ||2, (4)

[0077] Among them, f m is the semantic encoding result, e n It is the semantic embedding space shared by the sender and the receiver. After vector quantization, the discrete semantic features are input into the channel encoder together with the depth frame and then sent to the channel for transmission.

[0078] 3.2 Receiver Design

[0079] The role of the receiver is to recover the extracted semantic information and use the recovery results for semantic mapping. Figure 6 As shown in the figure, it can be divided into three parts: attention module, semantic decoder and semantic mapping. After passing through the attention module, the semantic decoder restores the semantic features to the original image. In the network design, the components of the decoder network are similar to those of the encoder network, but the functions are opposite. Figure 6 The upsampling part shown is composed of a deconvolution layer, a residual block, and a batch normalization layer connected in sequence. Like the semantic encoder, the role of ReLU is to add nonlinearity. After deconvolution and upsampling, the image will be restored by tanh to [0,1]. In addition to the semantic recovery part, the receiver also includes a semantic mapping part. In order to ensure the real-time operation of the system, three threads are used in parallel. For the positioning thread, this method is designed according to ORB-SLAM3, removing the mapping part and retaining only self-localization and closed loop detection. In the semantic extraction module, semantic segmentation is selected to obtain high-precision semantic information. In addition, considering the real-time requirements of the system, it is necessary to complete the semantic segmentation task as soon as possible. Therefore, SCTNet is selected as the semantic segmentation network, and its structure effectively separates training and reasoning to ensure real-time performance. After obtaining the semantic information, the semantic mapping module will construct a semantic point cloud based on the depth information and semantic formation. In order to better fuse the semantic information and the point cloud, the Bayesian fusion method is selected to achieve high-precision semantic point cloud fusion. Finally, a semantic map is generated in the form of an octree map to save storage space and improve mapping efficiency.

[0080] Step 3: Integrate the attention module framework

[0081] In neural networks, attention is a mechanism that redistributes resources that were originally evenly distributed according to the importance of the attention object. First, consider the difference between channel attention and spatial attention. Channel attention aims to calculate the (C×1×1) channel weights, while spatial attention calculates the spatial weights of (1×H×W), where C represents the number of channels, and H and W represent the height and width of the feature, respectively. Channel attention focuses on the input of information from different channels, while spatial attention focuses mainly on the input of information from different positions. However, channel attention and spatial attention do not focus on resource allocation under different channel conditions. In response to this, we propose an attention module based on signal-to-noise ratio perception, which automatically learns the corresponding attention weights for channels with different signal-to-noise ratios by training the model. As Figure 4 As shown in Figure 1, a fully connected layer is used with the signal-to-noise ratio (SNR) as input, and the sigmoid function is used in the attention module to obtain the attention weights. The attention weights are then expanded to match the shape of the input features and multiplied by the input features.

[0082] The attention layer can be expressed as:

[0083] z f+1 =f FCA (z f ,snr)*z f (5)

[0084] Among them, z f and z f+1 represent the input and output features respectively. FCA(·) It consists of a connection layer and a sigmoid function.

[0085] The present invention discloses a real-time three-dimensional semantic map construction method based on semantic communication, which plays an increasingly important role in high-precision robot positioning and scene understanding in recent years. However, the real-time construction of semantic maps requires mobile edge devices with extremely high computing power, which are expensive and limit the widespread application of semantic maps. In view of this limitation, inspired by cloud-edge collaborative computing and high transmission efficiency of semantic communication, the present invention proposes a method for implementing real-time semantic mapping tasks using mobile devices with limited resources. In the present invention, an encoding-decoding semantic communication framework is designed to complete real-time semantic mapping tasks under limited resources. In addition, considering the impact of different channel conditions on communication, a module based on the attention mechanism is designed to achieve stable data transmission under different channel conditions. In terms of simulation experiments, based on the TUM dataset, it is verified that the system has an error of less than 0.1% compared with the ground truth in terms of mapping and positioning accuracy, and is superior to traditional communication algorithms in terms of real-time performance and channel adaptation. In addition, by implementing a prototype system, the effectiveness of the proposed framework and designed modules in actual indoor scenes is verified. The results show that this method can complete the real-time semantic mapping task of indoor objects (chairs, computers, people, etc.) under limited resources, and the mapping update time is less than 1 second.

Claims

1. A real-time three-dimensional semantic map construction method based on semantic communication, characterized in that: The steps include: (1) Based on the requirements of real-time 3D semantic mapping tasks under limited resources, a communication system scenario model combining cloud-edge collaboration is designed and simplified to the basic communication transmission model; The communication system model considers two types of devices: transmitters represented by edge devices and receivers represented by edge servers. The communication tasks performed by the two types of devices include: The edge server trains the semantic communication encoder-decoder model and then distributes the semantic encoder model to all edge devices via wireless transmission; The edge device fine-tunes the model according to the actual scenario and extracts semantic features, which are then sent to the edge server for semantic recovery; The edge server recovers data based on the received semantic information and completes the semantic mapping task; (2) According to the communication transmission model, a semantic communication framework with an encoding and decoding structure is proposed, with the edge node as the encoding end and the edge server as the decoding end. At the same time, a vector quantization module is added to the framework design to achieve maximum compression of semantic information, ensuring that real-time semantic mapping tasks can be completed on edge nodes with limited resources. Encoding end: including semantic extraction module, channel encoder and knowledge base, the knowledge base is used to store background knowledge shared between the transmitter and the receiver; RGB-D sensor is selected as the data acquisition sensor to capture RGB frame and depth frame, RGB image will be sent to the semantic extraction module, and the depth image will be directly sent to the channel encoding module; Decoding end: including channel decoder, semantic recovery module, frame alignment module and semantic mapping module. The frame alignment module matches RGB frames and depth frames according to preset timestamps. Only the successfully aligned image frame pairs will be sent to the subsequent semantic mapping module. The semantic mapping module includes semantic positioning, semantic extraction and semantic map generation. (3) According to the different channel conditions in the communication scenario, an attention module is designed based on the attention mechanism and the perceived signal-to-noise ratio, and the attention module is embedded in the codec structure semantic communication framework described in step (2) to ensure that the method can complete the semantic mapping task in real time and correctly under various channel conditions.

2. The method for constructing a real-time three-dimensional semantic map based on semantic communication according to claim 1, characterized in that: In the simplified model, the transmitter is responsible for extracting semantic features and transmitting them through the channel. The transmitter includes semantic extraction and channel encoder. Assuming that the image source data is s, the entire transmission process can be expressed as: x=P μ (E α (s)) Among them, E α (·) represents the semantic extraction network with training parameter α, P μ (·) refers to the channel encoder network with training parameters μ, and the output x is later fed into the channel for transmission; In order to consider channel fluctuations and achieve channel variability, an adaptive white noise channel is selected as the system channel, and the formula is expressed as follows: y=x+n Where n represents the channel noise, which follows the normal distribution N(0,σ 2 ), y represents the channel output, which will be transmitted to the receiver in the subsequent steps; The receiver is responsible for recovering the semantic information and building a semantic map. The process includes channel decoder, semantic recovery and semantic mapping, which is expressed as follows: Where Q ν (·) represents the channel decoder network with training parameters ν, R β (·) represents the semantic decoder network with training parameter β; The semantic restoration process is the inverse process of semantic feature extraction. The output of semantic restoration It will be used as input for subsequent semantic mapping; In terms of semantic mapping, the method includes using semantic synchronous positioning and mapping technology to generate a semantic map M in real time; the simplified model also includes a knowledge base for sharing background knowledge between the transmitter and the receiver, and the knowledge base is represented by the structure and parameters of the neural network, which helps to understand and interpret the exchanged information.

3. The method for constructing a real-time three-dimensional semantic map based on semantic communication according to claim 1, characterized in that: The semantic extraction module described in step (2) includes a semantic encoder, an attention module and a vector quantization module; the semantic encoder extracts semantic features using a method based on a convolutional neural network to obtain semantic features; the attention module is designed based on the perceptual signal-to-noise ratio and is used to adapt to various channel conditions; the vector quantization module is used to compress semantic information and map continuous information into finite-dimensional discrete vectors, thereby achieving maximum compression without destroying the semantic data.

4. The method for constructing a real-time three-dimensional semantic map based on semantic communication according to claim 1, characterized in that: The designed semantic restoration module includes an attention module and a CNN-based semantic decoder to restore the semantic feature information to the original RGB frame.

5. The method for constructing a real-time three-dimensional semantic map based on semantic communication according to claim 1 or 3, characterized in that: The vector quantization model finds the closest discrete semantic embedding vector for continuous input data, ensuring that the sender only needs to send these semantic vectors instead of the entire semantic features, thereby reducing the amount of data transmission; the vector quantization module will search for the nearest vector in the semantic embedding space, and the mathematical expression is as follows: n=angry n my‖f m -e n ‖2, Among them, f m is the semantic encoding result, e n It is the semantic embedding space shared by the sender and the receiver; After vector quantization, the discrete semantic features are input into the channel encoder together with the depth frame and then sent to the channel for transmission.

6. The method for constructing a real-time three-dimensional semantic map based on semantic communication according to claim 3, characterized in that: In order to adapt to various channel conditions in the communication process, the attention module is designed based on the attention mechanism and the perceived signal-to-noise ratio. The corresponding attention weights of channels with different signal-to-noise ratios are automatically learned through the training model. The fully connected layer is used with the signal-to-noise ratio as input. The sigmoid function is used in the attention module to obtain the attention weights, and then the attention weights are expanded to match the shape of the input features and then multiplied by the input features. The attention layer is expressed as: With f+1 =f FCA (With f ,snr)*z f Among them, z f and z f+1 denote the input and output features respectively, and f FCA(·) It consists of a connection layer and a sigmoid function.

Images