Terminal equipment, network equipment and wireless communication methods

By utilizing CSI to determine the semantic features of text and images in MIMO channels and performing differentiated channel subspace mapping, the challenge of complex spatial fading in MIMO channels to semantic information transmission is solved, achieving efficient and reliable semantic information transmission and image reconstruction.

CN123074534APending Publication Date: 2026-11-10LENOVO (BEIJING) LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611153001.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-30
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

In multiple-input multiple-output (MIMO) channels, the complex spatial fading characteristics of MIMO channels place higher demands on the accurate transmission of semantic information, and existing technologies are unable to effectively utilize channel characteristics for efficient information transmission.

Method used

The terminal device receives MIMO Channel State Information (CSI) from the network device to determine text semantic features, region of interest (ROI) semantic features, and non-ROI semantic features of the image. It then performs joint source-channel coding on these features and maps them to channel subspaces of different channel qualities based on the CSI, transmitting text semantic signals, ROI semantic signals, and non-ROI semantic signals. The network device then performs signal detection and decoding to reconstruct the image.

Benefits of technology

It enables reliable transmission of key semantic information in complex channel environments, improves the robustness and reconstruction quality of cross-modal semantic communication, and ensures the identification and differentiation of key semantic regions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123074534A_ABST
    Figure CN123074534A_ABST
Patent Text Reader

Abstract

This application discloses a terminal device, a network device, and a wireless communication method. The terminal device receives Channel State Information (CSI) of a Multiple-Input Multiple-Output (MIMO) channel sent by the network device; determines text semantic features, Region of Interest (ROI) semantic features of an image, and non-ROI semantic features of the image; the ROI semantic features and the non-ROI semantic features are determined using text, which is associated with the image; joint source-channel coding is performed on the text semantic features, the ROI semantic features, and the non-ROI semantic features to obtain text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols; based on the CSI, the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols are mapped to channel subspaces of different channel qualities to obtain text semantic signals, ROI semantic signals, and non-ROI semantic signals; and the text semantic signals, the ROI semantic signals, and the non-ROI semantic signals are transmitted to the network device through the MIMO channel.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of communication technology, specifically to a terminal device, a network device, and a wireless communication method. Background Technology

[0002] In wireless communication systems, efficient information transmission between terminal devices and network devices increasingly relies on the precise utilization of channel characteristics. With the widespread application of Multiple-Input Multiple-Output (MIMO) technology, communication systems can achieve spatial multiplexing and diversity gain through multiple antennas, thereby improving data transmission rates and reliability. However, the complex spatial fading characteristics of MIMO channels place higher demands on the accurate transmission of semantic information. Summary of the Invention

[0003] This application provides at least one terminal device, a network device, and a wireless communication method.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a terminal device, the terminal device including a first transceiver; and A first processor, coupled to the first transceiver; the first processor is configured to: The first transceiver receives Channel State Information (CSI) of the Multiple-Input Multiple-Output MIMO channel sent by the network device. Determine textual semantic features, region of interest (ROI) semantic features of the image, and non-ROI semantic features of the image; the ROI semantic features and the non-ROI semantic features are determined using text, which is associated with the image; Joint source-channel coding is performed on the text semantic features, the ROI semantic features, and the non-ROI semantic features to obtain text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols. Based on the CSI, the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols are mapped to channel subspaces of different channel qualities to obtain text semantic signals, ROI semantic signals, and non-ROI semantic signals; The text semantic signal, the ROI semantic signal, and the non-ROI semantic signal are transmitted to the network device via the first transceiver and the MIMO channel.

[0005] This application provides a network device, the network device including a second transceiver; and A second processor, coupled to the second transceiver; the second processor is configured to: The channel state information (CSI) of the multiple-input multiple-output (MIMO) channel is sent to the terminal device via the second transceiver. The second transceiver receives text semantic signals, ROI semantic signals, and non-ROI semantic signals sent by the terminal device via the MIMO channel. Signal detection is performed on the text semantic signal, the ROI semantic signal, and the non-ROI semantic signal respectively to obtain text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols; Joint source-channel decoding is performed on the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols respectively to obtain text semantic features, ROI semantic features, and non-ROI semantic features; Based on the CSI, the text semantic features, the ROI semantic features, and the non-ROI semantic features are restored to the reconstructed image.

[0006] This application provides a wireless communication method, which is applied to a terminal device, and the method includes: Receive Channel State Information (CSI) for the Multiple-Input Multiple-Output (MIMO) channel sent by the network device; Determine textual semantic features, region of interest (ROI) semantic features of the image, and non-ROI semantic features of the image; the ROI semantic features and the non-ROI semantic features are determined using text, which is associated with the image; Joint source-channel coding is performed on the text semantic features, the ROI semantic features, and the non-ROI semantic features to obtain text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols. Based on the CSI, the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols are mapped to channel subspaces of different channel qualities to obtain text semantic signals, ROI semantic signals, and non-ROI semantic signals; The text semantic signal, the ROI semantic signal, and the non-ROI semantic signal are transmitted to the network device through the MIMO channel.

[0007] This application provides a wireless communication method, which is applied to a network device, and the method includes: Send Channel State Information (CSI) of the Multiple-Input Multiple-Output MIMO channel to the terminal device; The terminal device receives text semantic signals, region of interest (ROI) semantic signals, and non-ROI semantic signals via a MIMO channel. Signal detection is performed on the text semantic signal, the ROI semantic signal, and the non-ROI semantic signal respectively to obtain text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols; Joint source-channel decoding is performed on the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols respectively to obtain text semantic features, ROI semantic features, and non-ROI semantic features; Based on the CSI, the text semantic features, the ROI semantic features, and the non-ROI semantic features are restored to the reconstructed image.

[0008] This application provides a wireless communication device, the device comprising: The first receiving module is configured to receive channel state information (CSI) of the multiple-input multiple-output (MIMO) channel sent by the network device; The first determining module is configured to determine text semantic features, region of interest (ROI) semantic features of the image, and non-ROI semantic features of the image; the ROI semantic features and the non-ROI semantic features are determined using text, and the text is associated with the image; The encoding module is configured to perform joint source-channel coding on the text semantic features, the ROI semantic features, and the non-ROI semantic features respectively to obtain text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols; The mapping module is configured to map the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols to channel subspaces of different channel qualities based on the CSI, thereby obtaining text semantic signals, ROI semantic signals, and non-ROI semantic signals; The first transmitting module is configured to transmit the text semantic signal, the ROI semantic signal, and the non-ROI semantic signal to the network device through the MIMO channel.

[0009] This application provides a wireless communication device, the device comprising: The second transmitting module is configured to transmit Channel State Information (CSI) of the Multiple Input Multiple Output (MIMO) channel to the terminal device; The second receiving module is configured to receive text semantic signals, region of interest (ROI) semantic signals, and non-ROI semantic signals sent by the terminal device via a MIMO channel. The signal detection module is configured to perform signal detection on the text semantic signal, the ROI semantic signal and the non-ROI semantic signal respectively to obtain text semantic symbols, ROI semantic symbols and non-ROI semantic symbols; The decoding module is configured to perform joint source-channel decoding on the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols respectively to obtain text semantic features, ROI semantic features, and non-ROI semantic features. The recovery module is configured to recover the text semantic features, the ROI semantic features, and the non-ROI semantic features into a reconstructed image based on the CSI.

[0010] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the wireless communication method provided in this application when executed by a processor.

[0011] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the wireless communication method provided in this application.

[0012] In this embodiment, the terminal device receives CSI sent by the network device to perceive the quality distribution of the current MIMO channel. Then, based on the correlation between text and image, it determines text semantic features, ROI semantic features, and non-ROI semantic features. The ROI and non-ROI semantic features are delineated by text guidance to ensure that key semantic regions are identified and distinguished. Next, joint source-channel coding is performed on the three types of semantic features. Subsequently, differentiated mapping is performed on the three types of semantic symbols based on the CSI. Finally, the three types of semantic signals are transmitted through the MIMO channel, improving the transmission reliability of key semantic information in complex channel environments. This achieves coordinated matching of semantic symbols and channel quality, significantly improving the robustness and reconstruction quality of cross-modal semantic communication under MIMO channels.

[0013] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this application. Attached Figure Description

[0014] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application. Obviously, the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0015] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0016] Figure 1 This application provides a schematic diagram of a wireless communication implementation process. Figure 1 ; Figure 2 This application provides a schematic diagram of a wireless communication implementation process. Figure 2 ; Figure 3 A schematic diagram illustrating the implementation process of allocating channel subspaces for different semantic features, provided in an embodiment of this application; Figure 4 A schematic diagram illustrating an implementation process for determining semantic features, provided in an embodiment of this application; Figure 5 A schematic diagram illustrating the implementation process of determining an ROI mask provided in an embodiment of this application; Figure 6 This is a schematic diagram illustrating the implementation process of mapping different semantic symbols to channel subspaces of different channel qualities, provided in an embodiment of this application. Figure 7 A schematic diagram illustrating the implementation process of restoring different semantic features into a reconstructed image, provided in an embodiment of this application; Figure 8 A schematic diagram illustrating an implementation process for obtaining a second fused feature map, provided in an embodiment of this application; Figure 9 This application provides a schematic diagram of a wireless communication implementation process. Figure 3 ; Figure 10 A schematic diagram of a semantic importance region segmentation module provided in an embodiment of this application; Figure 11 A schematic diagram of a channel state-aware cross-modal fusion module provided in an embodiment of this application; Figure 12 This application provides a schematic diagram of a wireless communication implementation process. Figure 4 ; Figure 13 A schematic diagram illustrating the functional module interaction and signaling transmission of a communication system provided in an embodiment of this application; Figure 14 A schematic diagram illustrating the signaling interaction between a user equipment and a base station, provided as an embodiment of this application; Figure 15An example of a simulation result comparison chart provided in this application embodiment. Figure 1 ; Figure 16 An example of a simulation result comparison chart provided in this application embodiment. Figure 2 ; Figure 17 This is a schematic diagram of the structure of a device provided in an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] This application provides a terminal device, a network device, and a wireless communication method.

[0019] In a first aspect, embodiments of this application provide a wireless communication method applied to a terminal device, such as... Figure 1 As shown, it includes: S101. Receive the Channel State Information (CSI) of the Multiple-Input Multiple-Output MIMO channel sent by the network device; S102, determine the semantic features of the text, the semantic features of the region of interest (ROI) of the image, and the semantic features of the non-ROI of the image; the semantic features of the ROI and the semantic features of the non-ROI are determined using text, which is associated with the image; S103. Perform joint source-channel coding on the text semantic features, the ROI semantic features and the non-ROI semantic features respectively to obtain text semantic symbols, ROI semantic symbols and non-ROI semantic symbols; S104. Based on the CSI, the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols are mapped to channel subspaces of different channel qualities to obtain text semantic signals, ROI semantic signals, and non-ROI semantic signals; S105. The text semantic signal, the ROI semantic signal, and the non-ROI semantic signal are sent to the network device through the MIMO channel.

[0020] Secondly, the wireless communication method provided in this application embodiment is applied to network devices, such as... Figure 2 As shown, it includes: S201. Send Channel State Information (CSI) of the Multiple-Input Multiple-Output MIMO channel to the terminal device; S202. Receive text semantic signals, ROI semantic signals, and non-ROI semantic signals sent by the terminal device through the MIMO channel; The text semantic signal, the ROI semantic signal, and the non-ROI semantic signal are obtained by mapping text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols to channel subspaces of different channel qualities based on the CSI; the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols are obtained by jointly source-channel coding the text semantic features, the ROI semantic features of the image, and the non-ROI semantic features of the image; the ROI semantic features and the non-ROI semantic features are determined using text, and the text is associated with the image; S203. Perform signal detection on the text semantic signal, the ROI semantic signal and the non-ROI semantic signal respectively to obtain text semantic symbols, ROI semantic symbols and non-ROI semantic symbols; S204. Perform joint source-channel decoding on the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols respectively to obtain text semantic features, ROI semantic features, and non-ROI semantic features; S205. Based on the CSI, restore the text semantic features, the ROI semantic features, and the non-ROI semantic features to the reconstructed image.

[0021] Thirdly, the wireless communication method provided in this application is applied to a wireless communication system including a terminal device and a network device, wherein the description of the terminal device can be found in [reference needed]. Figure 1 The wireless communication method and network device described herein can be found in [reference needed]. Figure 2 The wireless communication method shown.

[0022] Below, on Figure 1 or Figure 2 The wireless communication method shown is described.

[0023] In this embodiment, the terminal device can be understood as the originator or sender of text semantic signals, ROI semantic signals, and non-ROI semantic signals. The network device can be understood as the receiver or recipient of text semantic signals, ROI semantic signals, and non-ROI semantic signals. The terminal device can capture images itself or receive images sent by other peripheral devices; the terminal device can generate text based on images or receive text input from images by the user. The text is associated with the image.

[0024] Communication between terminal devices and network devices is based on MIMO channels. A MIMO channel is a wireless communication technology that utilizes multiple transmit antennas and multiple receive antennas for data transmission. Through spatial diversity and spatial multiplexing, MIMO channels can significantly improve system capacity, reliability, and anti-interference capabilities. MIMO channels can simultaneously transmit multiple independent data streams, thereby improving spectrum utilization. In the embodiments of this application, MIMO channels are used to carry text semantic signals, image ROI semantic signals, and non-ROI semantic signals.

[0025] In this embodiment, the network device can send the Channel Information System (CSI) of the MIMO channel to the terminal device, and the terminal device receives the CSI sent by the network device. The CSI can be understood as a channel matrix H. In one possible implementation, the network device can periodically send the CSI to the terminal device to ensure that the terminal device has the latest channel conditions before each data transmission. In another possible implementation, the network device can also send the updated CSI to the terminal device when the CSI is updated. In yet another possible implementation, the network device can send the CSI to the terminal device based on a request from the terminal device. In this case, the terminal device can request the CSI from the network device when it needs to send an image to the network device.

[0026] CSI refers to a set of parameters describing the channel characteristics between terminal equipment and network equipment in a wireless communication system. It includes the Channel Quality Indicator (CQI), and can also include at least one of the following: Rank Indicator (RI) or Precoding Matrix Indicator (PMI) information. CSI reflects the current quality status of the channel.

[0027] The terminal device uses the text associated with the image to determine the ROI semantic features and non-ROI semantic features of the image, and also determines the text semantic features of the text. Text semantic features refer to abstract representations extracted from text that characterize its inherent meaning or content. The ROI semantic features and non-ROI semantic features of the image are the non-overlapping parts of the image semantic features, which are abstract representations extracted from the image that characterize its inherent meaning or content. Image semantic features can represent at least one of the following information about the image: object category, scene structure, and texture information. In the embodiments of this application, text semantic features, ROI semantic features, and non-ROI semantic features of the image all belong to semantic features.

[0028] The semantic features of an image's Region of Interest (ROI) can be understood as the semantic features of the ROIs within the image. An ROI refers to the region in the image that has a high semantic similarity to the text. The semantic features of an image's Non-ROI can be understood as the semantic features of the regions outside the ROIs, which can be understood as regions in the image that have a low semantic similarity to the text. In an example, in an image containing a person and a background, and where the text describes the person, the region corresponding to the person might be an ROI, and the region corresponding to the background might be a non-ROI.

[0029] For example, in one possible implementation, the text and the image associated with the text are first encoded to obtain text semantic features and image semantic features; using the text semantic features corresponding to the text, ROI semantic features and non-ROI semantic features are determined. For example, in another possible implementation, the text is first used to determine the ROI and non-ROI of the image; the text, the ROI of the image, and the non-ROI of the image are encoded to obtain text semantic features, ROI semantic features of the image, and non-ROI semantic features of the image.

[0030] In this embodiment of the application, the terminal device performs joint source-channel coding on the determined text semantic features, ROI semantic features and non-ROI semantic features to obtain text semantic symbols, ROI semantic symbols and non-ROI semantic symbols.

[0031] In this embodiment, Joint Source-Channel Coding (JSCC) is a coding technique that combines source coding and channel coding. JSCC can better adapt to channel characteristics, reduce redundancy, and improve transmission efficiency. In this embodiment, text semantic features, image ROI semantic features, and non-ROI semantic features are respectively encoded by JSCC to generate corresponding semantic symbols: text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols.

[0032] Terminal devices can map text semantic symbols, ROI semantic symbols and non-ROI semantic symbols to channel subspaces of different channel qualities based on CSI to obtain text semantic signals, ROI semantic signals and non-ROI semantic signals.

[0033] In this embodiment of the application, in a MIMO channel, the channel matrix can be decomposed into multiple orthogonal channel subspaces using Singular Value Decomposition (SVD). In one example, for a 4×4 MIMO channel, SVD yields four channel subspaces.

[0034] In this embodiment, the terminal device maps text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols to channel subspaces of different channel qualities. The text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols are then converted into corresponding transmitted signal matrices, such as text semantic signals. ROI semantic signals Non-ROI semantic signals And combined into a total transmitted signal matrix through spatial multiplexing. .

[0035] After completing the semantic symbol mapping, the terminal device transmits the total transmitted signal matrix through the first transceiver. The signal is transmitted to network devices via a MIMO channel. This process may involve physical layer operations such as signal modulation, amplification, and filtering to ensure that the signal can propagate effectively in the wireless channel. In one example, Quadrature Amplitude Modulation (QAM) or Orthogonal Frequency-Division Multiplexing (OFDM) modulation is used to convert the digital signal into an analog signal and transmit it simultaneously through multiple transmit antennas.

[0036] In this embodiment of the application, the network device receives a composite received signal matrix through a MIMO channel. The network device decouples the signals and separates their respective semantic signals. For example, in one possible implementation, the network device performs minimum mean square error (MMSE) detection on the received signals to separate the text semantic signals, ROI semantic signals, and non-ROI semantic signals.

[0037] In this embodiment, the network device performs signal detection on text semantic signals, ROI semantic signals, and non-ROI semantic signals separately. The signal detection process includes noise suppression and channel equalization for the signals in each channel subspace, ultimately recovering the corresponding semantic symbols. In one example, for ROI semantic signals in the high-channel-quality subspace, a more refined algorithm can be used to improve recovery accuracy; while for non-ROI semantic signals and text semantic signals in the low-channel-quality subspace, the detection threshold can be appropriately relaxed to balance computational complexity and recovery quality.

[0038] In this embodiment, after obtaining various semantic symbols, the network device performs JSCC decoding on each type of semantic symbol. JSCC decoding is a reverse processing method that matches the encoding process, capable of recovering the original semantic features from the received symbols. In some embodiments, JSCC decoding includes a deep neural network structure, such as a CNN or Transformer, used to map low-dimensional semantic symbols back to high-dimensional semantic features. In one example, a text semantic symbol with a dimension of 128 may be decoded into a text semantic feature vector with a dimension of 512; similarly, ROI semantic symbols and non-ROI semantic symbols of an image will also be recovered as corresponding semantic features.

[0039] In this embodiment, the terminal device receives CSI sent by the network device to perceive the quality distribution of the current MIMO channel. Then, based on the correlation between text and image, it determines text semantic features, ROI semantic features, and non-ROI semantic features. The ROI and non-ROI semantic features are delineated by text guidance to ensure that key semantic regions are identified and distinguished. Next, joint source-channel coding is performed on the three types of semantic features. Subsequently, differentiated mapping is performed on the three types of semantic symbols based on the CSI. Finally, the three types of semantic signals are transmitted through the MIMO channel, improving the transmission reliability of key semantic information in complex channel environments. This achieves coordinated matching of semantic symbols and channel quality, significantly improving the robustness and reconstruction quality of cross-modal semantic communication under MIMO channels.

[0040] In this embodiment of the application, the text semantic signal, the ROI semantic signal, and the non-ROI semantic signal on the terminal device are different from the text semantic signal, the ROI semantic signal, and the non-ROI semantic signal on the network device. The text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols on the terminal device are different from the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols on the network device. The text semantic features, the ROI semantic features, and the non-ROI semantic features of the terminal device are different from the text semantic features, the ROI semantic features, and the non-ROI semantic features of the network device.

[0041] In this embodiment of the application, the text semantic features carried by the text semantic signals and text semantic symbols on the terminal device and the text semantic features carried by the text semantic signals and text semantic symbols on the network device belong to the same text semantic features.

[0042] In this embodiment of the application, the ROI semantic features carried by the ROI semantic signal and the ROI semantic symbol on the terminal device and the ROI semantic features carried by the ROI semantic signal and the ROI semantic symbol on the network device belong to the semantic features of the ROI of the same image.

[0043] In this embodiment of the application, the non-ROI semantic features carried by the non-ROI semantic signals and non-ROI semantic symbols on the terminal device side and the non-ROI semantic features carried by the non-ROI semantic signals and non-ROI semantic symbols on the network device side belong to the non-ROI semantic features of the same image.

[0044] In some embodiments, S104 maps the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols to channel subspaces of different channel qualities based on the CSI to obtain text semantic signals, ROI semantic signals, and non-ROI semantic signals, including: assigning channel subspaces of different channel qualities to text semantic features, ROI semantic features, and non-ROI semantic features respectively based on the CSI; and mapping the text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols to channel subspaces of different channel qualities to obtain text semantic signals, ROI semantic signals, and non-ROI semantic signals.

[0045] In this embodiment, the channel subspace is an independent transmission channel formed by singular value decomposition in a MIMO channel, and each subspace corresponds to different channel resources. In this embodiment, the channel subspace is used to carry different types of semantic features to achieve differentiated allocation of resources. In one example, in a MIMO channel, by performing singular value decomposition on the channel matrix, a set of orthogonal channel subspaces can be obtained, each channel subspace having different channel quality.

[0046] In this embodiment, for the terminal device, text semantic symbols are digital signals of text semantic features modulated for transmission at the physical layer. Text semantic symbols can be converted from semantic features into signal waveforms suitable for channel transmission using digital modulation techniques, and then spatially mapped according to the allocated channel subspace. The mapping process of text semantic symbols involves mapping high-dimensional semantic vectors to low-dimensional signal space while preserving the core semantic information of the semantic features.

[0047] In this embodiment, the ROI semantic symbol is a digital signal form of the ROI semantic features modulated and used for transmission at the physical layer. The ROI semantic symbol is generated using digital modulation technology and spatially mapped according to the allocated channel subspace. Because the ROI semantic features have high semantic importance, the ROI semantic symbol can be mapped to a high-quality channel subspace to ensure the reliability and accuracy of transmission.

[0048] In this embodiment, the non-ROI semantic symbol is a digital signal form of non-ROI semantic features modulated for transmission at the physical layer. The non-ROI semantic symbol is mapped to a low-quality channel subspace, allowing for a certain degree of distortion or error, because the semantic information corresponding to the non-ROI semantic features is relatively minor.

[0049] In this embodiment, channel subspace mapping is a process of spatially multiplexing text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols according to a pre-allocated channel subspace. In one example, a subchannel mapping matrix is ​​constructed to map text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols to their corresponding subspace locations.

[0050] In this embodiment, text semantic features, ROI semantic features, and non-ROI semantic features are converted into corresponding symbol sequences, namely text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols. Then, according to the channel subspace allocation scheme, these symbol sequences are mapped to the corresponding subspaces.

[0051] For example, in one possible implementation, assuming MIMO has four channel subspaces, where the first channel subspace is allocated to ROI semantic symbols, the second channel subspace is allocated to text semantic symbols, and the third channel subspace is allocated to non-ROI semantic symbols. ROI semantic symbols are filled into the first channel subspace, text semantic symbols into the second channel subspace, and non-ROI semantic symbols into the third channel subspace. Subsequently, a total transmit signal matrix is ​​formed through spatial multiplexing and transmitted through the MIMO channel.

[0052] For example, in one possible implementation, MIMO has two channel subspaces, where the first channel subspace is allocated to ROI semantic symbols, and the second channel subspace is allocated to text semantic symbols and non-ROI semantic symbols. ROI semantic symbols are filled into the first channel subspace, and text semantic symbols and non-ROI semantic symbols are filled into the second channel subspace. Subsequently, a total transmit signal matrix is ​​formed through spatial multiplexing and transmitted through the MIMO channel.

[0053] In this embodiment, the terminal device allocates channel subspaces of different channel qualities to text semantic features, ROI semantic features, and non-ROI semantic features based on CSI, and maps text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols to designated channel subspaces, thus achieving semantically important-driven differentiated resource allocation. This prioritizes the transmission of key semantic information on high-quality channels, thereby reducing the risk of distortion of key semantic information and significantly improving the recovery quality of target images and user experience in complex wireless channel environments.

[0054] The terminal device evaluates the channel quality of each channel subspace based on CSI (Channel Signal Indicator), and then performs differentiated mapping on text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols. The mapping process is implemented through a channel subspace allocation matrix, which binds each semantic symbol to a specific subspace index. In one example, if the CSI shows that the first subspace has the highest gain, then the ROI semantic symbols of the image are mapped to the first subspace; the second subspace, with the next highest gain, is used to carry text semantic symbols; and the third subspace is used for non-ROI semantic symbols of the image. After mapping, each type of semantic symbol is converted into a corresponding transmitted signal matrix, such as the text semantic signal. ROI semantic signals Non-ROI semantic signals And combined into a total transmitted signal matrix through spatial multiplexing. It is ready to transmit via the MIMO channel.

[0055] In some embodiments, for terminal devices, such as Figure 3 As shown, based on the CSI, channel subspaces with different channel qualities are allocated to the text semantic features, the ROI semantic features and the non-ROI semantic features, respectively, including the following steps S301 to S303.

[0056] S301. Determine semantic importance indication information. Semantic importance indication information is used to reflect the relative semantic importance between text semantic features, ROI semantic features and non-ROI semantic features.

[0057] S302. Based on the CSI, determine the gain of multiple channel subspaces of the MIMO channel.

[0058] S303. Based on the semantic importance indication information and the gain of the multiple channel subspaces, channel subspaces with different channel qualities are assigned to the text semantic features, the ROI semantic features and the non-ROI semantic features respectively.

[0059] In this application embodiment, semantic importance indication information is an indicator or data structure used to quantify and express the relative importance among different semantic features (such as text semantic features, image ROI region semantic features, and non-ROI region semantic features). Semantic importance indication information can exist in numerical form and is used to guide subsequent resource allocation and transmission strategies. In some embodiments, the semantic importance of ROI semantic features is higher than that of non-ROI semantic features and text semantic features; text semantic features can carry global context information, and their importance may be higher than that of the non-ROI semantic features.

[0060] In this embodiment, semantic importance indication information and the gains of multiple channel subspaces determined based on CSI are used to assign channel subspaces of different channel qualities to textual semantic features, ROI semantic features, and non-ROI semantic features. CSI is used to evaluate the transmission performance of each channel subspace in the MIMO channel. A channel subspace is an orthogonal space obtained by singular value decomposition (SVD) of the MIMO channel matrix. Each channel subspace corresponds to an independent transmission path, and its gain is determined by the magnitude of the corresponding singular value. A larger gain indicates better channel quality and higher transmission reliability for that channel subspace. In one example, in a 4×4 MIMO channel, if the singular value decomposition result is [σ1, σ2, σ3, σ4], and σ1>σ2>σ3>σ4, then the first channel subspace has the highest gain and is suitable for transmitting the semantic information with the highest importance. There is a direct data correlation between CSI and the gains of the channel subspaces: CSI provides the original channel matrix, which is processed by SVD to generate gain values.

[0061] In this embodiment, semantic importance indication information is matched with channel subspace gain to form a joint optimization decision framework. In one example, if the ROI semantic feature has the highest importance score and the first channel subspace gain is the largest, then the ROI semantic feature is mapped to the first channel subspace; if the non-ROI semantic feature has the next highest importance score and the second subspace gain is higher, then the non-ROI semantic feature is assigned to the second channel subspace; textual semantic features are assigned to the remaining subspaces. This allocation strategy ensures that highly important semantic features are transmitted on high-quality channels, thereby minimizing the risk of distortion.

[0062] For example, in one possible implementation, allocation can be accomplished using a sorting algorithm or a weighted matching algorithm, such as arranging semantic importance scores and channel gains in descending order and then mapping them one-to-one. Furthermore, a dynamic adjustment mechanism can be introduced to update the allocation scheme in real time when channel conditions change rapidly. This process embodies a deep integration of the semantic and physical layers, achieving semantic-priority intelligent transmission.

[0063] In this embodiment, semantic importance indication information is determined, the gain of multiple channel subspaces of the MIMO channel is determined based on CSI, and channel subspaces of different channel qualities are assigned to different semantic features by combining the semantic importance indication information and the channel subspace gain. This enables synergistic optimization of semantic importance indication information and channel subspace gain, thereby improving the transmission robustness of key semantic information and significantly enhancing the overall performance of cross-modal semantic communication and image restoration quality in complex MIMO channel environments.

[0064] In some embodiments, for terminal devices, such as Figure 4 As shown, S102 determines the semantic features of the text, the semantic features of the ROI of the image, and the semantic features of the non-ROI of the image, including the following steps S401 to S403.

[0065] S401. Semantic encoding is performed on the text and the image respectively to obtain text semantic features and image semantic features.

[0066] S402. Based on the first similarity between the text semantic features and the image semantic features, determine the ROI mask, which includes the mask value corresponding to each semantic feature in the image semantic features.

[0067] S403. Based on the ROI mask, the image semantic features are divided into ROI semantic features and non-ROI semantic features.

[0068] In this embodiment, text semantic features are high-dimensional vector representations extracted from input text data through deep semantic encoding, used to capture the semantic information of the text content. Text semantic features can be generated by pre-trained text semantic encoding models (such as Bidirectional Encoder Representations from Transformers (BERT), Transformer, etc.), and can reflect the vocabulary, syntactic structure, and contextual semantic relationships in the text. A text semantic encoding model can also be called a text semantic encoder. In one example, in the text "A cat is sleeping on the sofa," the text semantic encoding model will semantically encode the text "A cat is sleeping on the sofa" to obtain a comprehensive semantic representation.

[0069] In this embodiment, image semantic features are high-dimensional vector representations extracted from input image data through deep visual encoding, used to describe the semantic information of the image content. Image semantic features can be generated by a Convolutional Neural Network (CNN) or a Vision Transformer (ViT), capable of recognizing objects, scenes, actions, and spatial layouts in an image. Deep visual encoding can also be called an image semantic encoder or an image semantic encoding model. In one example, in an image containing a cat sleeping on a sofa, the image semantic encoder encodes the image to obtain its image semantic features.

[0070] In this application embodiment, there is a cross-modal association between text semantic features and image semantic features. In some embodiments, text semantic features and image semantic features are aligned through a shared semantic space, so that the same semantic content has similar vector representations in different modalities.

[0071] For example, in one possible implementation, a pre-trained text semantic encoder and an image semantic encoder are first loaded. For text and images associated with the text acquired by the terminal device, the text encoder converts the text into a fixed-dimensional semantic feature vector, and the image encoder maps the image into a semantic feature map.

[0072] In this embodiment, the first similarity refers to the degree of similarity between text semantic features and image semantic features, used to evaluate the matching degree between text semantic features and image semantic features in the semantic space. The first similarity can be obtained by methods such as cosine similarity, dot product, Euclidean distance, or a neural network model used to determine the similarity between text semantic features and image semantic features. In one example, if the text semantic feature vector is A and the image semantic feature vector is B, then the first similarity can be represented as cos(A, B). The higher the first similarity, the more semantically matched the text and the image. In another example, if the text semantic feature vector is A and the image semantic feature vector is B, inputting the text semantic feature vector A and the image semantic feature vector B into a neural network model used to determine the similarity between text semantic features and image semantic features yields the first semantic similarity.

[0073] In this embodiment, the terminal device determines the ROI mask based on a first similarity. The ROI mask is a two-dimensional or three-dimensional continuous-value mask used to identify the probability distribution of semantic importance at various locations in an image. Each value in the ROI mask corresponds to the importance weight of a certain location in the image's semantic feature map; a larger value indicates that the location is more likely to be a semantically critical region (i.e., ROI).

[0074] In this embodiment, ROI semantic features are a subset of semantic features extracted from image semantic features, corresponding to high-value regions in the ROI mask. ROI semantic features can collectively include regions in the image that have a high degree of similarity to text semantic features. In one example, in an image of a cat sleeping on a sofa, the text is "Cat sleeping on a sofa," and the ROI semantic features mainly contain semantic information about the cat and its posture.

[0075] In this embodiment, non-ROI semantic features are a subset of semantic features extracted from image semantic features, corresponding to low-value regions in the ROI mask. Non-ROI semantic features may collectively contain secondary or background information in the image, such as environmental details, irrelevant objects, or blurred areas. In one example, in an image of a cat sleeping on a sofa, with the text "Cat sleeping on a sofa," non-ROI semantic features might include sofa texture, background walls, etc.

[0076] In this embodiment, ROI semantic features and non-ROI semantic features are complementary. ROI semantic features and non-ROI semantic features together constitute the complete image semantic features, but their importance differs.

[0077] For example, in one possible implementation, the ROI semantic features are obtained by multiplying the ROI mask by the image semantic features element-wise; the non-ROI semantic features are obtained by subtracting the ROI mask from 1 and then multiplying it by the image semantic features. In one example, let the original image semantic feature map be F and the ROI mask be M, then the ROI semantic feature map = F × M, and the non-ROI semantic feature map = F × (1 - M).

[0078] For example, in another possible implementation, the image semantic features corresponding to the mask values ​​with a value of 1 in the ROI mask are determined as ROI semantic features, and the image semantic features corresponding to the mask values ​​with a value of 0 in the ROI mask are determined as non-ROI semantic features.

[0079] In some embodiments, for terminal devices, such as Figure 5 As shown, S302 determines the ROI mask based on the similarity between the text semantic features and the image semantic features, including the following steps S501 to S503.

[0080] S501. Determine the first attention weight based on the first similarity between the text semantic features and the image semantic features.

[0081] S502. Using the first attention weight, the text semantic features and the image semantic features are fused to obtain a first fused feature map.

[0082] S503. Convert the first fused feature map into a ROI mask.

[0083] In this embodiment, the first attention weight is a weighted coefficient calculated based on the first similarity between the text semantic features and the image semantic features, used to measure the importance of different semantic features in the fusion process. In some embodiments, the first attention weight is determined through a cross-attention mechanism, using text semantic features as key vectors and image semantic features as query and value vectors. Attention scores are generated through dot product or scaled dot product operations, and then normalized by the Softmax function to form a weight distribution. In one example, when the text is a running dog and the image content is a running dog and other information, the semantic features of "dog" and "running" are given higher first attention weights, while the semantic features of the background region are ignored. In other embodiments, the calculation of the first attention weight can employ a multi-head attention mechanism to capture semantic associations in different subspaces and improve feature alignment accuracy. Multi-head attention mechanisms include, but are not limited to, self-attention mechanisms based on the Transformer architecture, convolutional attention mechanisms, or learnable parameterized attention networks.

[0084] For example, in one possible implementation, when the text is "birds flying in the sky" and the image content is "birds flying in the sky" and other information, the text is first encoded into text semantic features, and the image is encoded into image semantic features. Then, an attention mechanism is used to calculate the similarity score between each pair of text semantic features and image semantic features, and a first attention weight matrix is ​​generated accordingly. The first attention weight matrix can reflect which features in the image semantic features are highly correlated with the input text description, such as the bird's position and flight posture.

[0085] In this embodiment, the terminal device obtains a first fused feature map based on a first attention weight. The first fused feature map is a joint semantic representation generated by weighted fusion of textual semantic features and image semantic features. The spatial structure of this first fused feature map is inherited from the image feature map, while the semantic information is enhanced by the text. The process of fusing textual semantic features and image semantic features through the first attention weight can be understood as a weighted summation operation based on an attention mechanism. That is, for each position in the image feature map, the feature values ​​in the image semantic features are linearly combined according to the corresponding first attention weight, thereby generating a first fused feature map rich in cross-modal contextual information. In one example, if the image shows a tiger in a forest and the text describes it as a tiger, the tiger feature in the image semantic features has a larger corresponding first attention weight, thus strengthening the recognition ability of the tiger's location.

[0086] In this embodiment, the terminal device obtains a Region of Interest (ROI) mask based on the first fused feature map. The ROI mask is a continuous probability map reflecting the semantic importance of each spatial location in the image. The ROI mask ranges from [0, 1], with higher values ​​indicating that the spatial location in the image's semantic features is more likely to be a semantically critical region. In some embodiments, the ROI mask is generated as follows: the spatial resolution of the first fused feature map is gradually restored through a deconvolutional upsampling network to match the original image size; subsequently, the generated feature map is mapped to a probability interval using a sigmoid function.

[0087] In this embodiment, a first attention weight is determined based on the first similarity between text semantic features and image semantic features. Then, the text semantic features and image semantic features are fused using the first attention weight to obtain a first fused feature map, which is finally converted into a ROI mask. This enables a semantically driven attention mechanism, accurately locating key semantic regions in the image. This effectively improves the accuracy of cross-modal feature alignment, supports subsequent differentiated channel resource allocation strategies, and enhances the robustness of image semantic recovery in complex wireless channel environments.

[0088] In other embodiments, S102 determines text semantic features, region of interest (ROI) semantic features of the image, and non-ROI semantic features of the image; the ROI semantic features and the non-ROI semantic features are determined using text, which is associated with the image, including: the terminal device dividing the image into ROI regions and non-ROI regions based on a second similarity between the text and the image; and performing semantic encoding on the text, the ROI regions, and the non-ROI regions respectively to obtain the text semantic features, the ROI semantic features, and the non-ROI semantic features.

[0089] In this embodiment, the second similarity refers to the degree of semantic matching or relevance between text and image. In some embodiments, the second similarity is calculated through a cross-modal attention mechanism and used to evaluate the association strength between the text description and the semantic information of various parts of the image. In this embodiment, the second similarity serves as the basis for dividing the region of interest (ROI) and non-ROI regions of an image; a higher second similarity value indicates a stronger semantic relevance between the corresponding image region and the text. In one example, under the text description of a cat sitting on a sofa, the second similarity of the area where the cat is located in the image will be significantly higher than that of the background sofa area, thus it will be classified as a region of interest (ROI) of the image.

[0090] In some embodiments, the second similarity can be obtained by inputting text and an image associated with the text into a pre-trained neural network. A region of interest (ROI) probability mask of the input image is generated based on the second similarity distribution, and then binarized by setting a threshold to form the ROI mask of the image. This ROI mask is applied to the image to obtain the ROI regions and non-ROI regions of the image.

[0091] In this embodiment, the division of the ROI (Region of Interest) and non-ROI (Region of Interest) regions of an image depends on the calculation result of the second similarity. The ROI and non-ROI regions together constitute the complete semantic structure of the image. The ROI region refers to the area in the image that is highly related to the semantics of the text and has high semantic importance. The non-ROI region refers to the part of the image that is less related to the semantics of the text and has lower semantic importance.

[0092] For example, in one possible implementation, when the text description is a person wearing a red coat standing in a park, the area in the image corresponding to the person and the red coat is identified as the region with a high degree of matching with the person and the red coat in the text description, and is divided into the ROI region of the image; while the park background, trees, and other areas with a low degree of matching with the person and the red coat in the text description are divided into the non-ROI region of the image.

[0093] In this embodiment, the semantic features of an image's ROI refer to the semantic feature vectors extracted from the ROIs of the image. These semantic feature vectors are obtained by independently encoding the ROIs of the image using an image semantic encoder, preserving high-level semantic information of key regions of the image. Since the ROIs of an image are highly correlated with text, the semantic features of the ROIs have high information density and representativeness.

[0094] In this embodiment, the non-ROI semantic features of an image refer to the semantic feature vectors extracted from the non-ROI regions of the image. These non-ROI semantic features are generated by an image semantic encoder, but the semantic importance of non-ROI regions is relatively low. Although the information in non-ROI regions is relatively minor, it still plays a supplementary role in complete image reconstruction, such as providing scene context or environmental details.

[0095] In some embodiments, for terminal devices, such as Figure 6 As shown, S104 maps the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols to channel subspaces with different channel qualities to obtain text semantic signals, ROI semantic signals, and non-ROI semantic signals, including the following steps S601 to S603.

[0096] S601. Generate corresponding association identifiers for the text semantic symbols, the ROI semantic symbols and the non-ROI semantic symbols respectively. The association identifiers are used to characterize the association relationship between the text semantic features, the ROI semantic features and the non-ROI semantic features.

[0097] S602. Based on text semantic symbols, ROI semantic symbols and non-ROI semantic symbols and their corresponding association identifiers, determine text data packets, ROI data packets and non-ROI data packets.

[0098] S603. Map the text data packet, the ROI data packet, and the non-ROI data packet to channel subspaces of different channel qualities to obtain text semantic signals, ROI semantic signals, and non-ROI semantic signals.

[0099] In this embodiment, the association identifier refers to a unique or distinguishable tag assigned to text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols during semantic transmission to establish the correspondence among them. The association identifier can be in the form of a numerical code, hash value, sequence number, or structured tag. In some embodiments, the association identifier is attached to the original semantic symbol through metadata or embedded fields to form an identified semantic unit. This enables network devices to quickly match the corresponding semantic features based on the association identifier after decoding, avoiding reconstruction errors caused by semantic confusion.

[0100] In this embodiment, text data packets, ROI data packets, and non-ROI data packets are independent transmission units constructed based on semantic symbols and their associated identifiers. Text data packets contain text semantic symbols and their associated identifiers, and can be encapsulated in a structured format (such as JavaScript object notation or binary protocol) to carry plain text semantic information; ROI data packets contain ROI semantic symbols and their associated identifiers; and non-ROI data packets contain non-ROI semantic symbols and their associated identifiers.

[0101] In this embodiment of the application, mapping text data packets, ROI data packets and non-ROI data packets to different channel subspaces can be understood as follows: since ROI data packets contain important semantic information of the image, ROI data packets are mapped to high-gain channel subspaces; while text data packets and non-ROI data packets are mapped to low-gain channel subspaces to save high-quality channel resources.

[0102] In the embodiments of this application, a channel subspace refers to multiple orthogonal channel subspaces obtained through singular value decomposition (SVD) in a MIMO channel, each with different channel gain and transmission quality. Channel subspaces with high channel gain have higher signal-to-noise ratio and lower bit error rate, making them suitable as high-quality channel resources; while subspaces with low channel gain are suitable for carrying non-critical or redundant information.

[0103] In this embodiment, association identifiers are generated for text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols respectively. Independent data packets are then constructed based on these association identifiers, and different data packets are mapped to channel subspaces with different channel qualities. This enables precise association between text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols, thereby improving the accuracy and reliability of cross-modal semantic transmission. Consequently, it enhances the visual consistency and semantic fidelity of image restoration under MIMO fading channel conditions.

[0104] In some embodiments, for a terminal device, the wireless communication processing method further includes: transmitting a pilot signal on the MIMO channel, the pilot signal being used by the network device to determine the CSI.

[0105] In this embodiment, the pilot signal is a known reference signal actively transmitted by a terminal device (such as a user equipment) to a network device (such as a base station) in a wireless communication system. The pilot signal has a predefined structure and content, does not carry user data, and its main function is to help the network device estimate the current channel state, thereby obtaining an accurate CSI. In multiple-input multiple-output (MIMO) channels, due to the presence of multiple transmit and receive antennas and a high channel matrix dimension, it is necessary to use the pilot signal to independently estimate each subchannel. The pilot signal can be transmitted periodically or semi-persistently in specific time-frequency resource blocks, for example, occupying certain symbol positions in the time domain and specific subcarriers in the frequency domain.

[0106] In this embodiment, there is a direct physical correlation between the pilot signal and the MIMO channel. The MIMO channel is a complex matrix composed of propagation paths from multiple transmit antennas to multiple receive antennas. The elements of this complex matrix reflect the gain, phase, and delay characteristics of each path. After the pilot signal is transmitted through these paths, it is detected at the network device. The channel response matrix, i.e., the channel state information (CSI), can be recovered using algorithms such as least squares (LS) or minimum mean square error (MMSE). This mechanism ensures that the network device can monitor the dynamic changes of the channel in real time, providing a basis for subsequent adaptive modulation, beamforming, and resource scheduling.

[0107] For example, in one possible implementation, in a 5G or 6G mobile communication system, after a user equipment (UE) enters the coverage area of ​​a base station and establishes a connection, it sends pilot signals to the base station according to the pilot pattern specified in the protocol. For instance, in a Time Division Duplex (TDD) system, the pilot signal may be sent in a specific time slot of the uplink; in a Frequency Division Duplex (FDD) system, it is sent in a dedicated uplink frequency band. After receiving the pilot signal, the base station generates Channel State Information (CSI) using a channel estimation algorithm and feeds back the generated CSI to the UE or uses it for downlink precoding processing.

[0108] In the embodiments of this application, the pilot signal can take various specific forms, such as: a pilot sequence based on Discrete Fourier Transform (DFT), a randomly generated pseudo-random noise sequence, or a Zadoff-Chu sequence. Different types of pilot sequences are suitable for different scenarios: DFT sequences have good frequency domain clustering and are suitable for low-speed movement scenarios; while Zadoff-Chu sequences have excellent cyclic correlation and are more suitable for high-speed movement or high-interference environments. In addition, the pilot signal can be designed as an open-loop or closed-loop mode according to system requirements. In the open-loop mode, the pilot signal is fixed, while in the closed-loop mode, it can be dynamically optimized based on historical CSI.

[0109] In this embodiment of the application, the terminal device transmits pilot signals on the MIMO channel, enabling the network device to accurately estimate the channel state and thus obtain the CSI of the MIMO channel.

[0110] In this embodiment, a MIMO channel is a technique that utilizes multiple transmit antennas and multiple receive antennas to improve the capacity and reliability of a wireless communication system. In a MIMO channel, signals are transmitted simultaneously through multiple spatial channels, each corresponding to a transmit-receive antenna pair. The mathematical model of this MIMO channel can be represented as an M×N complex matrix H, where M represents the number of receive antennas, N represents the number of transmit antennas, and the matrix elements h... ij This represents the channel gain between the i-th receiving antenna and the j-th transmitting antenna.

[0111] In this application embodiment, the Channel Identity (CSI) of the MIMO channel can be divided into different types, such as narrowband CSI, wideband CSI, long-term CSI, and instantaneous CSI. Narrowband CSI is suitable for low-frequency or small-bandwidth systems, while wideband CSI is suitable for high-frequency or large-bandwidth systems. Long-term CSI reflects the average characteristics of the channel and is often used for resource allocation; instantaneous CSI reflects the real-time changes of the channel and is used for beamforming and modulation adjustment. Different types of CSI are suitable for different levels of communication optimization.

[0112] In some embodiments, for a terminal device, the wireless communication method further includes: receiving a reconstructed image sent by the network device through the MIMO channel, wherein the reconstructed image is obtained by recovering the text semantic signal, the ROI semantic signal, and the non-ROI semantic signal based on the CSI of the MIMO channel.

[0113] In this embodiment, the reconstructed image refers to the final output image generated by the network device after signal detection, JSCC decoding, fusion, and image restoration network processing of the transmitted text semantic signals, ROI semantic signals, and non-ROI semantic signals. The reconstructed image not only restores the visual content of the original image but also retains key information associated with the text semantics. The quality of the reconstructed image directly reflects the performance of the cross-modal semantic communication system. Evaluation metrics for the reconstructed image may include Peak Signal-to-Noise Ratio (PSNR) and / or Structural Similarity (SSIM), used to measure the fidelity and structural consistency of the image restoration.

[0114] For example, in one possible implementation, when an abnormal event occurs in a certain area, the user equipment, based on SCI, performs JSCC encoding and differential channel mapping on the text semantic features (such as the location of the intruder) related to the abnormal event, the ROI semantic features (such as the ROI semantic features of the intruder's location) and non-ROI semantic features of the text-related image (such as the ROI semantic features of the intruder's location), and sends them to the base station to obtain a clear and semantically accurate reconstructed image sent by the base station, helping the user to quickly identify risks.

[0115] In some embodiments, for network devices, such as Figure 7 As shown, S205 restores the text semantic features, the ROI semantic features and the non-ROI semantic features into a reconstructed image based on the CSI, including the following steps S701 to S703.

[0116] S701. The semantic features of ROI and non-ROI are fused to obtain the semantic features of the image.

[0117] S702. Based on the CSI, the text semantic features and the image semantic features are fused to obtain a second fused feature map.

[0118] S703. Based on the reconstruction of the second fused feature map, the reconstructed image is obtained.

[0119] In this embodiment, fusing ROI semantic features and non-ROI semantic features aims to obtain a unified and complete image semantic representation to support subsequent cross-modal fusion and image reconstruction. In some embodiments, the fusion method can employ various approaches such as weighted summation, stitching, gating mechanisms, or attention mechanisms. In one example, the ROI semantic features correspond to the semantic features of the cat's face, while the non-ROI semantic features correspond to the semantic features of the background grass; the fusion results in a complete semantic representation of the cat on the grass.

[0120] In this embodiment, textual semantic features and image semantic features are fused based on CSI to obtain a second fused feature map. This enables channel quality-adaptive semantic fusion, thereby improving the robustness of the second fused feature map in complex channel environments, and ultimately enhancing the accuracy and stability of the final reconstructed image.

[0121] In this embodiment, the second fused feature map refers to a multimodal joint semantic feature map. The second fused feature map incorporates a deep fusion of textual and image semantic features and is optimized for channel awareness based on CSI. In some embodiments, the reconstruction process of the second fused feature map relies on the collaborative work of the semantic decoder and the image reconstruction network to ensure that semantic information is accurately converted into visual content. In one example, guided by the semantic concept of a cat on grass, a second fused feature map containing elements such as cat and grass is reconstructed. Even if some image semantic features are damaged during transmission, they can be reasonably supplemented with the assistance of textual semantic features. This enables high-fidelity conversion from abstract semantics to concrete images, thereby improving the visual consistency of the second fused feature map under low-quality channels. Furthermore, this fusion process enhances the overall performance of the cross-modal semantic communication system.

[0122] Furthermore, in some embodiments, for network devices, S702 fuses the text semantic features and the image semantic features based on the CSI to obtain a second fused feature map, including: performing feature mapping on the CSI to determine a CSI modulation mask; and fusing the text semantic features and the image semantic features based on the CSI modulation mask to obtain a second fused feature map.

[0123] In this embodiment, feature mapping refers to the process of converting CSI data into a feature representation with specific semantics or structure through a masking network (neural network or other mapping function). The feature mapping process can employ deep learning models such as convolutional neural networks or fully connected layers to transform high-dimensional or unstructured CSI data into feature vectors. In one example, the CSI matrix is ​​input into a lightweight CNN, and after several convolutional and activation layers, a fixed-dimensional feature vector is output. This output feature vector reflects the overall quality of the current channel. The purpose of feature mapping is to extract key information from the CSI and eliminate redundancy.

[0124] In this embodiment, the CSI modulation mask is a matrix generated based on the CSI feature mapping result, used to dynamically adjust the contribution of different semantic features. The CSI modulation mask can be normalized to the [0, 1] interval using the Sigmoid function; a larger value indicates that the semantic feature at the corresponding position is more important or more reliable. When the channel quality is poor, the CSI modulation mask reduces the weight of low-quality regions or low-priority features; when the channel quality is good, it allows more features to participate in the fusion. In one example, in a MIMO channel, if the signal-to-noise ratio decreases due to multipath fading in some channel subspaces, the CSI modulation mask will correspondingly suppress the influence of semantic features transmitted with these channel subspaces on the second fused feature map, thereby improving the robustness of the second fused feature map.

[0125] In this embodiment, fusion refers to the process of integrating semantic features from different modalities (text and image) into a unified representation through a specific mechanism. The second fused feature map refers to the multimodal semantic representation generated after fusion, and it serves as the input for subsequent semantic decoding and image reconstruction. The second fused feature map not only contains joint semantic information of textual and image semantic features, but also embeds channel quality information of the MIMO channel, enabling it to maintain high semantic consistency even under complex channel conditions.

[0126] In some embodiments, for network devices, such as Figure 8 As shown, based on the CSI modulation mask, the text semantic features and the image semantic features are fused to obtain a second fused feature map, including the following steps S801 to S804.

[0127] S801. Map the image semantic features to key vectors and value vectors, and map the text semantic features to query vectors.

[0128] S802. Determine the third similarity between the key vector and the query vector.

[0129] S803. Based on the third similarity and CSI modulation mask, determine the second attention weight of the value vector.

[0130] S804. Based on the second attention weight and the value vector, a second fused feature map is obtained.

[0131] In this embodiment, input features are mapped to different role vectors through linear transformation. Image semantic features are mapped to key vectors and value vectors, providing reference and content for semantic information; text semantic features are mapped to query vectors, used to initiate queries or searches for image semantic information. This mapping process can be accomplished using a learnable weight matrix. In one example, image semantic features are processed through a linear layer to generate key and value vectors respectively, while text semantic features are processed through a linear layer to generate query vectors. The mapped vectors have consistent dimensions, facilitating subsequent similarity calculations.

[0132] In this embodiment, third similarity refers to the degree of matching between the query vector and the key vector, calculated using a dot product or other similarity functions in a cross-modal attention mechanism. Third similarity reflects the correlation strength between text semantic features and image semantic features. In one example, third similarity can be obtained by calculating the dot product of the query vector and the key vector and dividing it by the square root of the key vector dimension (scaling the dot product attention), or by using other functions such as cosine similarity. The third similarity result forms an attention score matrix, the size of which can be [text sequence length, number of image feature map spatial locations]. In one example, when processing the text "a cat sitting on a sofa" and its corresponding image, the query vector corresponding to the cat may have a high third similarity with the key vector corresponding to the area where the cat is located in the image, thus obtaining a high attention score.

[0133] The third similarity matrix is ​​multiplied element-wise with the CSI modulation mask, and the result is processed by the Softmax function to obtain the second attention weights. The specific formula is: Output = The Softmax function ensures that the sum of the second attention weights is 1, giving the output a probability distribution characteristic. Indicates the CSI modulation mask; Represents the query vector. This represents the key vector.

[0134] In this embodiment, the second attention weight is multiplied and accumulated with the corresponding value vector using a weighted summation method to generate the second fused feature map. The second fused feature map integrates the guiding role of textual semantic features on image semantic features, while also being influenced by the CSI modulation mask, thus preserving key semantic information even under channel-limited conditions. In one example, if some image semantic feature regions experience signal distortion due to channel fading during transmission, this mechanism can reduce their negative impact on the final fusion result by decreasing their second attention weight, thereby improving the overall semantic recovery quality.

[0135] In this embodiment, by mapping image semantic features to key vectors and value vectors, and text semantic features to query vectors, a third similarity is calculated between the key vectors and value vectors mapped from image semantic features and the query vectors mapped from text semantic features. Then, a second attention weight is dynamically adjusted using a CSI modulation mask, and finally, a weighted fusion is performed to generate a second fused feature map. This achieves precise alignment and interaction between text semantic features and image semantic features, allowing for adaptive adjustment of the contribution of different semantic regions based on channel quality. This, in turn, improves the robustness and image reconstruction performance of cross-modal semantic communication systems in complex wireless channel environments.

[0136] In some embodiments, for a network device, the wireless communication processing method further includes: performing signal detection on the text semantic signal, the ROI semantic signal, and the non-ROI semantic signal respectively to obtain the association identifiers of the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols; wherein, if the association identifiers of the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols match, the text semantic features, the ROI semantic features, and the non-ROI semantic features are restored to a reconstructed image based on the CSI.

[0137] In this embodiment, at the network device end, signal detection operations are performed on the received text semantic signals, ROI semantic signals, and non-ROI semantic signals. The signal detection process is the process of restoring the received analog or digital signals to their original transmitted symbols. The purpose of signal detection is to recover the semantic symbols before encoding, such as text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols. In one possible implementation, the signal detection process employs detection algorithms such as minimum mean square error, combined with the estimated CSI of the MIMO channel, to improve the accuracy of semantic symbol recovery.

[0138] In this embodiment, the association identifier for text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols refers to a mark or label used to characterize the correspondence between text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols. The association identifier is used to indicate whether text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols belong to the same semantic context or data frame.

[0139] In this embodiment, after signal detection and obtaining text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols at the network device end, the association identifiers of the text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols are checked for matching. When the association identifiers of the text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols match, they are considered to belong to the same semantic context. This helps ensure that the text semantic features, ROI semantic features, and non-ROI semantic features used are consistent at the semantic level, avoiding the use of irrelevant or incorrectly aligned features for image restoration.

[0140] In one example, during a complete cross-modal communication process, the textual semantic symbol corresponding to a textual description of a cat sitting on grass should be associated with the ROI semantic symbol of the cat in the image and the non-ROI semantic symbol of the grass in order to perform joint semantic fusion. The association identifier can be a timestamp, sequence number, hash value, or specific synchronization code, etc., to ensure the temporal and spatial alignment of the textual semantic symbol, ROI semantic symbol, and non-ROI semantic symbol.

[0141] It is understood that, in this embodiment of the application, by introducing association identifiers, precise alignment and management of textual semantic symbols, ROI semantic symbols, and non-ROI semantic symbols are achieved. This ensures the correctness of subsequent cross-modal fusion, thereby improving the robustness and quality of image restoration in the entire semantic communication system.

[0142] In this embodiment, under the premise of matching the association identifiers of text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols, an attention modulation mask related to channel quality is constructed using the CSI information of the MIMO channel. This attention modulation mask dynamically adjusts the weights of text semantic features, ROI semantic features, and non-ROI semantic features in the fusion process based on the channel subspace gain reflected by the CSI of the MIMO channel.

[0143] In this embodiment, image restoration is performed when the associated identifiers of text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols are matched, ensuring semantic consistency. Simultaneously, by dynamically adjusting the attention weights using the CSI of the MIMO channel, intelligent fusion with channel awareness is achieved. This effectively suppresses the impact of channel impairments on image restoration, thereby improving the accuracy and robustness of reconstructed images, and ultimately enabling high-quality cross-modal semantic communication.

[0144] In some embodiments, for a network device, the wireless communication processing method further includes: receiving a pilot signal transmitted by the terminal device through the MIMO channel; and determining the CSI of the MIMO channel based on the pilot signal.

[0145] In this embodiment, the pilot signal transmitted by the terminal device is a known reference signal that is periodically or on demand transmitted by the terminal device, used by the network device for channel estimation. The pilot signal transmitted by the terminal device has a fixed structure and known content, which facilitates accurate extraction of the channel response by the network device.

[0146] In this embodiment, the Channel Response Indicator (CSI) of the MIMO channel is determined based on the pilot signal. In a MIMO channel, the CSI can be represented in the form of a channel matrix H, where each element represents a complex channel response between a pair of transmit and receive antennas. In some embodiments, the network device estimates the channel matrix H using algorithms such as least squares or least mean square error by receiving the pilot signal transmitted by the terminal device; secondly, the estimated CSI is quantized for efficient transmission within a limited bandwidth; finally, the CSI is fed back to the terminal device or directly used for image restoration at the network device end. The accuracy of the CSI directly affects the effectiveness of image restoration.

[0147] In some embodiments, for network devices, the wireless communication method further includes: sending the reconstructed image to the terminal device.

[0148] In the embodiments of this application, such as Figure 9 As shown, it includes: S901. The network device sends the Channel State Information (CSI) of the Multiple-Input Multiple-Output MIMO channel to the terminal device. S902. The terminal device receives the Channel State Information (CSI) of the Multiple-Input Multiple-Output MIMO channel sent by the network device. S903, the terminal device determines the semantic features of the text, the semantic features of the region of interest (ROI) of the image, and the semantic features of the non-ROI of the image; the semantic features of the ROI and the semantic features of the non-ROI are determined using text, and the text is associated with the image; S904. The terminal device performs joint source-channel coding on the text semantic features, the ROI semantic features, and the non-ROI semantic features to obtain text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols. S905. Based on the CSI, the terminal device maps the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols to channel subspaces of different channel qualities to obtain text semantic signals, ROI semantic signals, and non-ROI semantic signals. S906. The terminal device transmits the text semantic signal, the ROI semantic signal, and the non-ROI semantic signal to the network device via the first transceiver and the MIMO channel. S907. Network devices receive text semantic signals, ROI semantic signals and non-ROI semantic signals sent by terminal devices through MIMO channels. S908. The network device performs signal detection on the text semantic signal, the ROI semantic signal, and the non-ROI semantic signal respectively to obtain text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols. S909. The network device performs joint source-channel decoding on the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols respectively to obtain text semantic features, ROI semantic features, and non-ROI semantic features. S910, The network device, based on the CSI, restores the text semantic features, the ROI semantic features, and the non-ROI semantic features into a reconstructed image; The following describes an exemplary application of the embodiments of this application in a real-world application scenario.

[0149] This application specifically relates to a channel state-aware cross-modal semantic communication method (i.e., an example of a wireless communication method), which is suitable for the efficient transmission and semantic recovery of multimodal information such as text and images.

[0150] Cross-modal semantic communication, as a cutting-edge research direction in the field of communication, has demonstrated enormous potential in various application scenarios in recent years, especially in the cross-modal semantic transmission of images and text. With the rapid development of artificial intelligence and deep learning technologies, the semantic relationships between images and text have been deeply explored. Cross-modal transmission between them can not only improve the intelligence level of communication systems but also provide technical support for applications such as automatic image annotation, image question answering, and image description generation. In these tasks, images and text are not only carriers of information but also carry rich semantic information. Achieving cross-modal semantic communication between images and text can provide intelligent systems with a more efficient and accurate way to exchange information.

[0151] Although semantic communication has made significant progress in various single-mode transmissions, current research largely focuses on idealized channel models, particularly single-input single-output Gaussian channels or Rayleigh fading channels. While these channel models hold important positions in theoretical research, their limitations in practical wireless communication environments are becoming increasingly apparent. Many real-world communication scenarios, especially large-scale wireless networks, commonly utilize MIMO channels. MIMO channels exhibit complex spatial propagation characteristics, with even more complex fading properties and significant multipath effects, greatly impacting the accurate transmission of semantic information.

[0152] One solution to MIMO fading is to utilize feedback MIMO channel state information. CSI can provide real-time feedback on channel quality, enabling the transmitter to dynamically adjust resource allocation based on channel conditions, thereby improving the system's anti-interference capability and robustness. However, existing CSI utilization mainly focuses on physical layer adaptive modulation and power allocation, failing to adequately consider how to integrate CSI with semantic layer information.

[0153] In view of this, this application proposes a channel state-aware cross-modal semantic communication method, also known as a wireless communication method. In the context of MIMO channels, this method combines a CSI (Channel State Indicator) dynamic attention adjustment mechanism to achieve more accurate semantic alignment and information recovery. Specifically, in a wireless communication system including terminal devices and network devices, by mapping CSI to a dynamic attention mask, the feature fusion process of text and image is optimized. Furthermore, the resource allocation of ROI (Real Area of ​​Interest) semantic features and non-ROI semantic features in the image is adaptively adjusted according to channel quality, thereby effectively mitigating the impact of low-quality channels on image semantic recovery. Simulation verification shows that this method not only improves the visual consistency of image recovery but also enhances the robustness of cross-modal semantic communication under unstable channel conditions.

[0154] In this application embodiment, a channel state-aware cross-modal semantic communication method is provided. In MIMO communication scenarios, this method introduces Channel State Indicator (CSI) to guide the cross-modal feature fusion process, thereby achieving channel-aware semantic transmission. In the image semantic representation stage, image features are further divided into ROI semantic features and non-ROI semantic features based on semantic importance. Considering the differences in semantic importance in different regions, differentiated resource allocation strategies are adopted for the two types of features during subsequent channel transmission. Based on this, this application further designs a CSI-based modal fusion mechanism. By introducing CSI information into the cross-modal attention module, a channel quality-related attention mask is constructed to dynamically adjust the weights of different semantic features. When the channel quality is poor, this mask mechanism suppresses features that contribute less to semantics and prioritizes the retention of important semantic information, thereby improving the robustness of semantic features during channel transmission. At the receiving end (i.e., the network device end), semantic symbols are obtained through a semantic decoding module, and cross-modal attention fusion is performed on these symbols to obtain the fused semantic representation; then, an image reconstruction network is used to reconstruct the image (i.e., the fused semantic representation). Through this overall design, images can still achieve high-quality reconstruction results under complex channel conditions, which is beneficial for future research on cross-modal semantic communication under MIMO channel conditions.

[0155] It is understood that, in this embodiment, the advantage of the channel state-aware dynamic attention mechanism lies in the following: First, by introducing CSI information, the communication system can perceive changes in the current channel quality in real time and dynamically adjust resource allocation strategies accordingly. The CSI-based dynamic attention mechanism can prioritize the protection of important semantic regions during channel fading, preventing the loss of important information due to signal distortion. Second, the communication system guides the image restoration process at the receiving end (i.e., the network device end) using text-assisted image restoration, thereby effectively reducing semantic errors during image restoration and improving the accuracy of image semantic reconstruction. Finally, the dynamic attention mechanism not only improves restoration performance but also effectively reduces computational complexity because the system (i.e., the network device end) can selectively activate processing for different regions based on the actual channel quality, thus avoiding unnecessary computational overhead.

[0156] In some embodiments, a method for cross-modal semantic communication between text and images based on channel state information feedback includes: - Semantic Importance Region Segmentation Module: Used to evaluate the semantic importance of different regions based on image semantic features, and to divide image features into Region of Interest (ROI) semantic features and non-semantic ROI features.

[0157] - Semantic Differentiated Transmission Control Module: Based on the semantic importance difference between ROI semantic features and non-ROI semantic features, it allocates differentiated transmission resources for different semantic features and performs channel coding and transmission control.

[0158] - Channel State Aware Cross-Modal Fusion Module: This module is used to introduce channel state information to construct an attention modulation mechanism and perform weighted fusion of text and image semantic features to achieve channel-aware cross-modal semantic representation.

[0159] In some embodiments, the semantic importance region segmentation module includes: - Cross-modal feature fusion unit, used to fuse text semantic features and image semantic features through a cross-modal attention mechanism to obtain fused features containing semantic relevance information; - ROI mask generation unit, used to generate continuous ROI probability masks that reflect the semantic importance of each spatial location in the image based on the fusion features; - A region segmentation unit is used to perform thresholding on the ROI probability mask to obtain a binary ROI mask, and to divide the image into ROI semantic features and non-ROI semantic features based on the binary ROI mask.

[0160] In some embodiments, the semantically differentiated transmission control module includes: - The channel quality assessment unit is used to analyze the multiple-input multiple-output (MIMO) channel and obtain channel quality information for each channel subspace. - Semantic signal classification unit, used to classify input semantic features according to semantic importance, to obtain ROI semantic features, non-ROI semantic features and text semantic features; - Sub-channel allocation unit, used to map semantic symbols corresponding to different semantic features to the corresponding channel subspace according to the channel quality information and the priority of semantic features, so as to realize semantic importance-driven resource allocation; - Signal mapping and transmission unit, used to map various semantic signals into corresponding transmission signals and transmit them through MIMO channel; - Signal detection and separation unit, used to perform signal detection and decoupling on the received signal at the receiving end in order to recover different semantic signals.

[0161] In some embodiments, the channel state-aware cross-modal fusion module includes: - Semantic feature integration unit, used to preprocess the different modal semantic features recovered by the receiver, and integrate the ROI semantic features and non-ROI semantic features in the image modality to obtain a unified image semantic representation; - Cross-modal attention fusion unit, used to fuse text semantic features and image semantic features based on cross-attention mechanism, where text semantic features are used as query vectors and image semantic features are used as keys and values ​​to achieve cross-modal semantic alignment and information interaction; - The channel state-aware modulation unit is used to generate a modulation mask related to channel quality based on the acquired channel state information, and to modulate the attention calculation process to dynamically adjust the weights of different semantic features. - Fusion feature generation unit, used to generate multimodal fusion semantic features based on the modulated attention results, and output them for subsequent semantic decoding or reconstruction.

[0162] In some embodiments, the text and image cross-modal semantic communication method based on channel state information feedback is characterized by including the following steps S11 to S18: S11. Perform semantic encoding on the input text to obtain text semantic features; S12. Perform semantic encoding on the input image to obtain image semantic features, and divide the image semantic features according to semantic importance to obtain ROI semantic features and non-ROI semantic features; S13. Based on the difference in semantic importance between the ROI semantic features and non-ROI semantic features, allocate differentiated channel transmission resources to different semantic features and perform channel coding. S14. Transmit the encoded semantic features through a multiple-input multiple-output (MIMO) channel; S15. The receiving end decodes the received signal to recover the text semantic features and image semantic features; S16. Obtain Channel State Information (CSI) and construct an attention modulation mask related to channel quality based on the CSI; S17. The text semantic features and image semantic features are fused based on a cross-modal attention mechanism, and the attention modulation mask is introduced during the fusion process to dynamically adjust the weights of different semantic features to obtain joint semantic features. S18. Based on the joint semantic features, generate the target image through an image reconstruction network.

[0163] In some embodiments, the specific implementation of the text and image cross-modal semantic communication method based on channel state information feedback is described. The communication system includes a transmitting end (i.e., a terminal device end) and a receiving end (i.e., a network device end), wherein the transmitting end is used to complete semantic encoding of text and images, ROI partitioning, and differentiated channel transmission, and the receiving end is used to complete channel estimation, cross-modal semantic fusion, and image reconstruction.

[0164] In some embodiments, for the semantic importance module, the cross-modal feature fusion unit first maps the input features into query, key, and value vectors through a linear mapping. Wherein, the image feature vector x I For the query vector and value vector, the text feature vector x T It is a key vector. The schematic diagram of the semantic importance region partitioning module is shown below. Figure 10 As shown, a multi-head attention mechanism (i.e., feature fusion network 1011, including multi-head attention layers and fully connected layers) is used, with the input being text semantic features x. I and image semantic features x T Attention weights are calculated on the query vector and key vector to obtain the fused cross-modal feature representation x. F .

[0165] After obtaining the cross-modal fusion features, the feature map (i.e., the cross-modal feature representation x) is recovered step by step through a deconvolution-based upsampling network 1002. F The spatial resolution maps low-resolution semantic features back to the pixel space of the original image, thereby obtaining feature representations with fine spatial structure. After multiple upsampling layers, the ROI probability mask is finally generated using the Sigmoid function 1003. .

[0166] Obtaining the ROI probability mask Next, the region separation unit 1004 is responsible for converting the continuous probability representation into discrete region partitioning results. Specifically, the generated ROI probability mask first needs to be binarized. Let the threshold be... Then the binary ROI mask can be expressed as the following formula (1). (1) in, This represents a binary ROI mask. A pixel value of 1 indicates that the location belongs to the ROI region; a pixel value of 0 indicates that it belongs to the non-ROI region. After obtaining the binary ROI mask, the original image can be separated into ROI regions and non-ROI regions through element-wise multiplication. The ROI regions and non-ROI regions can then be represented as... , .in, This represents element-wise multiplication. Through this operation, the image is divided into two non-overlapping regions. Using the above steps, the ROI detector can automatically identify key regions in an image, guided by textual semantics, and generate corresponding region segmentation results.

[0167] In some embodiments, for the semantically differentiated transmission control module: the communication system adopts a method consisting of N t One transmitting antenna and N rA MIMO channel model consisting of multiple receiving antennas. To achieve differentiated channel resource allocation based on semantic importance, singular value decomposition is performed on the channel matrix H: .in, The diagonal elements represent the channel subspace gain. Based on the channel subspace gain, the available subchannels are sorted, and high-gain subchannels are preferentially allocated to ROI semantic symbols, while low-gain subchannels are allocated to non-ROI semantic symbols and other semantic signals (i.e., text semantic symbols), thereby realizing a semantic-priority channel resource allocation strategy.

[0168] Based on this, spatial mapping of different semantic signals is performed using the sub-channel mapping matrix P, resulting in the transmitted signal representation as follows: , , .in, , , These represent the transmission signal matrices corresponding to ROI semantic signals, non-ROI semantic signals, and text semantic signals, respectively. Subsequently, these signals are spatially multiplexed to obtain the total transmission signal matrix. And it is transmitted through a MIMO channel, and the received signal is represented as .

[0169] The receiver performs channel estimation using pilot signals: The channel matrix is ​​estimated using the least squares method, as shown in formula (2) below: (2) in, The pilot signal for the transmitting end. For channel noise, This is the pilot signal for the receiving end.

[0170] Based on the estimated channel matrix, the MMSE detection method is used to recover the transmitted signal: Finally, the recovered signal is separated according to the sub-channel mapping relationship to obtain the ROI semantic signal, the non-ROI semantic signal, and the text semantic signal: , , .

[0171] In some embodiments, for the channel state-aware cross-modal fusion module: a flowchart of the channel state-aware cross-modal fusion module is shown below. Figure 11 As shown. The semantic feature integration unit at the receiving end obtains the text semantic features. and image semantic features , By integrating image modal semantic features, a unified image semantic representation is obtained. .

[0172] The cross-modal attention fusion unit is used to achieve information interaction and alignment between text semantic features and image semantic features. Specifically, the input features are mapped to query vectors, key vectors, and value vectors: , , .in, , , This is the learnable parameter matrix. Based on the above mapping results, the cross-modal attention representation is calculated. A flowchart of the channel state-aware cross-modal fusion module is shown below. Figure 11 As shown, The channel state-aware modulation unit is used to modulate the cross-modal attention calculation process based on channel state information. Specifically, it modulates the channel state information obtained through channel estimation. The input neural network performs feature mapping (i.e., CSI mask generation network 1101), and generates a CSI modulation mask through the Sigmoid function (not shown in the figure): .in, This represents a convolutional neural network. During the attention calculation process (i.e., the processing of linear layer 1102 and multi-head attention layer 1103), the CSI modulation mask is introduced to adjust the attention weights, resulting in: (3) in, This represents element-wise multiplication. For feature dimensions.

[0173] The fusion feature generation unit is used to perform a linear mapping on the modulated attention result (that is, to input the modulated attention result into the fully connected layer 1104) to obtain the final multimodal fusion semantic representation. The data is then fed into a semantic decoding network to reconstruct the target image.

[0174] In some embodiments, the overall implementation architecture flowchart, combining the above modules, is as follows: Figure 12 As shown, the process includes the following steps S21 to S27: S21: Semantic encoding steps: The input consists of text data and image data, wherein the input text is encoded using a text semantic encoder 1201 to obtain text semantic features x. T The input image is encoded using an image semantic encoder 1202 to obtain image semantic features x. I .

[0175] S22: Steps for dividing semantically important regions: The text semantic features x TImage semantic features x I Input ROI detector 1203, generating image semantic importance distribution through cross-modal semantic interaction, and obtaining ROI semantic features. semantic features of non-ROI Among them, ROI semantic features are used to represent the semantic features of regions with high semantic importance in the image, while non-ROI semantic features are used to represent the semantic features of regions with low semantic importance.

[0176] S23. Differentiated semantic encoding steps: Semantic features of the text x T ROI semantic features and non-ROI semantic features Perform independent joint source channel JSC encoder The encoding process of 1204 yields the corresponding encoded semantic representation. , , .

[0177] S24, MIMO channel transmission steps: Encoded text semantic symbols ROI semantic symbols and non-ROI semantic symbols The signal is mapped to MIMO channel 1205 for transmission, and a transmission signal matrix is ​​formed through spatial multiplexing, and the transmission process is completed through the wireless channel.

[0178] S25. Channel recovery and detection steps: The receiver performs channel estimation on the received signal to obtain Channel State Information (CSI), and separates the received signal from the JSC based on the Minimum Mean Square Error (MMSE) detection method. Decoding and recovering 1206 yields the corresponding decoded text semantic features y. T ROI semantic features and non-ROI semantic features y R y NR .

[0179] S26. Cross-modal semantic fusion steps: Semantic features of ROI and semantic features of non-ROI regions are combined. R y NR Integrate into y I The recovered text semantic features y T Image semantic features y I Input cross-modal semantic fusion (i.e., modal semantic fusion) (1207) Based on the cross-attention mechanism, cross-modal semantic alignment and information fusion are achieved, and the modulation mask generated by CSI is introduced to adaptively adjust the attention weights, thereby obtaining the fused semantic representation. .

[0180] S27. Semantic Decoding and Image Reconstruction Steps: The fused semantic representation is input into the semantic decoder. 1208 performs semantic decoding to obtain the semantic features of the decoded image. And further through image restoration networks Reconstruction is performed using 1209, and the final output is a restored image. .

[0181] In some embodiments, the communication system in this application includes a transmitter and a receiver, and can be divided into an application processing module (i.e., application processing layer 1301), a semantic processing module (semantic processing layer 1302), and a physical transmission module (i.e., physical transmission layer 1303) according to their functions. A schematic diagram of functional module interaction and signaling transmission is shown below. Figure 13 As shown. The application processing layer 1301 generates the text and image data to be transmitted and receives the final recovered target image (i.e., the recovered image). The semantic processing layer 1302 includes a text semantic encoder, an image semantic encoder, a ROI detector, a cross-modal semantic fusion unit, and an image semantic decoder, used to perform text semantic encoding, image semantic encoding, semantic importance analysis, cross-modal semantic fusion, and image semantic decoding. The physical transmission layer 1303 includes a joint source-channel encoder, a MIMO transmitter, a MIMO receiver, and an MMSE detector, used to perform joint source-channel coding, resource allocation and mapping, sub-channel selection, MIMO wireless transmission, channel estimation, signal detection, and joint source-channel decoding.

[0182] At the sending end, the application processing module first sends the input text data and image data to the semantic processing module. The semantic processing module extracts corresponding semantic features using a text semantic encoder and an image semantic encoder, respectively, and performs semantic importance analysis on the image semantic features using a ROI detector to obtain ROI semantic features and non-ROI semantic features. Further, based on the semantic importance analysis results, semantic importance indication information is generated and sent to the physical transmission module. The physical transmission module configures differentiated transmission resources for different semantic features according to the semantic importance indication information. Specifically, the physical transmission module, considering the current MIMO channel conditions, prioritizes allocating spatial sub-channels and transmission resources with better channel quality to ROI features with higher semantic importance; and allocates remaining transmission resources to non-ROI features with lower semantic importance, thereby improving the transmission reliability of key semantic information.

[0183] At the receiving end, the physical transmission module performs channel estimation and MMSE detection on the received signal, and decodes to recover the corresponding semantic features. Simultaneously, the physical transmission module acquires the current CSI and sends it to the semantic processing module. The semantic processing module constructs a channel quality-related attention modulation mask based on the CSI and introduces this modulation mask into the cross-modal semantic fusion process. By dynamically adjusting the cross-modal attention weights using the channel quality information reflected by the CSI, the fusion process can adaptively enhance the contribution of reliable semantic features and suppress the influence of semantic features severely affected by channel impairment, thereby improving the accuracy and robustness of the fused semantic representation. After completing the cross-modal semantic fusion, the fused semantic features are sequentially processed by semantic decoding and an image reconstruction network, ultimately generating a reconstructed image and outputting it to the application processing module.

[0184] Therefore, the embodiments of this application form a two-way collaborative mechanism that drives resource allocation based on semantic importance at the transmitting end and semantic fusion based on channel state at the receiving end, thereby realizing joint optimization configuration between semantic information and wireless transmission resources and improving the image restoration performance of cross-modal semantic communication systems in complex wireless channel environments.

[0185] In some embodiments, the signaling interaction process between User Equipment (UE) and Base Station (BS) is described based on the aforementioned cross-modal semantic communication method for text and images based on channel state information feedback.

[0186] In actual communication, to ensure that the sending and receiving ends can complete cross-modal semantic transmission based on a unified semantic representation space, the UE and BS first need to complete semantic communication capability negotiation, semantic model synchronization, and transmission parameter configuration. Subsequently, the two parties establish a semantic communication link based on the negotiation results, and combine channel state information to complete semantic feature transmission, cross-modal semantic fusion, and image reconstruction.

[0187] During signaling interaction, in addition to exchanging control signaling, the UE and BS respectively perform corresponding local semantic processing and physical layer processing operations. The UE is primarily responsible for text semantic encoding, image semantic encoding, ROI detection, semantic importance assessment, differentiated resource mapping, joint source channel coding, and MIMO transmission. The UE generates semantic importance indication information based on the ROI detection results and completes differentiated semantic transmission according to this information and the resource configuration parameters fed back by the BS. The BS is primarily responsible for channel estimation, CSI acquisition, MMSE signal detection, joint source channel decoding, cross-modal semantic fusion, semantic decoding, and image reconstruction. The BS obtains the CSI based on the received pilot and generates a channel-aware attention modulation mask based on the CSI to assist in weight adjustment during cross-modal semantic fusion, ultimately completing the reconstructed image generation.

[0188] The signaling interaction flow diagram between UE and BS is as follows: Figure 14 As shown, the specific process is as follows: S31. The UE sends a semantic communication establishment request to the BS. S32, BS returns a semantic communication establishment response to UE; Returns the supported semantic communication model identifier, model version number, and model parameters including embedding layer dimension, number of attention heads, and ROI detector threshold; S33, UE and BS perform model loading; Based on the negotiation results, the UE and BS load the transmitting end model, such as the text semantic coding model and the image semantic coding model, as well as the receiving end model, such as the cross-modal semantic fusion model and the semantic decoder, to complete the semantic model synchronization. S34. The UE sends a pilot signal to the BS; S35, BS performs channel estimation; Among them, channel estimation obtains CSI and BS, and resource configuration parameters are generated based on CSI; S36, BS sends resource configuration parameters to UE; Resource configuration parameters include: sub-channel allocation parameters and power allocation parameters; S37. The UE performs text image encoding, ROI detection, JSCC encoding, differential resource mapping, and generates cross-modal identifiers; Among them, the UE performs ROI-aware differentiated channel allocation based on resource configuration parameters; S38 and UE perform semantic data transmission; The UE transmits semantic information through the MIMO channel; S39, BS parsing of cross-modal association identifiers, JSCC decoding, modal fusion, semantic decoding, and image reconstruction; BS completes signal detection, uses CSI to construct attention modulation masks for cross-modal semantic fusion, and semantic recovery; S310 and BS send reconstructed images.

[0189] The BS outputs the final recovered image, completing one semantic communication process.

[0190] In this embodiment, to verify the effectiveness of the proposed cross-modal semantic communication method for text and images based on channel state information feedback, a single-modal image transmission method based on deep joint source-channel coding and a traditional communication method were selected as comparison schemes for performance testing. Specifically, comparison method one employs an end-to-end semantic transmission approach based solely on image information; comparison method two uses a traditional separate source coding and channel coding structure, encoding and transmitting text and images independently. Image reconstruction performance was measured under Rayleigh fading channel conditions in environments with different signal-to-noise ratios.

[0191] Performance is measured using Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) metrics for image semantic restoration. PSNR measures the difference in signal quality between the original image and the compressed, transmitted, or restored image. It quantifies image quality by calculating the ratio of the maximum possible signal value to the noise in the image. A higher PSNR value indicates better image quality and lower distortion. SSIM measures the similarity in structural information between the original image and the compressed, transmitted, or restored image. Unlike evaluation metrics based solely on pixel error, SSIM considers human visual perception, comprehensively taking into account brightness, contrast, and structure. It assesses image quality by comparing the consistency of two images across these dimensions. SSIM values ​​range from 0 to 1; values ​​closer to 1 indicate greater structural similarity and higher visual quality, while values ​​further away indicate more severe image distortion.

[0192] Simulation results are as follows Figure 15 and Figure 16As shown, the image reconstruction performance of each method shows an improving trend with the increase of signal-to-noise ratio (SNR). Under low SNR conditions, the performance of the method provided in this application is basically close to that of the first comparative method. As the SNR further increases, the performance of the method provided in this application gradually improves, and surpasses the first comparative method in the medium-to-high SNR range. Compared with traditional communication methods, the method provided in this application exhibits superior image reconstruction performance across the entire SNR range. Furthermore, under high SNR conditions, the method provided in this application improves the PSNR by approximately 22% compared to the first comparative method and by approximately 30% compared to traditional communication methods; it also improves the SSIM by approximately 10% compared to the first comparative method and by approximately 24% compared to traditional communication methods. Under low SNR conditions, the method provided in this application shows significant improvements in both PSNR and SSIM compared to traditional communication methods.

[0193] Fourthly, to implement the above-mentioned wireless communication method, an embodiment of this application provides a device 1700 (terminal device or network device), such as... Figure 17 As shown, it may include at least one processor 1701 and at least one transceiver 1702 coupled to at least one processor 1401. The transceiver 1702 may include at least one separate receiving circuitry and transmitting circuitry, or at least one integrated receiving circuitry and transmitting circuitry. The at least one processor 1701 may be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field-programmable gate array (FPGA), etc.

[0194] According to some embodiments of this application, when device 1700 is a terminal device, processor 1701 in 1700 may be referred to as a first processor, and transceiver 1702 may be referred to as a first transceiver.

[0195] The terminal device includes a first transceiver; and a first processor coupled to the first transceiver; the first processor is configured to: The CSI of the MIMO channel transmitted by the network device is received via the first transceiver. Determine textual semantic features, ROI semantic features of the image, and non-ROI semantic features of the image; the ROI semantic features and the non-ROI semantic features are determined using text, which is associated with the image; Joint source-channel coding is performed on the text semantic features, the ROI semantic features, and the non-ROI semantic features to obtain text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols. Based on the CSI, the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols are mapped to channel subspaces of different channel qualities to obtain text semantic signals, ROI semantic signals, and non-ROI semantic signals; The text semantic signal, the ROI semantic signal, and the non-ROI semantic signal are transmitted to the network device via the first transceiver and the MIMO channel.

[0196] In some embodiments, the first processor, based on the CSI, maps the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols to channel subspaces of different channel qualities to obtain text semantic signals, ROI semantic signals, and non-ROI semantic signals, which are then configured to: Based on the CSI, channel subspaces with different channel qualities are assigned to the text semantic features, the ROI semantic features, and the non-ROI semantic features, respectively; The text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols are mapped to channel subspaces of different channel qualities to obtain text semantic signals, ROI semantic signals, and non-ROI semantic signals.

[0197] In some embodiments, the first processor, based on the CSI, allocates channel subspaces of different channel qualities to the text semantic features, the ROI semantic features, and the non-ROI semantic features, configured to: Determine semantic importance indication information, which is used to reflect the relative semantic importance among the text semantic features, the ROI semantic features, and the non-ROI semantic features; Based on the CSI, the gain of multiple channel subspaces of the MIMO channel is determined; Based on the semantic importance indication information and the gain of the multiple channel subspaces, channel subspaces with different channel qualities are assigned to the text semantic features, the ROI semantic features, and the non-ROI semantic features, respectively.

[0198] In some embodiments, the first processor, in determining text semantic features, ROI semantic features of the image, and non-ROI semantic features of the image, is configured to: Semantic encoding is performed on the text and the image respectively to obtain text semantic features and image semantic features; Based on the first similarity between the text semantic features and the image semantic features, a ROI mask is determined, wherein the ROI mask includes the mask value corresponding to each semantic feature in the image semantic features; Based on the ROI mask, the image semantic features are divided into ROI semantic features and non-ROI semantic features.

[0199] In some embodiments, the first processor determines a ROI mask based on the similarity between the text semantic features and the image semantic features, and configures it to: A first attention weight is determined based on the first similarity between the text semantic features and the image semantic features; The text semantic features and the image semantic features are fused using the first attention weight to obtain a first fused feature map; The first fused feature map is converted into a ROI mask.

[0200] In some embodiments, the first processor, in determining text semantic features, region of interest (ROI) semantic features of the image, and non-ROI semantic features of the image, is configured to: Based on the second similarity between the text and the image, the image is divided into ROI regions and non-ROI regions; Semantic encoding is performed on the text, the ROI region, and the non-ROI region respectively to obtain the text semantic features, the ROI semantic features, and the non-ROI semantic features.

[0201] In some embodiments, the first processor maps the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols to channel subspaces of different channel qualities to obtain text semantic signals, ROI semantic signals, and non-ROI semantic signals, which are then configured to: Corresponding association identifiers are generated for the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols, respectively. The association identifiers are used to characterize the association relationships between the text semantic features, the ROI semantic features, and the non-ROI semantic features. Based on the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols and their corresponding association identifiers, text data packets, ROI data packets, and non-ROI data packets are determined. The text data packets, ROI data packets, and non-ROI data packets are mapped to channel subspaces of different channel qualities to obtain text semantic signals, ROI semantic signals, and non-ROI semantic signals.

[0202] In some embodiments, the first processor is configured to: A pilot signal is transmitted on the MIMO channel, and the pilot signal is used by the network device to determine the CSI.

[0203] In some embodiments, the first processor is configured to: The network device receives a reconstructed image via the MIMO channel. The reconstructed image is obtained by recovering the text semantic signal, the ROI semantic signal, and the non-ROI semantic signal based on the CSI of the MIMO channel.

[0204] According to some embodiments of this application, when device 1700 is a network device, processor 1701 in 1700 may be referred to as a second processor, and transceiver 1702 may be referred to as a second transceiver.

[0205] The network device includes a second transceiver; and a second processor coupled to the second transceiver; the second processor is configured to: The channel state information (CSI) of the multiple-input multiple-output (MIMO) channel is sent to the terminal device via the second transceiver. The second transceiver receives text semantic signals, ROI semantic signals, and non-ROI semantic signals sent by the terminal device via the MIMO channel. Signal detection is performed on the text semantic signal, the ROI semantic signal, and the non-ROI semantic signal respectively to obtain text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols; Joint source-channel decoding is performed on the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols respectively to obtain text semantic features, ROI semantic features, and non-ROI semantic features; Based on the CSI, the text semantic features, the ROI semantic features, and the non-ROI semantic features are restored to the reconstructed image.

[0206] In some embodiments, the second processor, based on the CSI, recovers the text semantic features, the ROI semantic features, and the non-ROI semantic features into a reconstructed image, configured to: The semantic features of the ROI and the semantic features of the non-ROI are fused to obtain the image semantic features; Based on the CSI, the text semantic features and the image semantic features are fused to obtain a second fused feature map; The reconstructed image is obtained based on the reconstruction of the second fused feature map.

[0207] In some embodiments, the second processor fuses the text semantic features and the image semantic features based on the CSI to obtain a second fused feature map, which is configured to: Perform feature mapping on the CSI to determine the CSI modulation mask; Based on the CSI modulation mask, the text semantic features and the image semantic features are fused to obtain a second fused feature map.

[0208] In some embodiments, the second processor fuses the text semantic features and the image semantic features based on the CSI modulation mask to obtain a second fused feature map, which is configured to: The image semantic features are mapped to key vectors and value vectors, and the text semantic features are mapped to query vectors; Determine the third similarity between the key vector and the query vector; Based on the third similarity and the CSI modulation mask, the second attention weight of the value vector is determined; Based on the second attention weight and the value vector, a second fused feature map is obtained.

[0209] In some embodiments, the second processor is configured to: Signal detection is performed on the text semantic signal, the ROI semantic signal, and the non-ROI semantic signal respectively to obtain the association identifiers of the text semantic symbol, the ROI semantic symbol, and the non-ROI semantic symbol; Specifically, when the association identifiers of the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols are matched, the text semantic features, the ROI semantic features, and the non-ROI semantic features are restored to a reconstructed image based on the CSI.

[0210] In some embodiments, the second processor is configured to: The pilot signal transmitted by the terminal device is received through the MIMO channel; Based on the pilot signal, the CSI of the MIMO channel is determined.

[0211] In some embodiments, the second processor is configured to: The reconstructed image is sent to the terminal device.

[0212] Fifthly, embodiments of this application provide a wireless communication device 100, the device 100 comprising: The first receiving module is configured to receive channel state information (CSI) of the multiple-input multiple-output (MIMO) channel sent by the network device; The first determining module is configured to determine text semantic features, region of interest (ROI) semantic features of the image, and non-ROI semantic features of the image; the ROI semantic features and the non-ROI semantic features are determined using text, and the text is associated with the image; The encoding module is configured to perform joint source-channel coding on the text semantic features, the ROI semantic features, and the non-ROI semantic features respectively to obtain text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols; The mapping module is configured to map the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols to channel subspaces of different channel qualities based on the CSI, thereby obtaining text semantic signals, ROI semantic signals, and non-ROI semantic signals; The first transmitting module is configured to transmit the text semantic signal, the ROI semantic signal, and the non-ROI semantic signal to the network device through the MIMO channel.

[0213] In some embodiments, the mapping module is further configured to, based on the CSI, allocate channel subspaces of different channel qualities to the text semantic features, the ROI semantic features, and the non-ROI semantic features respectively; and map the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols to channel subspaces of different channel qualities to obtain text semantic signals, ROI semantic signals, and non-ROI semantic signals.

[0214] In some embodiments, the mapping module is further configured to determine semantic importance indication information, which reflects the relative semantic importance among the text semantic features, the ROI semantic features, and the non-ROI semantic features; determine the gain of multiple channel subspaces of the MIMO channel based on the CSI; and allocate channel subspaces of different channel qualities to the text semantic features, the ROI semantic features, and the non-ROI semantic features based on the semantic importance indication information and the gain of the multiple channel subspaces.

[0215] In some embodiments, the first determining module is further configured to perform semantic encoding on the text and the image respectively to obtain text semantic features and image semantic features; determine a ROI mask based on a first similarity between the text semantic features and the image semantic features, wherein the ROI mask includes the mask value corresponding to each semantic feature in the image semantic features; and divide the image semantic features into ROI semantic features and non-ROI semantic features based on the ROI mask.

[0216] In some embodiments, the first determining module is further configured to determine a first attention weight based on a first similarity between the text semantic features and the image semantic features; fuse the text semantic features and the image semantic features using the first attention weight to obtain a first fused feature map; and convert the first fused feature map into the ROI mask.

[0217] In some embodiments, the first determining module is further configured to divide the image into ROI regions and non-ROI regions based on a second similarity between the text and the image; and to perform semantic encoding on the text, the ROI regions and the non-ROI regions respectively to obtain the text semantic features, the ROI semantic features and the non-ROI semantic features.

[0218] In some embodiments, the mapping module is further configured to generate corresponding association identifiers for the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols, respectively. The association identifiers are used to characterize the association relationships between the text semantic features, the ROI semantic features, and the non-ROI semantic features. Based on the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols and their corresponding association identifiers, text data packets, ROI data packets, and non-ROI data packets are determined. The text data packets, ROI data packets, and non-ROI data packets are mapped to channel subspaces of different channel qualities to obtain text semantic signals, ROI semantic signals, and non-ROI semantic signals.

[0219] In some embodiments, the first transmitting module is further configured to transmit a pilot signal on the MIMO channel, the pilot signal being used by the network device to determine the CSI.

[0220] In some embodiments, the first receiving module is further configured to receive a reconstructed image sent by the network device through the MIMO channel, the reconstructed image being obtained by recovering the text semantic signal, the ROI semantic signal, and the non-ROI semantic signal based on the CSI of the MIMO channel.

[0221] Sixthly, embodiments of this application provide a wireless communication device 110, the device 110 comprising: The second transmitting module is configured to transmit Channel State Information (CSI) of the Multiple Input Multiple Output (MIMO) channel to the terminal device. The second receiving module is configured to receive text semantic signals, region of interest (ROI) semantic signals, and non-ROI semantic signals sent by the terminal device via a MIMO channel. The signal detection module is configured to perform signal detection on the text semantic signal, the ROI semantic signal and the non-ROI semantic signal respectively to obtain text semantic symbols, ROI semantic symbols and non-ROI semantic symbols; The decoding module is configured to perform joint source-channel decoding on the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols respectively to obtain text semantic features, ROI semantic features, and non-ROI semantic features. The recovery module is configured to recover the text semantic features, the ROI semantic features, and the non-ROI semantic features into a reconstructed image based on the CSI.

[0222] In some embodiments, the recovery module is further configured to fuse the ROI semantic features and the non-ROI semantic features to obtain image semantic features; based on the CSI, fuse the text semantic features and the image semantic features to obtain a second fused feature map; and based on the reconstruction of the second fused feature map, obtain the reconstructed image. In some embodiments, the recovery module is further configured to perform feature mapping on the CSI to determine a CSI modulation mask; and to fuse the text semantic features and the image semantic features based on the CSI modulation mask to obtain a second fused feature map.

[0223] In some embodiments, the recovery module is further configured to map the image semantic features to key vectors and value vectors, and map the text semantic features to query vectors; determine a third similarity between the key vectors and query vectors; determine a second attention weight for the value vectors based on the third similarity and the CSI modulation mask; and obtain a second fused feature map based on the second attention weight and the value vectors.

[0224] In some embodiments, the signal detection module is further configured to perform signal detection on the text semantic signal, the ROI semantic signal, and the non-ROI semantic signal respectively to obtain the association identifiers of the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols; wherein, if the association identifiers of the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols match, the text semantic features, the ROI semantic features, and the non-ROI semantic features are restored to a reconstructed image based on the CSI.

[0225] In some embodiments, the second receiving module is further configured to receive a pilot signal sent by the terminal device through the MIMO channel; the device 110 further includes a second determining module; the second determining module is further configured to determine the CSI of the MIMO channel based on the pilot signal.

[0226] In some embodiments, the second sending module is further configured to send the reconstructed image to the terminal device.

[0227] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0228] It should be noted that, in the embodiments of this application, if the above-mentioned wireless communication method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0229] In a seventh aspect, to implement the above-described wireless communication method, embodiments of this application provide an electronic device, including a memory and a processor. The memory stores a computer program that can run on the processor, and when the processor executes the program, it implements the steps in the wireless communication method provided in the above embodiments.

[0230] Eighthly, embodiments of this application provide a storage medium, namely a computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements the steps in the wireless communication method provided in the above embodiments.

[0231] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0232] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0233] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0234] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0235] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0236] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0237] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0238] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0239] The above are merely embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A terminal device, the terminal device comprising a first transceiver; and A first processor, coupled to the first transceiver; the first processor is configured to: The channel state information (CSI) of the multiple-input multiple-output (MIMO) channel sent by the network device is received via the first transceiver. Determine textual semantic features, region of interest (ROI) semantic features of the image, and non-ROI semantic features of the image; the ROI semantic features and the non-ROI semantic features are determined using text, which is associated with the image; Joint source-channel coding is performed on the text semantic features, the ROI semantic features, and the non-ROI semantic features to obtain text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols. Based on the CSI, the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols are mapped to channel subspaces of different channel qualities to obtain text semantic signals, ROI semantic signals, and non-ROI semantic signals; The text semantic signal, the ROI semantic signal, and the non-ROI semantic signal are transmitted to the network device via the first transceiver and the MIMO channel.

2. The terminal device according to claim 1, wherein the first processor, based on the CSI, maps the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols to channel subspaces of different channel qualities to obtain text semantic signals, ROI semantic signals, and non-ROI semantic signals, and configures them as follows: Based on the CSI, channel subspaces with different channel qualities are assigned to the text semantic features, the ROI semantic features, and the non-ROI semantic features, respectively; The text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols are mapped to channel subspaces of different channel qualities to obtain text semantic signals, ROI semantic signals, and non-ROI semantic signals.

3. The terminal device according to claim 1, wherein the first processor, based on the CSI, allocates channel subspaces of different channel qualities to the text semantic features, the ROI semantic features, and the non-ROI semantic features, and configures them to: Determine semantic importance indication information, which is used to reflect the relative semantic importance among the text semantic features, the ROI semantic features, and the non-ROI semantic features; Based on the CSI, the gain of multiple channel subspaces of the MIMO channel is determined; Based on the semantic importance indication information and the gain of the multiple channel subspaces, channel subspaces with different channel qualities are assigned to the text semantic features, the ROI semantic features, and the non-ROI semantic features, respectively.

4. The terminal device according to claim 1, wherein the first processor, after determining the text semantic features, the ROI semantic features of the image, and the non-ROI semantic features of the image, is configured to: Semantic encoding is performed on the text and the image respectively to obtain text semantic features and image semantic features; Based on the first similarity between the text semantic features and the image semantic features, a ROI mask is determined, wherein the ROI mask includes the mask value corresponding to each semantic feature in the image semantic features; Based on the ROI mask, the image semantic features are divided into ROI semantic features and non-ROI semantic features.

5. The terminal device according to claim 1, wherein the first processor, after determining the semantic features of the text, the semantic features of the region of interest (ROI) of the image, and the semantic features of the non-ROI of the image, is configured to: Based on the second similarity between the text and the image, the image is divided into ROI regions and non-ROI regions; Semantic encoding is performed on the text, the ROI region, and the non-ROI region respectively to obtain the text semantic features, the ROI semantic features, and the non-ROI semantic features.

6. The terminal device according to claim 1, wherein the first processor maps the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols to channel subspaces of different channel qualities to obtain text semantic signals, ROI semantic signals, and non-ROI semantic signals, and configures them as follows: Corresponding association identifiers are generated for the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols, respectively. The association identifiers are used to characterize the association relationships between the text semantic features, the ROI semantic features, and the non-ROI semantic features. Based on the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols and their corresponding association identifiers, text data packets, ROI data packets, and non-ROI data packets are determined. The text data packets, ROI data packets, and non-ROI data packets are mapped to channel subspaces of different channel qualities to obtain text semantic signals, ROI semantic signals, and non-ROI semantic signals.

7. The terminal device according to any one of claims 1 to 6, wherein the first processor is configured to: The network device receives a reconstructed image via the MIMO channel. The reconstructed image is obtained by recovering the text semantic signal, the ROI semantic signal, and the non-ROI semantic signal based on the CSI of the MIMO channel.

8. A network device, the network device comprising a second transceiver; and A second processor, coupled to the second transceiver; the second processor is configured to: The channel state information (CSI) of the multiple-input multiple-output (MIMO) channel is sent to the terminal device via the second transceiver. The second transceiver receives text semantic signals, region of interest (ROI) semantic signals, and non-ROI semantic signals transmitted by the terminal device via a MIMO channel. Signal detection is performed on the text semantic signal, the ROI semantic signal, and the non-ROI semantic signal respectively to obtain text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols; Joint source-channel decoding is performed on the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols respectively to obtain text semantic features, ROI semantic features, and non-ROI semantic features; Based on the CSI, the text semantic features, the ROI semantic features, and the non-ROI semantic features are restored to the reconstructed image.

9. The network device according to claim 8, wherein the second processor, based on the CSI, restores the text semantic features, the ROI semantic features, and the non-ROI semantic features into a reconstructed image, configured to: The semantic features of the ROI and the semantic features of the non-ROI are fused to obtain the image semantic features; Based on the CSI, the text semantic features and the image semantic features are fused to obtain a second fused feature map; The reconstructed image is obtained based on the reconstruction of the second fused feature map.

10. A wireless communication method, the method being applied to a terminal device, the method comprising: Receive Channel State Information (CSI) for the Multiple-Input Multiple-Output (MIMO) channel sent by the network device; Determine textual semantic features, region of interest (ROI) semantic features of the image, and non-ROI semantic features of the image; the ROI semantic features and the non-ROI semantic features are determined using text, which is associated with the image; Joint source-channel coding is performed on the text semantic features, the ROI semantic features, and the non-ROI semantic features to obtain text semantic symbols, ROI semantic symbols, and non-ROI semantic symbols. Based on the CSI, the text semantic symbols, the ROI semantic symbols, and the non-ROI semantic symbols are mapped to channel subspaces of different channel qualities to obtain text semantic signals, ROI semantic signals, and non-ROI semantic signals; The text semantic signal, the ROI semantic signal, and the non-ROI semantic signal are transmitted to the network device through the MIMO channel.