Semantic communication method, semantic communication device, semantic communication system, splicing model training method and splicing model training device

Abstract

The invention provides a semantic communication method and device, and relates to the technical field of communication, in particular to the technical fields of large models, deep learning, image processing and the like. The specific implementation scheme is as follows: receiving semantic extraction information of at least two to-be-spliced images transmitted by a channel; performing received signal processing on the semantic extraction information to obtain decoding information; performing multi-scale semantic feature extraction on the decoded information to generate multiple groups of semantic feature information of different scales; and obtaining a panoramic image corresponding to the to-be-spliced image based on the semantic feature information.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, specifically to technical fields such as large models, deep learning, and image processing, and in particular to a semantic communication method and device, a semantic communication system, a splicing model training method and device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] At present, before the panoramic image data stitched by multiple videos reaches the user end, it needs to be shot by a panoramic camera to obtain multiple videos, then sent to a device with sufficient computing power for stitching and other processing, and finally transmitted through a communication link. Such a solution is inefficient for real-time transmission. In addition, due to the huge amount of data corresponding to the panoramic image business, the channel resources are still under great pressure when using traditional source compression methods and channel coding. Summary of the invention

[0003] The present disclosure provides a semantic communication method and device, a semantic communication system, a splicing model training method and device, an electronic device, a computer-readable storage medium, and a computer program product.

[0004] According to a first aspect, a semantic communication method is provided, the method comprising: receiving semantic extraction information of at least two images to be stitched transmitted by a channel; performing received signal processing on the semantic extraction information to obtain decoded information; performing multi-scale semantic feature extraction on the decoded information to generate multiple groups of semantic feature information of different scales; and obtaining a panoramic image corresponding to the images to be stitched based on the semantic feature information.

[0005] According to a second aspect, another semantic communication method is provided, the method comprising: receiving at least two images to be stitched; inputting the images to be stitched into a semantic information extractor to obtain initial semantic information output by the semantic information extractor; performing sending signal processing on the initial semantic information to obtain semantic extraction information; sending the semantic extraction information to a receiving end through a channel so that the receiving end performs multi-scale semantic feature extraction on the semantic extraction information, and obtains a panoramic image of the images to be stitched based on the extracted semantic feature information.

[0006] According to a third aspect, another semantic communication method is provided, the method comprising: a transmitting end obtains semantic extraction information of at least two images to be stitched; the transmitting end sends the semantic extraction information to a receiving end through a channel; the receiving end receives the semantic extraction information; the receiving end performs received signal processing on the semantic extraction information to obtain decoded information; the receiving end performs multi-scale semantic feature extraction on the decoded information to generate multiple groups of semantic feature information of different scales; the receiving end obtains a panoramic image corresponding to the image to be stitched based on the semantic feature information.

[0007] According to a fourth aspect, a stitching model training method is provided, the method comprising: selecting image samples from an image sample set; inputting the image samples into a semantic information extractor in an initial stitching model to obtain initial semantic information; performing sending signal processing on the initial semantic information to obtain semantic extraction information; adding noise to the semantic extraction information, and performing receiving signal processing on the semantic extraction information to obtain decoding information; using a multi-scale semantic feature extractor in the stitching model to perform multi-scale semantic feature extraction on the decoding information to obtain multiple groups of semantic feature information of different scales; using a diffusion model in the stitching model to process the semantic feature information and image samples to obtain a panoramic image corresponding to the image samples; based on the panoramic image, calculating the loss of the stitching model; in response to determining that the stitching model meets the training completion condition based on the loss, a trained stitching model is obtained.

[0008] According to a fifth aspect, a semantic communication device is provided, which includes: an information receiving unit, configured to receive semantic extraction information of at least two images to be stitched transmitted by a channel; a processing unit, configured to perform received signal processing on the semantic extraction information to obtain decoded information; a generating unit, configured to perform multi-scale semantic feature extraction on the decoded information to generate multiple groups of semantic feature information of different scales; and an image obtaining unit, configured to obtain a panoramic image corresponding to the image to be stitched based on the semantic feature information.

[0009] According to a sixth aspect, another semantic communication device is provided, which includes: an image receiving unit, configured to receive at least two images to be stitched; an input unit, configured to input the images to be stitched into a semantic information extractor, and obtain initial semantic information output by the semantic information extractor; an information obtaining unit, configured to perform sending signal processing on the initial semantic information, and obtain semantic extraction information; a sending unit, configured to send the semantic extraction information to a receiving end through a channel, so that the receiving end performs multi-scale semantic feature extraction on the semantic extraction information, and obtains a panoramic image of the images to be stitched based on the extracted semantic feature information.

[0010] According to the seventh aspect, a stitching model training device is provided, which includes: a selection unit, configured to select image samples from an image sample set; a training unit, configured to input the image samples into a semantic information extractor in an initial stitching model to obtain initial semantic information; perform sending signal processing on the initial semantic information to obtain semantic extraction information; add noise to the semantic extraction information, and perform receiving signal processing on the semantic extraction information to obtain decoding information; use a multi-scale semantic feature extractor in the stitching model to perform multi-scale semantic feature extraction on the decoding information to obtain multiple groups of semantic feature information of different scales; use a diffusion model in the stitching model to process the semantic feature information and image samples to obtain a panoramic image corresponding to the image samples; a calculation unit, configured to calculate the loss of the stitching model based on the panoramic image; a model acquisition unit, configured to respond to determining that the stitching model meets the training completion condition based on the loss, and obtain a stitching model that has been trained.

[0011] According to an eighth aspect, a semantic communication system is provided, which includes: a receiving end, a channel and a transmitting end; the transmitting end is used to receive images to be stitched including at least two images; the images to be stitched are input into a semantic information extractor to obtain initial semantic information output by the semantic information extractor; the initial semantic information is subjected to sending signal processing to obtain semantic extraction information; the semantic extraction information is sent to the receiving end through the channel; the receiving end is used to receive semantic extraction information of at least two images to be stitched transmitted through the channel; the semantic extraction information is subjected to receiving signal processing to obtain decoding information; multi-scale semantic feature extraction is performed on the decoded information to generate multiple groups of semantic feature information of different scales; based on the semantic feature information, a panoramic image corresponding to the images to be stitched is obtained.

[0012] According to the ninth aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any implementation of the first aspect, the second aspect, or the third aspect.

[0013] According to the tenth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method described in any implementation of the first aspect, the second aspect, or the third aspect.

[0014] The semantic communication method and device provided by the embodiments of the present disclosure firstly receive the semantic extraction information of at least two images to be stitched transmitted by the channel; secondly, perform receiving signal processing on the semantic extraction information to obtain decoding information; thirdly, perform multi-scale semantic feature extraction on the decoded information to generate multiple groups of semantic feature information of different scales; finally, obtain a panoramic image corresponding to the image to be stitched based on the semantic feature information. Thus, the semantic extraction information is processed at the receiving end to generate a panoramic image of at least two images to be stitched, thereby placing the image stitching at the receiving end, and the channel only needs to transmit the semantic extraction information, which reduces the pressure of channel data transmission and improves the transmission efficiency of the panoramic image data; multiple groups of semantic feature information of different scales are used to obtain the panoramic image, which improves the reliability of obtaining the panoramic image.

[0015] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0017] Figure 1 is a flow chart of an embodiment of a semantic communication method according to the present disclosure;

[0018] Figure 2 It is a structural diagram of a system corresponding to the semantic communication method disclosed in the present invention;

[0019] Figure 3 is a flow chart of another embodiment of the semantic communication method according to the present disclosure;

[0020] Figure 4 is a flow chart of an embodiment of a splicing model training method according to the present disclosure;

[0021] Figure 5 is a structural schematic diagram of an embodiment of a semantic communication device according to the present disclosure;

[0022] Figure 6 is a structural schematic diagram of another embodiment of the semantic communication device according to the present disclosure;

[0023] Figure 7 is a structural schematic diagram of an embodiment of a splicing model training device according to the present disclosure;

[0024] Figure 8 is a structural schematic diagram of an embodiment of a semantic communication system according to the present disclosure;

[0025] Fig. 9It is a block diagram of an electronic device used to implement the semantic communication method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0027] With the advent of 6G, VR (Virtual Reality) is considered one of the most important high-traffic applications. Ubiquitous VR is one of the visions of 6G, but the huge amount of VR data and the need for data splicing and other processing make most VR applications on-demand services, while live broadcast services require huge computing power to support.

[0028] At present, before VR data reaches the user end, it must first be captured by a panoramic camera to obtain multiple videos, then sent to a device with sufficient computing power for stitching and other processing, and then undergo source coding and channel coding before being transmitted through a communication link.

[0029] Such a solution is inefficient for real-time transmission. In addition, due to the huge amount of data in VR services, using traditional source compression methods and channel coding will still put great pressure on channel resources and produce a "cliff effect" under harsh and changeable channel conditions.

[0030] In view of the above defects, the present invention proposes a semantic communication method to improve the efficiency of panoramic image data transmission. Figure 1 A process 100 according to an embodiment of a semantic communication method of the present disclosure is shown. The semantic communication method comprises the following steps:

[0031] Step 101: receiving semantic extraction information of at least two images to be stitched transmitted through a channel.

[0032] In this embodiment, the channel is a communication device that connects the sending end and the receiving end, and transmits the signal from the sending end to the receiving end. The channel is divided into: wireless channel and wired channel according to the transmission medium. Among them, the wireless channel uses the propagation of electromagnetic waves in space to transmit signals, and the wired channel uses artificial photoconductive or optical signal media to transmit signals. Optical fiber is a transmission medium widely used in current wired optical communication systems.

[0033] In this embodiment, the at least two images to be stitched may be images obtained by photographing an entity using multiple different camera devices in the same scene, and any two adjacent images to be stitched in the at least two images to be stitched may have the same display portion or may not have the same display portion. Figure 2 In the at least two images to be stitched D include two images to be stitched, and the two images to be stitched have the same display portion. It should be noted that the device for shooting the images to be stitched can be any type of camera with different numbers, and the type of camera can be a fisheye lens or a pinhole lens.

[0034] In this embodiment, the semantic extraction information is semantic information obtained after semantic extraction of at least two images to be stitched, and the semantic extraction information is used to characterize the image meaning contained in the at least two images to be stitched. For example, the semantic extraction information is a semantic feature map of the at least two images to be stitched.

[0035] In this embodiment, the semantic extraction information can be obtained by identifying and extracting meaningful objects, scenes or concepts from at least two images to be stitched through image processing and deep learning technology. This obtaining process includes: performing contour extraction and semantic segmentation on at least two images to be stitched to obtain the semantic extraction information.

[0036] In this embodiment, the execution subject on which the semantic communication method runs can be as follows: Figure 2 The receiving end shown in FIG. 1 receives the semantic extraction information transmitted in the channel. Figure 2 In the embodiment, the channel between the transmitter and the receiver can be a wireless channel.

[0037] In the technical solution disclosed in this disclosure, the collection, storage, use, processing, transmission, provision and disclosure of semantically extracted information are carried out after authorization and in compliance with relevant laws and regulations. The information related to the user in the semantically extracted information is the information obtained after the user's permission.

[0038] Step 102: Perform received signal processing on the semantic extraction information to obtain decoded information.

[0039] In this embodiment, the received signal processing refers to signal processing performed on the received channel data, and the received signal processing includes: decoding processing, demodulation processing, signal amplification processing, etc.

[0040] In this embodiment, the received signal processing is performed by the semantic extraction information transmitter on the transmitted signal. For example, the transmitter encodes the transmitted signal to obtain the semantic extraction information, and the received signal processing performed by the execution subject after receiving the semantic extraction information includes decoding the semantic extraction information. For another example, the transmitter adjusts the transmitted signal to obtain the semantic extraction information, and the received signal processing performed by the execution subject after receiving the semantic extraction information includes demodulating the semantic extraction information.

[0041] In one example, the above step 102 includes: performing signal amplification and decoding processing on the semantic extraction information in sequence to obtain decoded information.

[0042] In this embodiment, the decoded information is information representing semantic features of at least two images to be stitched together. By processing the decoded information, the at least two images to be stitched together can be stitched together to obtain a panoramic image.

[0043] Step 103: extract multi-scale semantic features from the decoded information to generate multiple groups of semantic feature information at different scales.

[0044] In this embodiment, the scale of semantic features is a key concept in computer vision, especially in the semantic segmentation task. The goal of semantic segmentation is to assign a semantic label to each pixel in the image, which requires the model to not only identify the objects in the image, but also accurately determine the boundaries of these objects. In order to achieve this goal, the model needs to process feature information of different scales.

[0045] In this embodiment, the semantic feature information is information obtained after extracting semantic features of different scales from the decoded information. For example, the semantic feature information of different scales refers to semantic feature images of different scales.

[0046] Step 104: obtaining a panoramic image corresponding to the image to be stitched based on the semantic feature information.

[0047] In this embodiment, the panoramic image is an image obtained by stitching at least two images to be stitched.

[0048] In this embodiment, semantic feature information of different scales can describe semantic information of the images to be stitched in different ranges, and the semantic feature information of different scales can be processed by the diffusion model for stitching images to obtain a panoramic image of at least two images to be stitched. The diffusion model can be a diffusion model in the stitching model trained by the stitching model training method disclosed in the present invention.

[0049] In this embodiment, the execution subject on which the semantic communication method runs can be as follows: Figure 2The receiving end shown obtains semantic extraction information through a wireless channel, processes the semantic extraction information, obtains and outputs a panoramic image Q.

[0050] The semantic communication method provided in this embodiment can be used to generate VR panoramic images. The receiving end directly extracts semantic information from at least two original images to be stitched taken by a panoramic camera and transmits the extracted semantic information. The receiving end uses a diffusion model and semantic feature information to directly generate a stitched panoramic image, thereby integrating the panoramic image stitching process and the transmission process, reducing the amount of data transmitted during the communication process and the end-to-end transmission efficiency.

[0051] The semantic communication method provided by the embodiment of the present disclosure first receives the semantic extraction information of at least two images to be stitched transmitted by the channel; secondly, performs receiving signal processing on the semantic extraction information to obtain decoding information; thirdly, performs multi-scale semantic feature extraction on the decoded information to generate multiple groups of semantic feature information of different scales; finally, obtains a panoramic image corresponding to the image to be stitched based on the semantic feature information. Thus, the semantic extraction information is processed at the receiving end to generate a panoramic image of at least two images to be stitched, thereby placing the stitching of the images at the receiving end, and the channel only needs to transmit the semantic extraction information, which reduces the pressure of channel data transmission and improves the transmission efficiency of the panoramic image data; multiple groups of semantic feature information of different scales are used to obtain the panoramic image, which improves the reliability of obtaining the panoramic image.

[0052] In some optional implementations of the present disclosure, the above-mentioned multi-scale semantic feature extraction of the decoded information to generate multiple groups of semantic feature information of different scales includes: inputting the decoded information into a pre-trained multi-scale semantic feature extractor, and obtaining multiple groups of semantic feature information of different scales output by the multi-scale conditional extractor, the multi-scale semantic feature extractor includes at least two feature extractors, each feature extractor includes: a residual network, a transposed convolutional network and a self-attention layer.

[0053] In this optional implementation, if Figure 2 As shown, the multi-scale semantic feature extractor includes at least two feature extractors, and therefore, the semantic feature information input to the diffusion model is multi-scale feature information.

[0054] In this optional implementation, the feature extractor generates semantic feature information based on the decoded information. The core idea of ​​the residual network is to solve the training problem of deep networks by introducing residual blocks. Each residual block contains one or more convolutional layers and a skip connection, which directly adds the input to the output of the convolutional layer.

[0055] In this optional implementation, the transposed convolutional network is a deconvolution operation network. The transposed convolutional network expands the size of the feature map by inserting some padding values ​​between the input feature maps. Image upsampling can be achieved through the transposed convolutional network.

[0056] In this optional implementation, the self-attention layer is a network layer that helps the model understand complex relationships in the image. The self-attention layer can help the feature extractor understand the distorted images taken by the panoramic camera.

[0057] In this optional implementation, the number and size of scales of the semantic feature information extracted by each feature extractor in the multi-scale semantic feature extractor can be set based on development requirements.

[0058] The method for generating multiple groups of semantic feature information of different scales provided by this optional implementation adopts a multi-scale semantic feature extractor to perform multi-scale semantic feature information on decoded information, which can fully consider the semantic feature information at different scales when generating panoramic images, thereby improving the reliability of panoramic image generation.

[0059] when Figure 2 When the system shown processes VR data, the specific processing flow of the system is as follows: a VR dual fisheye camera captures at least two images to be stitched, and the initial semantic information is extracted from the images to be stitched by a semantic information extractor; the initial semantic information is encoded into noise-robust encoding information after passing through an encoder; the encoding information is quantized into quantized information with a finite value by a quantizer. The quantized information is modulated by an adjuster, and then transmitted through a wireless channel to the receiving end, where it is demodulated by a demodulator to obtain demodulated information; a decoder is used to decode the received and demodulated information to obtain decoded information; the decoded information is then input into a multi-scale semantic feature extractor to extract multiple groups of semantic feature information of different scales; the semantic feature information and the noise image are input into a diffusion model to generate a panoramic picture of the stitched image.

[0060] In some optional implementations of the present disclosure, the above-mentioned obtaining of a panoramic image corresponding to the image to be stitched based on semantic feature information includes: inputting the semantic feature information and a random noise image into a trained diffusion model to obtain a stitched image output by the diffusion model, and the panoramic image is the image after stitching the images to be stitched.

[0061] In this optional implementation, the diffusion model may use a model of the UNET architecture, and the semantic feature information of four groups of scales is transmitted to the diffusion model and connected to the layers of the corresponding scales of the UNET respectively.

[0062] In this optional implementation, the random noise image is an image obtained by adding random image noise information to the image, the diffusion model is a model for splicing images, inputting semantic feature information reflecting the semantic features of the panoramic image into the diffusion model allows the diffusion model to use the semantic feature information of different scales as input feature conditions of the panoramic image to be generated, and inputting the noise image and the input feature conditions into the diffusion model allows the diffusion model to perform denoising on the noise image; specifically, Figure 2 As shown, the diffusion model receives a noisy image ( Figure 2 The panoramic image Q is outputted based on the semantic feature information output by the multi-scale semantic feature extractor (not shown).

[0063] The method for obtaining a panoramic image provided by this optional implementation inputs semantic feature information and a random noise image into a trained diffusion model, so that the diffusion model can fully learn semantic feature information of different scales, restore the panoramic image of the image to be stitched, and improve the accuracy and reliability of obtaining the panoramic image.

[0064] In some optional implementations of the present disclosure, the above-mentioned receiving signal processing of the semantic extraction information to obtain decoded information includes: demodulating the semantic extraction information to obtain demodulated information; and decoding the demodulated information to obtain decoded information.

[0065] In this optional implementation, demodulation is the process of recovering a message from a modulated signal carrying information. In various information transmission or processing systems, a transmitter modulates a carrier with a message to be transmitted to generate a signal carrying the message. The execution subject on which the semantic communication method runs recovers the transmitted message through demodulation.

[0066] In this optional implementation, the demodulation process varies according to the modulation method, and the demodulation process includes sine wave amplitude demodulation, sine wave angle demodulation, pulse amplitude demodulation, pulse phase demodulation, etc. Sine wave demodulation can be further divided into amplitude demodulation, frequency demodulation and phase demodulation. In addition, there are some variants such as single sideband signal demodulation and residual sideband signal demodulation.

[0067] In this optional implementation, the demodulated information is information obtained after the semantic extraction information is demodulated, and the demodulated information carries the semantic information of the image to be spliced. The decoded information is information obtained after the demodulated information is decoded, and the decoded information carries the semantic information of the image to be spliced.

[0068] In this optional implementation, decoding is the process of converting electrical pulse signals, optical signals, radio waves, etc. into the information, data, etc. represented by them. Decoding is the process of the receiver restoring the received symbols or codes into the original information.

[0069] In this optional implementation, the demodulation method for demodulating the semantic extraction information can be determined based on the modulation method of the transmitting end. The decoding method for decoding the demodulated information can be determined based on the encoding method of the transmitting end. Specifically, the method can be implemented as follows: Figure 2 The demodulator shown in FIG. 1 demodulates the semantic extraction information by Figure 2 The decoder shown implements the decoding process of the demodulated information.

[0070] The method for obtaining decoded information provided by this optional implementation method demodulates the semantic extraction information to obtain demodulated information; and decodes the demodulated information to obtain decoded information, thereby effectively restoring the semantic extraction information to the original semantic information. The operation is simple and the reliability of obtaining the decoded information is improved.

[0071] Optionally, the above-mentioned performing received signal processing on the semantic extraction information to obtain decoded information further includes: performing denoising processing on the decoded information.

[0072] Optionally, the above-mentioned performing received signal processing on the semantic extraction information to obtain decoded information includes: amplifying the decoded information.

[0073] Figure 3 A process 300 according to another embodiment of the semantic communication method of the present disclosure is shown. The semantic communication method comprises the following steps:

[0074] Step 301: receiving at least two images to be stitched.

[0075] In this embodiment, the at least two images to be stitched may be images obtained by photographing an entity using multiple different camera devices in the same scene, and any two adjacent images to be stitched in the at least two images to be stitched may have the same display portion or may not have the same display portion. Figure 2 In the embodiment, at least two images to be stitched D include two images to be stitched, and the two images to be stitched have a same display portion.

[0076] In this embodiment, the execution subject on which the semantic communication method of the present disclosure runs may be the sending end. Figure 2 As shown, the execution body includes a semantic information extractor, an encoder, a quantizer, and a modulator.

[0077] Step 302: input the image to be spliced ​​into a semantic information extractor to obtain initial semantic information output by the semantic information extractor.

[0078] In this embodiment, the semantic information extractor has the ability to extract the semantic features of the image. The initial semantic information is information reflecting the image semantic features of the image to be spliced, and the initial semantic information is information that has not been processed by the sending signal.

[0079] In this embodiment, the semantic information extractor includes: four groups of residual networks and four downsampling convolutional networks. When the image to be spliced ​​is input into the semantic information extractor, the residual network processing can be performed in the semantic information extractor first, and then the downsampling convolutional network processing can be performed to obtain the initial semantic information.

[0080] Step 303: Perform signal processing on the initial semantic information to obtain semantic extraction information.

[0081] In this embodiment, the transmission signal processing refers to signal processing of data (initial semantic information) to be sent to the channel, and the transmission signal processing includes: encoding processing, modulation processing, signal amplification processing, quantization processing, etc.

[0082] In this embodiment, the above step 303 includes: encoding the initial semantic information to obtain the encoded information; modulating the encoded information to obtain the semantic extraction information. The encoding process refers to the process of converting information from one form or format to another form, which can be done by Figure 2 The encoder shown in the figure realizes the encoding of information. Modulation processing refers to the process of processing the information of the signal source and adding it to the carrier to make it suitable for channel transmission. It is a technology that changes the carrier with the signal. The modulation method: According to the form of the modulated signal, it can be divided into analog modulation and digital modulation. Figure 2 The modulator shown performs a modulation process on the information.

[0083] Step 304: Send the semantic extraction information to the receiving end through the channel.

[0084] In this embodiment, the receiving end performs multi-scale semantic feature extraction on the semantic extraction information, and obtains a panoramic image of the image to be stitched based on the extracted semantic feature information.

[0085] In this embodiment, the channel may be a wired channel or a wireless channel. The specific processing of the semantic extraction information by the receiving end may refer to Figure 1 An embodiment of the semantic communication method is shown.

[0086] The semantic communication method provided in the present embodiment receives at least two images to be stitched; inputs the images to be stitched into a semantic information extractor to obtain initial semantic information output by the semantic information extractor; performs transmission signal processing on the initial semantic information to obtain semantic extraction information; and transmits the semantic extraction information to a receiving end through a channel, so that the receiving end performs multi-scale semantic feature extraction on the semantic extraction information, and obtains a panoramic image of the images to be stitched based on the extracted semantic feature information. Thus, when obtaining the panoramic image of at least two images to be stitched, semantic information extraction is performed on the at least two images to be stitched at the transmitting end, and transmission signal processing is performed. The obtained semantic extraction information is sent to the receiving end through a channel. Compared with the traditional image stitching method, the signal obtaining step is saved and the information transmission pressure of the channel is reduced.

[0087] In some optional implementations of the present disclosure, the above-mentioned signal processing of the initial semantic information to obtain semantic extraction information includes: encoding the initial semantic information to obtain encoded information; quantizing the encoded information to obtain quantized information; and modulating the quantized information to obtain semantic extraction information.

[0088] In this optional implementation, quantization processing refers to the process of approximating the continuous value of a signal (or a large number of possible discrete values) to a finite number of discrete values. Quantization processing is mainly used in the conversion from continuous signals to digital signals. Continuous signals become discrete signals after sampling, and discrete signals become digital signals after quantization. In view of the continuity characteristics of image data, quantization processing of encoded information can reduce the amount of encoded information data.

[0089] In this optional implementation, the encoded information is information obtained by encoding the initial semantic information, which can be obtained by Figure 2 The encoder shown in FIG. encodes the initial semantic information. The quantized information is the information obtained after the coded information is quantized, which can be obtained by Figure 2 The quantizer shown in quantizes the coded information to obtain quantized information. Figure 2 The modulator shown performs modulation processing on the quantized information to obtain semantically extracted information.

[0090] In this optional implementation, the encoder includes a group of residual networks and a layer of downsampling convolution, the decoder includes a group of residual networks and a layer of transposed convolution, the quantizer can choose a quantizer that quantizes to integers of -7 to 8, and the quantizer can quantize to integers of the corresponding base according to the size of the channel resources and the required generation quality. The larger the base number used, the more channel resources used and the better the generated quality.

[0091] The method for obtaining semantic extraction information provided by this optional implementation method encodes the initial semantic information to obtain the encoded information; quantizes the encoded information to obtain the quantized information; and adjusts the quantized information to obtain the semantic extraction information, which provides a reliable implementation method for transmitting the initial semantic information in the channel and improves the reliability of the semantic extraction information.

[0092] The present disclosure proposes another embodiment of a semantic communication method, wherein the semantic communication method comprises: a transmitting end obtains semantic extraction information of at least two images to be stitched; the transmitting end sends the semantic extraction information to a receiving end through a channel; the receiving end receives the semantic extraction information; the receiving end performs reception signal processing on the semantic extraction information to obtain decoded information; the receiving end performs multi-scale semantic feature extraction on the decoded information to generate multiple groups of semantic feature information of different scales; the receiving end obtains a panoramic image corresponding to the image to be stitched based on the semantic feature information.

[0093] Specifically, Figure 2 As shown, the transmitter captures at least two images D to be stitched by multiple different cameras, and extracts initial semantic information from the image D to be stitched by a semantic information extractor at the transmitter; the initial semantic information is encoded into coded information with noise robustness after passing through an encoder; the coded information is quantized into quantized information with finite values ​​by a quantizer, and the quantized information is modulated by an adjuster to obtain semantic extraction information, which is transmitted through a wireless channel to reach the receiver, and the receiver demodulates the received semantic extraction information through a demodulator to obtain demodulated information; the receiver decodes the demodulated information using a decoder to obtain decoded information; the receiver then extracts multiple groups of semantic feature information of different scales through a multi-scale semantic feature extractor; the semantic feature information and the noise image are input into a diffusion model to generate a panoramic picture Q of the stitched image.

[0094] The semantic communication method provided by the present disclosure comprises the following steps: a transmitting end obtains semantic extraction information of at least two images to be stitched; the transmitting end sends the semantic extraction information to a receiving end through a channel; the receiving end receives the semantic extraction information; the receiving end processes a received signal of the semantic extraction information to obtain decoded information; the receiving end extracts multi-scale semantic features of the decoded information to generate multiple groups of semantic feature information of different scales; the receiving end obtains a panoramic image corresponding to the images to be stitched based on the semantic feature information, thereby directly extracting semantic extraction information from the images to be stitched taken by a camera at the transmitting end, and directly generating a stitched panoramic image at the receiving end using a diffusion generation model, thereby integrating the panoramic image stitching process and the information transmission process, without directly stitching images at either the transmitting end or the receiving end, thereby reducing the amount of data transmitted during the communication process and the end-to-end transmission efficiency.

[0095] Figure 4A process 400 according to an embodiment of a splicing model training method of the present disclosure is shown. The splicing model training method includes the following steps:

[0096] Step 401: Select image samples from an image sample set.

[0097] In this embodiment, the image sample set may include at least one image sample, the image sample includes at least two sample images and an annotated image after the sample images are spliced, and the annotated image is a panoramic image after the sample images are spliced. The sample images in each image sample may be images of objects in the same scene, and the sample images in each image sample may be the same display part.

[0098] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of the image sample sets involved are carried out after authorization and comply with relevant laws and regulations.

[0099] Step 402: input the image sample into the semantic information extractor in the initial splicing model to obtain initial semantic information.

[0100] In this embodiment, the semantic information extractor is used to extract the semantic information of the sample image in the image sample to obtain initial semantic information, and the initial semantic information is the initial semantic information directly extracted from the image sample.

[0101] In this embodiment, the semantic information extractor may include: four groups of residual networks and four downsampling convolutional networks. The semantic information extractor can effectively obtain the initial semantic information of the sample image in the image sample.

[0102] Step 403: Perform signal processing on the initial semantic information to obtain semantic extraction information.

[0103] In this embodiment, the sending signal processing refers to signal processing of the initial semantic information, wherein the initial semantic information is the data to be transmitted to the channel, and the sending signal processing includes: encoding processing, modulation processing, signal amplification processing, quantization processing, etc.

[0104] Step 404: adding noise to the semantic extraction information, and performing received signal processing on the semantic extraction information to obtain decoded information.

[0105] In this embodiment, the purpose of adding noise to the semantic extraction information is to simulate the transmission of the semantic extraction information in the channel. Specifically, noise with a preset signal-to-noise ratio can be added. When the splicing model is trained end-to-end, the wireless channel part will be replaced by the channel layer. The channel layer will add random noise of corresponding intensity to the transmission information according to the set signal-to-noise ratio to simulate the real channel, so that the entire system has noise robustness. For example, the signal-to-noise ratio used in the splicing model during training is 10dB.

[0106] In this embodiment, the received signal processing refers to performing signal processing on the semantic extraction information with added noise, wherein the received signal processing may include: decoding processing, demodulation processing, signal amplification processing, etc.

[0107] In this embodiment, the received signal processing is performed by the sending end on the sent signal based on the semantic extraction information.

[0108] Step 405 , using the multi-scale semantic feature extractor in the splicing model to perform multi-scale semantic feature extraction on the decoded information, to obtain multiple groups of semantic feature information of different scales.

[0109] In this embodiment, the semantic feature information is information obtained after extracting semantic features of different scales from the decoded information. For example, the semantic feature information of different scales refers to semantic feature images of different scales.

[0110] Step 406: Use the diffusion model in the stitching model to process the semantic feature information and the image samples to obtain a panoramic image corresponding to the image samples.

[0111] In this embodiment, the panoramic image corresponding to the image sample refers to the panoramic image obtained after the stitching model predicts the image sample. Compared with the annotated image of the sample image in the image sample, the panoramic image of the corresponding image sample may have a certain error. By adjusting the loss value of the stitching model during the training process, the error between the panoramic image of the corresponding image sample and the annotated image can be minimized as much as possible, where the loss value is calculated by the loss function.

[0112] In this embodiment, the above step 406 includes: adding Gaussian white noise to the annotated image in the image sample to obtain a noisy image; inputting the noisy image and semantic feature information into the diffusion model in the stitching model to obtain a panoramic image of the corresponding image sample output by the diffusion model.

[0113] Step 407, based on the panoramic image, calculate the loss of the stitching model.

[0114] In this embodiment, the calculation of the loss of the stitching model based on the panoramic image of the image sample includes: substituting the panoramic image and the annotated image of the image sample into the loss function to obtain the loss of the stitching model in each iterative training.

[0115] In this embodiment, an end-to-end training method is used to train the splicing model. p Represents the panoramic image of the corresponding image sample. The stitching model will remove the noise of the panoramic image as the training goal, iterate a preset number of times (such as 200 times), and the loss function of the stitching model is shown in formula (1):

[0116] LOSS=(1-α)d1(x0,d p )+αd2(x0,d p ) (1)

[0117] In formula (1), d i represents the i-th order Euclidean distance, x0 represents the output of the splicing model after a preset number of iterations, i.e., the labeled image, and d p represents the panoramic image of the corresponding image sample, α is a weight factor, and its value can be set based on the training requirements of the stitching model, for example, α is 0.2.

[0118] Step 408, in response to determining based on the loss that the splicing model meets the training completion condition, a trained splicing model is obtained.

[0119] In this embodiment, the execution subject on which the stitching model runs can select image samples from step 401 and execute the training steps from step 402 to step 407 to complete an iterative training of the stitching model. Among them, the selection method and the number of image samples selected from the image sample set are not limited in this application, and the number of iterative training of the stitching model is not limited.

[0120] In this embodiment, the loss of the splicing model can be used to detect whether the splicing model meets the training completion condition. After the splicing model meets the training completion condition, a trained splicing model is obtained.

[0121] In this embodiment, the training completion condition includes: the loss of the splicing model is less than a first loss value threshold. The first loss threshold can be determined based on specific training requirements, for example, the first loss threshold is 0.01.

[0122] Optionally, in this embodiment, in response to the splicing model not meeting the training completion condition, relevant parameters in the splicing model are adjusted so that the network loss value of the splicing model converges, and the above training steps 401-407 are continued to be executed based on the adjusted splicing model.

[0123] The stitching model training method provided in this embodiment selects image samples from an image sample set; inputs the image samples into a semantic information extractor in an initial stitching model to obtain initial semantic information; performs transmission signal processing on the initial semantic information to obtain semantic extraction information; adds noise to the semantic extraction information, and performs receiving signal processing on the semantic extraction information to obtain decoded information; uses a multi-scale semantic feature extractor in the stitching model to extract multi-scale semantic features from the decoded information to obtain multiple groups of semantic feature information of different scales; uses a diffusion model in the stitching model to process the semantic feature information and image samples to obtain a panoramic image of the corresponding image sample; calculates the loss of the stitching model based on the panoramic image; responds to the stitching model based on the loss to determine that the training completion condition is met, and obtains a stitching model that has been trained, thereby, after the semantic features are extracted from the unstitched original image taken by the panoramic camera at the sending end through the semantic information extractor, the multi-scale semantic feature extractor, the diffusion model, and by adding a noise simulation channel, the semantic information is processed at the receiving end and is directly used to guide the diffusion model to generate a stitched panoramic image. End-to-end communication is achieved by integrating the stitching process into the transmission process.

[0124] Optionally, after the stitching model is fully trained, the dual fisheye image is input, and the image input to the diffusion model is replaced with a random Gaussian noise image, and the number of diffusion model iterations is set to 4. After the models of the transmitter and receiver are deployed separately, they are transmitted through a real wireless channel. After 4 iterations at the receiver, the model will generate a panoramic image corresponding to the input dual fisheye image.

[0125] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a semantic communication device, which is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0126] like Figure 5 As shown, the semantic communication device 500 provided in this embodiment includes: an information receiving unit 501, a processing unit 502, a generating unit 503, and an image obtaining unit 504. Among them, the above-mentioned information receiving unit 501 can be configured to receive semantic extraction information of at least two images to be stitched transmitted by a channel. The above-mentioned processing unit 502 can be configured to perform received signal processing on the semantic extraction information to obtain decoded information. The above-mentioned generating unit 503 can be configured to perform multi-scale semantic feature extraction on the decoded information to generate multiple groups of semantic feature information of different scales. The above-mentioned image obtaining unit 504 can be configured to obtain a panoramic image corresponding to the image to be stitched based on the semantic feature information.

[0127] In this embodiment, the specific processing of the information receiving unit 501, the processing unit 502, the generating unit 503, and the image obtaining unit 504 and the technical effects thereof can be referred to in detail. Figure 1 The relevant descriptions of step 101, step 102, step 103, and step 104 in the corresponding embodiment are not repeated here.

[0128] In some optional implementations of this embodiment, the above-mentioned generation unit 503 is configured to: input the decoded information into a pre-trained multi-scale semantic feature extractor to obtain multiple groups of semantic feature information of different scales output by the multi-scale conditional extractor, the multi-scale semantic feature extractor includes at least two feature extractors, and each feature extractor includes: a residual network, a transposed convolutional network and a self-attention layer.

[0129] In some optional implementations of this embodiment, the above-mentioned image acquisition unit 504 is configured to: input semantic feature information and random noise images into a trained diffusion model to obtain a stitched image output by the diffusion model, and the panoramic image is the image after stitching the images to be stitched.

[0130] In some optional implementations of this embodiment, the processing unit 502 is configured to: demodulate the semantic extraction information to obtain demodulated information; and decode the demodulated information to obtain decoded information.

[0131] The semantic communication method provided by the embodiment of the present disclosure is as follows: first, the information receiving unit 501 receives the semantic extraction information of at least two images to be stitched transmitted by the channel; secondly, the processing unit 502 performs received signal processing on the semantic extraction information to obtain decoded information; thirdly, the generating unit 503 performs multi-scale semantic feature extraction on the decoded information to generate multiple groups of semantic feature information of different scales; finally, the image obtaining unit 504 obtains a panoramic image corresponding to the image to be stitched based on the semantic feature information. Thus, the semantic extraction information is processed at the receiving end to generate a panoramic image of at least two images to be stitched, thereby placing the stitching of the images at the receiving end, and the channel only needs to transmit the semantic extraction information, which reduces the pressure of channel data transmission and improves the transmission efficiency of the panoramic image data; multiple groups of semantic feature information of different scales are used to obtain the panoramic image, which improves the reliability of obtaining the panoramic image.

[0132] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0133] Further references Figure 6As an implementation of the methods shown in the above figures, the present disclosure provides another embodiment of a semantic communication device, which is similar to Figure 3 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0134] like Figure 6 As shown, the semantic communication device 600 provided in this embodiment includes: an image receiving unit 601, an input unit 602, an information obtaining unit 603, and a sending unit 604. Among them, the above-mentioned image receiving unit 601 can be configured to receive at least two images to be stitched. The above-mentioned input unit 602 can be configured to input the image to be stitched into a semantic information extractor to obtain initial semantic information output by the semantic information extractor. The above-mentioned information obtaining unit 603 can be configured to perform sending signal processing on the initial semantic information to obtain semantic extraction information. The above-mentioned sending unit 604 can be configured to send the semantic extraction information to the receiving end through the channel, so that the receiving end performs multi-scale semantic feature extraction on the semantic extraction information, and obtains a panoramic image of the image to be stitched based on the extracted semantic feature information.

[0135] In this embodiment, the specific processing of the image receiving unit 601, the input unit 602, the information obtaining unit 603, and the sending unit 604 and the technical effects thereof can be referred to respectively. Figure 3 The relevant descriptions of step 301, step 302, step 303, and step 304 in the corresponding embodiment are not repeated here.

[0136] In some optional implementations of this embodiment, the above-mentioned information acquisition unit 603 is configured to: encode the initial semantic information to obtain the encoded information; quantize the encoded information to obtain the quantized information; modulate the quantized information to obtain the semantic extraction information.

[0137] Further references Figure 7 As an implementation of the methods shown in the above figures, the present disclosure provides another embodiment of a semantic communication device, which is similar to Figure 4 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0138] like Figure 7As shown, the semantic communication device 700 provided in this embodiment includes: a selection unit 701, a training unit 702, a calculation unit 703, and a model acquisition unit 704. Among them, the above-mentioned selection unit 701 can be configured to select image samples from the image sample set. The above-mentioned training unit 702 can be configured to input the image sample into the semantic information extractor in the initial splicing model to obtain the initial semantic information; perform sending signal processing on the initial semantic information to obtain semantic extraction information; add noise to the semantic extraction information, and perform receiving signal processing on the semantic extraction information to obtain decoding information; use the multi-scale semantic feature extractor in the splicing model to perform multi-scale semantic feature extraction on the decoding information to obtain multiple groups of semantic feature information of different scales; use the diffusion model in the splicing model to process the semantic feature information and image samples to obtain a panoramic image of the corresponding image sample. The above-mentioned calculation unit 703 can be configured to calculate the loss of the splicing model based on the panoramic image. The above-mentioned model acquisition unit 704 can be configured to respond to the determination based on the loss that the splicing model meets the training completion condition, and obtain a splicing model that has been trained.

[0139] In this embodiment, the semantic communication device 700 includes the selection unit 701, the training unit 702, the calculation unit 703, and the model obtaining unit 704. The specific processing and technical effects thereof can be referred to in Figure 4 The relevant descriptions of step 401, step 402, step 403, and step 404 in the corresponding embodiment are not repeated here.

[0140] Further references Figure 8 As an implementation of the methods shown in the above figures, the present disclosure provides another embodiment of a semantic communication system. Figure 1 , Figure 3 Corresponding to the method embodiment shown, the system can be specifically applied to various electronic devices.

[0141] like Figure 8 As shown, the semantic communication system 800 provided in this embodiment includes: a transmitting end 801, a channel 802 and a receiving end 803. The transmitting end 801 is used to receive at least two images to be spliced; input the images to be spliced ​​into a semantic information extractor to obtain initial semantic information output by the semantic information extractor; perform transmission signal processing on the initial semantic information to obtain semantic extraction information; and send the semantic extraction information to the receiving end through the channel 802.

[0142] The above-mentioned receiving end 803 is used to receive semantic extraction information of at least two images to be stitched transmitted by channel 802; perform reception signal processing on the semantic extraction information to obtain decoded information; perform multi-scale semantic feature extraction on the decoded information to generate multiple groups of semantic feature information of different scales; based on the semantic feature information, obtain a panoramic image corresponding to the image to be stitched.

[0143] In this embodiment, the sending end 801 and the receiving end 803 can be clients or servers.

[0144] In this embodiment, in the semantic communication system 800, the specific processing of the sending end 801 and the technical effects thereof can be referred to in Figure 1 The corresponding embodiments of step 101, step 102, step 103, step 104; the specific processing of the receiving end 803 and the technical effects thereof can be referred to respectively. Figure 3 The relevant descriptions of step 301, step 302, step 303, and step 304 in the corresponding embodiment are not repeated here.

[0145] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0146] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0147] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0148] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0149] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as the semantic communication method. For example, in some embodiments, the semantic communication method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the semantic communication method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the semantic communication method in any other appropriate manner (e.g., by means of firmware).

[0150] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0151] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable semantic communication device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0152] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0153] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0154] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an information server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0155] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0156] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0157] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A semantic communication method, the method comprising: receiving semantic extraction information of at least two images to be stitched transmitted through a channel; Performing received signal processing on the semantic extraction information to obtain decoded information; Performing multi-scale semantic feature extraction on the decoded information to generate multiple groups of semantic feature information of different scales; Based on the semantic feature information, a panoramic image corresponding to the image to be stitched is obtained.

2. According to the method of claim 1, the step of extracting multi-scale semantic features from the decoded information to generate multiple groups of semantic feature information at different scales comprises: The decoded information is input into a pre-trained multi-scale semantic feature extractor to obtain multiple groups of semantic feature information of different scales output by the multi-scale conditional extractor, wherein the multi-scale semantic feature extractor includes at least two feature extractors, each of which includes: a residual network, a transposed convolutional network, and a self-attention layer.

3. According to the method of claim 1, obtaining a panoramic image corresponding to the image to be stitched based on the semantic feature information comprises: The semantic feature information and the random noise image are input into a trained diffusion model to obtain a spliced ​​image output by the diffusion model, and the panoramic image is an image after splicing the images to be spliced.

4. The method according to claim 1, wherein: The receiving signal processing of the semantic extraction information to obtain the decoded information comprises: Demodulating the semantic extraction information to obtain demodulated information; The demodulated information is decoded to obtain decoded information.

5. A semantic communication method, the method comprising: Receiving at least two images to be stitched; Inputting the image to be spliced ​​into a semantic information extractor to obtain initial semantic information output by the semantic information extractor; Performing signal processing on the initial semantic information to obtain semantic extraction information; The semantic extraction information is sent to a receiving end through a channel, so that the receiving end performs multi-scale semantic feature extraction on the semantic extraction information, and obtains a panoramic image of the image to be stitched based on the extracted semantic feature information.

6. According to the method of claim 5, the step of performing signal processing on the initial semantic information to obtain semantic extraction information comprises: Performing encoding processing on the initial semantic information to obtain encoded information; quantizing the encoded information to obtain quantized information; The quantized information is modulated to obtain semantic extraction information.

7. A semantic communication method, the method comprising: The sending end obtains semantic extraction information of at least two images to be stitched; The sending end sends the semantic extraction information to the receiving end through a channel; The receiving end receives the semantic extraction information; The receiving end performs received signal processing on the semantic extraction information to obtain decoded information; The receiving end performs multi-scale semantic feature extraction on the decoded information to generate multiple groups of semantic feature information of different scales; The receiving end obtains a panoramic image corresponding to the image to be stitched based on the semantic feature information.

8. A splicing model training method, the method comprising: Selecting an image sample from an image sample set; Inputting the image sample into a semantic information extractor in an initial splicing model to obtain initial semantic information; The initial semantic information is subjected to a sending signal processing to obtain semantic extraction information; noise is added to the semantic extraction information, and the semantic extraction information is subjected to a receiving signal processing to obtain decoded information; a multi-scale semantic feature extractor in the splicing model is used to perform multi-scale semantic feature extraction on the decoded information to obtain multiple groups of semantic feature information of different scales; the semantic feature information and the image sample are processed using a diffusion model in the splicing model to obtain a panoramic image corresponding to the image sample; Based on the panoramic image, calculating the loss of the stitching model; In response, it is determined based on the loss that the splicing model meets the training completion condition, and a trained splicing model is obtained.

9. A semantic communication device, comprising: An information receiving unit, configured to receive semantic extraction information of at least two images to be stitched transmitted through a channel; a processing unit configured to perform received signal processing on the semantic extraction information to obtain decoded information; A generating unit, configured to perform multi-scale semantic feature extraction on the decoded information to generate multiple groups of semantic feature information of different scales; The image obtaining unit is configured to obtain a panoramic image corresponding to the image to be stitched based on the semantic feature information.

10. A semantic communication device, comprising: An image receiving unit, configured to receive at least two images to be stitched; An input unit, configured to input the image to be spliced ​​into a semantic information extractor to obtain initial semantic information output by the semantic information extractor; an information obtaining unit, configured to perform a sending signal processing on the initial semantic information to obtain semantic extraction information; The sending unit is configured to send the semantic extraction information to the receiving end through a channel, so that the receiving end performs multi-scale semantic feature extraction on the semantic extraction information, and obtains the panoramic image of the image to be stitched based on the extracted semantic feature information.

11. A splicing model training device, the device comprising: A selection unit, configured to select an image sample from an image sample set; The training unit is configured to input the image sample into a semantic information extractor in an initial stitching model to obtain initial semantic information; perform sending signal processing on the initial semantic information to obtain semantic extraction information; add noise to the semantic extraction information, and perform receiving signal processing on the semantic extraction information to obtain decoding information; use the multi-scale semantic feature extractor in the stitching model to perform multi-scale semantic feature extraction on the decoding information to obtain multiple groups of semantic feature information of different scales; use the diffusion model in the stitching model to process the semantic feature information and the image sample to obtain a panoramic image corresponding to the image sample; A calculation unit configured to calculate the loss of the stitching model based on the panoramic image; The model obtaining unit is configured to obtain a trained splicing model in response to determining that the splicing model meets the training completion condition based on the loss.

12. A semantic communication system, comprising: The sender, the channel, and the receiver; The sending end is used to receive at least two images to be stitched; Inputting the image to be spliced ​​into a semantic information extractor to obtain initial semantic information output by the semantic information extractor; performing transmission signal processing on the initial semantic information to obtain semantic extraction information; and sending the semantic extraction information to a receiving end through the channel; The receiving end is used to receive semantic extraction information of at least two images to be spliced ​​transmitted by the channel; perform received signal processing on the semantic extraction information to obtain decoding information; Performing multi-scale semantic feature extraction on the decoded information to generate multiple groups of semantic feature information of different scales; Based on the semantic feature information, a panoramic image corresponding to the image to be stitched is obtained.

13. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

14. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable the computer to execute the method according to any one of claims 1 to 8.

Images