Method, system, device and medium for semantic communication based on image-text associated knowledge network

Abstract

The application discloses an image semantic communication method, system, device and medium based on a picture-text associated knowledge network, and steps are as follows: a picture-text associated knowledge network composed of an image preprocessing and coding subnetwork and an image semantic text extraction subnetwork is constructed; an image vector is obtained by preprocessing a required transmission image through the image preprocessing and coding subnetwork; an image semantic text is obtained by performing feature extraction on the image vector through the image semantic text extraction subnetwork; the image semantic text is sent from a sending end to a receiving end; and inference of a downstream task is performed based on the image semantic text received by the receiving end. The image to be sent is extracted into corresponding semantic texts, so that the communication resources required for sending the image are significantly reduced, and the extracted semantic texts are in a text form, so that the existing digital communication network adaptability, interpretability and downstream multi-task adaptability of the feature vectors of the prior art are better.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of semantic communication, in particular to a semantic communication method and system based on a graph-text associated knowledge network, a computer device and a storage medium. BACKGROUND

[0002] Modern wireless communication has developed from 1G to 5G, and future 6G, with higher and higher requirements for communication rate. Limited by limited communication resources, in order to meet the demand of high rate and low delay communication, it is of great practical significance to improve the efficiency of wireless communication. Semantic communication refers to a communication technology that focuses on the semantic integrity of the receiving end rather than the accuracy of the traditional bit information. Specifically, semantic communication uses a neural network to extract semantic features of information at the sending end for transmission, and uses a neural network to recover information or directly perform downstream task reasoning at the receiving end according to the received semantic features. Since the communication resources required for transmitting semantic features are generally less than those required for transmitting original information, semantic communication can significantly improve communication efficiency and is a promising future communication technology. The ability of the receiving end model to extract semantic features of information and the characteristics of the semantic features themselves directly determine the effectiveness of information recovery and downstream task reasoning at the receiving end, and determine the performance of the entire semantic communication system. In traditional semantic communication, taking wireless image communication as an example, the sending end model usually uses a convolutional neural network to extract a feature vector of the image as the semantic feature of the image for transmission, and the receiving end performs downstream task reasoning related to the image, such as image text question answering, image description, image classification, etc., after recovering the image according to the received image feature vector. However, such image feature vectors are floating-point vectors and are not suitable for direct transmission through existing digital communication. At the same time, since the image feature vector is a feature vector extracted by a neural network, it has the property of being uninterpretable, and neither humans nor machines can directly evaluate whether the extracted image feature vector contains the semantics of the image sent by the sending end. Only a semantic reconstruction network jointly trained with the sending end semantic extraction neural network can analyze such image feature vectors, and image feature vectors can only be used for the downstream task specified during training. Therefore, traditional semantic communication cannot adapt to multiple downstream tasks. As a kind of information with less redundancy than images, text can be more easily judged by humans and machines as to whether it contains the content of the original image. Text can be seamlessly used for multiple downstream tasks such as image caption generation, image text question answering, etc., rather than being limited to a specified task during training, and has the properties of being interpretable and universal, making it more suitable than image semantic features as a kind of image semantic for image communication. Therefore, there is an urgent need to construct a wireless image communication technology scheme based on a graph-text associated knowledge network using text as a semantic medium. SUMMARY

[0003] The purpose of the present invention is to solve the shortcomings of existing wireless image semantic communication technology, such as the use of image feature vectors as semantics for transmission, such as uninterpretable semantics and inability to adapt to downstream multi-tasks. A method, system, computer equipment and storable medium for image semantic communication based on a picture-text association knowledge network are provided. The semantic text of the image is extracted through the picture-text association knowledge network for communication, which not only significantly reduces the communication resources required for sending images, but also has better adaptability, interpretability and downstream multi-task adaptability to existing digital communication networks.

[0004] The first object of the present invention is to disclose a semantic communication method based on a picture-text association knowledge network, the semantic communication method comprising the following steps:

[0005] S1. Construct an image-text association knowledge network consisting of the image preprocessing and encoding subnetwork A and the image semantic text extraction subnetwork B connected sequentially;

[0006] S2, preprocessing the image to be transmitted through the image preprocessing and encoding sub-network to obtain an image vector;

[0007] S3, extract features from the image vector through the image semantic text extraction sub-network to obtain image semantic text;

[0008] S4, sending the image semantic text from the sending end to the receiving end via wireless communication;

[0009] S5. Perform reasoning on downstream tasks based on the image semantic text received at the receiving end.

[0010] Furthermore, the image preprocessing and encoding sub-network A includes an image preprocessing network P i and image encoder V, the image to be transmitted X passes through the image preprocessing network P i After scaling, embed the image vector Then the image vector The image feature vector is extracted by the image encoder V in:

[0011] Image preprocessing network P i It consists of a segmentation layer and an embedding layer. The input is the image to be transmitted X∈3×e×h. The image width w and height h are determined by the image. The output is an image vector N p is the image vector Length, L ei Set image embedding dimension; image preprocessing network P i The segmentation layer in the image segments the image to obtain smaller image blocks, which are convenient for the embedding layer to process; the image preprocessing network P iThe embedding layer in the image block embeds the image vector to facilitate the neural network to extract features. Image preprocessing network P i The structure is a commonly used prefix module design for image feature extraction.

[0012] The image encoder V is composed of N v It consists of a residual self-attention network and a pooling layer connected in sequence, and the input is an image vector The output is the extracted image feature vector L i The dimension of the image feature vector is set; the residual self-attention network can accurately extract the most critical features of the image using the self-attention mechanism, and the pooling layer is used to change the size of the features extracted by the residual self-attention network to facilitate reasoning with text features. v The structure can effectively extract the features that best represent the image content.

[0013] Furthermore, the image semantic text extraction sub-network B includes a text preprocessing network P t , text encoder E and text decoder D, used to transform the image feature vector Convert image to semantic text T∈n through loop iteration out ×1,n out The length of the final output image semantic text depends on the image feature vector

[0014] The text preprocessing network P t It consists of an embedding layer, the input is the intermediate image semantic text T0∈n×1, and the output is a text vector n is the current loop number, n∈[1,n out ],n out To output the text length, the image feature vector Decision, L et is the set text embedding dimension; this preprocessing network P t The embedding layer in [1] embeds the semantic text to obtain a text vector, which is convenient for neural network to extract features. It is a commonly used prefix module design for text feature extraction.

[0015] The text encoder E consists of N e The residual self-attention network is sequentially composed, and the input is the text vector Output is text feature vector L t is the set text feature vector dimension; the residual self-attention network uses the self-attention mechanism to accurately extract the most critical features in the text and effectively extract the features that best represent the text content.

[0016] The text decoder D is sequentially and alternately composed of N d residual self-attention networks and N d residual multi-modal attention networks, and a mapping layer is cascaded at the last stage, the input is a text feature vector and an image feature vector , and the output is an nth image semantic word y n . The residual self-attention network can accurately extract the most critical features in the text by using the self-attention mechanism; the residual multi-modal attention network can simultaneously abstract the features that can best represent the current image-text input according to the input image features and text features by using the cross-modal attention mechanism; the alternating use of the two attention networks can gradually focus on the most critical image information, thereby reasoning out the most relevant image semantic words, which is an efficient image-text association network structure.

[0017] Further, the step S2 is as follows:

[0018] S2.1, an image preprocessing network P i scales the to-be-transmitted image X of the sending end into a first intermediate image X0∈3×L0×L0, which has an image width and height of L0pixels, and cuts it into N p second intermediate images I0∈N p ×(3×L p ×L p ), each of which has an image width and height of L p pixels, wherein N p =(L0 / L p )*(L0 / L p ), and the second intermediate image I0is mapped into a high-dimensional image vector The above operation converts an image of any size into an image vector that can be processed by the model, which is conducive to the adaptation of the model to any image input.

[0019] S2.2, an image encoder V performs self-attention extraction on the input image vector to obtain an intermediate image feature vector The residual self-attention network in the image encoder V does not change the size of the input vector;

[0020] The self-attention extraction process is as follows:

[0021] The first residual self-attention network performs feature extraction on the input image vector to obtain a first feature map The j∈[2,N i ]th residual self-attention network performs feature extraction on the (j-1)th feature map to obtain a jth feature map The N i The feature map is taken as an intermediate image feature vector The above self-attention extraction process can effectively extract the feature vector that best represents the image content through feature extraction of multiple residual self-attention networks in series, which is beneficial to subsequent image semantic text generation.

[0022] The pooling layer of the image encoder V extracts the intermediate image feature vector to obtain an image feature vector

[0023] Further, the step S3 is as follows:

[0024] S3.1, set the intermediate image semantic text T0 as an n list with a length of n, T0∈n×1;

[0025] S3.2, the text preprocessing network P t The intermediate image semantic text T0 is embedded to obtain a text vector The above operation converts the text into a text vector that can be processed by the model, which is beneficial to feature extraction by the model.

[0026] S3.3, the text encoder E extracts the text vector to obtain a text feature vector The self-attention extraction is as follows:

[0027] The first residual self-attention network extracts the input text vector to obtain the first feature text The k∈[2,N e ]th residual self-attention network extracts the (k-1)th feature text to obtain the kth feature text The N e The feature text is taken as a text feature vector The above self-attention extraction process can effectively extract the feature vector that best represents the content of the intermediate image semantic text through feature extraction of multiple residual self-attention networks in series, which is beneficial to subsequent image semantic text generation.

[0028] S3.4, the text decoder D extracts the text feature vector and the image feature vector to obtain the nth image semantic word y n The multi-modal attention extraction is as follows:

[0029] The first residual self-attention network extracts the input text feature vector Perform feature extraction to obtain the first intermediate feature vector

[0030] The first residual multimodal attention network takes the first intermediate feature vector of the input and image feature vector Perform feature extraction to obtain the first intermediate mixed feature vector

[0031] The first l∈[2,N d ]The input of the residual self-attention network is the l-1th intermediate mixed feature vector The output is the lth intermediate eigenvector

[0032] The first l∈[2,N d ]The input of the residual multimodal attention network is the l-1th intermediate feature vector The output is the lth intermediate mixed feature vector

[0033] In the multimodal attention extraction process described above, the residual self-attention network and the residual multimodal attention network are used alternately. The residual self-attention network enables the model to focus on the most important features in the text, while the residual multimodal attention network, by introducing image feature vectors, effectively combines image and text information, guiding the model to generate text related to the image content. This helps the model's final generated image semantic text conform to human perception of image content, achieving the goal of transmitting only text while allowing the receiver to understand the sender's image content.

[0034] The last cascade mapping layer of the text decoder D converts the Nth d The intermediate mixed feature vector is mapped to obtain the nth image semantic word y n ;

[0035] S3.5, the nth image semantic word y n Spliced ​​to the end of the intermediate image semantic text T0 until the generated nth image semantic word y n The special termination symbol s is set e , and the obtained intermediate image semantic text T0 is output as the image semantic text T. This method of cyclically generating image semantic words enables the model to infer more reasonable new image semantic words based on the image semantic words generated in the previous cycle and image features, thereby improving the accuracy of the final generated image semantic text.

[0036] Furthermore, the process of step S4 is as follows:

[0037] S4.1, encoding the image semantic text T into Unicode and converting it into binary code M at the sending end; the above-mentioned image semantic text encoding converts the characters in the text into binary code for easy modulation and transmission. S4.1 embodies the advantage of the present method of converting the image to be sent into image semantic text: compared with the traditional semantic communication which converts the image into a feature vector that can only be analyzed by a neural network, the image semantic text obtained by using the neural network structure and feature extraction process in the present method is a kind of information that can be directly understood by humans, and it can be judged whether the semantic extraction is correct before transmission.

[0038] S4.2, modulating the binary code M at the sending end to obtain a wireless signal S and transmitting it to the receiving end through a wireless channel; the above-mentioned modulation and wireless communication method is a common wireless image communication transmission process.

[0039] S4.3, demodulating the received wireless signal S to obtain the binary code M at the receiving end; the above-mentioned demodulation and wireless communication method is a common wireless image communication receiving process. S4.2 and S4.3 embody the advantage of the present method of converting the image to be sent into image semantic text: compared with the traditional semantic communication which extracts a feature vector that is a continuous symbol and can only be sent directly as analog information, the image semantic text obtained by using the neural network structure and feature extraction process in the present method is a common information format, and existing digital communication technology can easily achieve lossless communication without the need to redesign the wireless communication scheme, reducing the cost of semantic communication deployment.

[0040] S4.4, decoding the binary code M to obtain the image semantic text T at the receiving end. The above-mentioned decoding process restores the received binary code back to the image semantic text, which is convenient for the receiving end model to perform downstream task inference. S4.4 embodies the advantage of the present method of converting the image to be sent into image semantic text: the receiving end can still parse and obtain certain information by using the image semantic text obtained by using the neural network structure and feature extraction process in the present method without deploying any neural network, which is beneficial to ensure the completion of downstream tasks in the case of communication resource shortage or receiving end neural network inference calculation resource shortage.

[0041] Further, the step S5 process is as follows:

[0042] At the receiving end, the received image semantic text T is subjected to inference of a plurality of visual text category sub-tasks, including image caption generation and image text question and answer tasks, wherein,

[0043] The received image semantic text t is an image caption, which is directly used as the output of the image caption generation subtask; the image caption generation is a direct image understanding manner, as a subtask of the receiving end, in the scene of communication resource shortage, the demand of some receiving end users can be met without the sending end transmitting the complete image.

[0044] In the image text question answering subtask, the pre-trained large language model L of the receiving end is connected to the received image semantic text T and the question Q related to the image content to obtain an intermediate feature vector, and the intermediate feature vector is subjected to text reasoning to obtain the answer S corresponding to the image X to be transmitted by the sending end and the question Q, and l is the word length of the answer itself. Thanks to the image semantic text generated by the method, the image processing step in the image text question answering subtask of the receiving end is simplified, the neural network reasoning calculation resources of the receiving end are saved, and the completion of the subtask is still ensured.

[0045] A second object of the application is to disclose a semantic communication system based on an image-text association knowledge network, for implementing the above-mentioned semantic communication method based on an image-text association knowledge network, the image semantic communication system comprising:

[0046] The image-text association knowledge network module is used to construct an image-text association knowledge network composed of an image preprocessing and coding subnetwork A and an image semantic text extraction subnetwork B connected in sequence.

[0047] The image feature extraction module is used to preprocess the image vector by the image preprocessing and coding subnetwork.

[0048] The image semantic text extraction module is used to extract the image semantic text by the image semantic text extraction subnetwork.

[0049] The wireless communication module is used to transmit the image semantic text from the sending end to the receiving end through wireless communication.

[0050] The downstream task reasoning module is used to perform the reasoning of the downstream task based on the image semantic text received by the receiving end.

[0051] A third object of the application is to provide a computer device comprising a processor and a memory for storing a processor-executable program, wherein the processor implements the above-mentioned semantic communication method based on an image-text association knowledge network when executing the program stored in the memory.

[0052] A fourth object of the application is to provide a storage medium storing a program, wherein the program is executed by a processor to implement the above-mentioned semantic communication method based on an image-text association knowledge network.

[0053] The present application has the following advantages and effects relative to the prior art:

[0054] 1) The method of the present application uses residual self-attention network to extract image feature vectors of images, and then uses the residual self-attention network and the residual multi-modal attention network alternately constructed to generate image semantic texts corresponding to the images for transmission. Compared with the image semantics defined by the traditional semantic communication, i.e. image feature vectors, the present application has more intuitive and explainable characteristics, and can better determine whether the semantics can replace the images for transmission.

[0055] 2) The present application converts images into semantic-related texts for transmission, and then performs downstream task reasoning according to the received semantic texts at the receiving end. Under the premise that the receiving end receives sufficient image semantic information and completes the downstream task, the communication resources required for transmitting images are significantly reduced.

[0056] 3) The present application converts images into semantic-related texts for transmission. Compared with the traditional image feature vectors which can only be used for training of specified downstream tasks, the image semantic texts can be directly used as the inference input of various downstream tasks such as image caption generation and image text question answering, and have downstream multi-task adaptability. BRIEF DESCRIPTION OF DRAWINGS

[0057] The accompanying drawings, which are included to provide a further understanding of the present application and constitute a part of this application, illustrate certain illustrative embodiments of the present application and are used to explain the present application, but do not limit the present application. In the drawings:

[0058] Figure 1 is a flow chart of an image semantic communication method based on a graph-text associated knowledge network disclosed by the present application;

[0059] Figure 2 is a structural block diagram of an image semantic communication method based on a graph-text associated knowledge network disclosed by the present application;

[0060] Figure 3 is an example graph used in embodiment 1;

[0061] Figure 4 is a communication effect schematic diagram of the image semantic communication based on a graph-text associated knowledge network of the present application in embodiment 1 of the present application;

[0062] Figure 5 is a structural block diagram of an image semantic communication system in embodiment 2 of the present application;

[0063] Figure 6 is a structural block diagram of a computer device in embodiment 3 of the present application. DETAILED DESCRIPTION

[0064] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0065] Embodiment 1

[0066] Reference Figure 1 , Figure 1 is a flowchart of a semantic communication method based on a graph-text association knowledge network in Embodiment 1. As shown in Figure 1 , the method is a semantic communication method for image communication in a case of limited communication resources, mainly based on a pre-trained graph-text association knowledge network of a sending end to extract image semantic text of a transmission image, and the communication content is converted from an entire image to an image semantic text, which greatly reduces the required communication resources and meets the requirement that a receiving end can still complete a downstream task in a case of limited communication resources.

[0067] The implementation steps of the semantic communication method based on the graph-text association knowledge network disclosed in the embodiment will be specifically described below with reference to Figure 1 :

[0068] Step S1, constructing a graph-text association knowledge network composed of an image preprocessing and encoding subnetwork A and an image semantic text extraction subnetwork B connected in sequence, and the specific network architecture is referred to Figure 2 , the method comprises the following steps:

[0069] constructing an image preprocessing and encoding subnetwork A composed of an image preprocessing network P i and an image encoder V in sequence, for converting a to-be-transmitted image X into an image feature vector , wherein:

[0070] The image preprocessing network P i is composed of a segmentation layer and an embedding layer, the input is a to-be-transmitted image X e 3 x w x h, the image width w and the image height h are determined by the image and are not limited, so the method of the embodiment is applicable to transmission of images of various sizes, and the output is an image vector N p is the length of the image vector , which is set to 256, and L ei is the image embedding dimension, which is set to 1024; the sample diagram used in the embodiment is shown in Figure 3 .

[0071] The image encoder V is composed of 24 residual self-attention networks and 1 pooling layer connected in sequence, and the input is the image vector The output is the extracted image feature vector L i The image feature vector dimension is set to 768;

[0072] The text preprocessing network P is constructed t , the text encoder E and the text decoder D are sequentially composed of the image semantic text extraction subnetwork B, which is used to convert the image feature vector to the image semantic text T∈n out ×1, n out The length of the final output image semantic text depends on the image feature vector This adaptive transmission of image semantic text length according to the complexity of image content helps to save communication resources; wherein:

[0073] The text preprocessing network P t is composed of an embedding layer, and the input is the intermediate image semantic text T0∈n×1, and the output is the text vector n is the current number of iterations, n∈[1,n out ], L et is the set text embedding dimension; L et is the text embedding dimension, which is set to 768;

[0074] The text encoder E is sequentially composed of 12 residual self-attention networks, and the input is the text vector The output is the text feature vector L t The text feature vector dimension is set to 768;

[0075] The text decoder D is sequentially composed of 12 residual self-attention networks and 12 residual multimodal attention networks, and a mapping layer is cascaded at the last stage, and the input is the text feature vector and the image feature vector The output is the nth image semantic word y n .

[0076] The image-text association knowledge network constructed in this embodiment will be trained in a large amount of image text data to obtain a pre-trained model file for inference. This embodiment only defines the network structure, and the training method is not limited.

[0077] Step S2, the image is preprocessed by the image preprocessing and encoding subnetwork to obtain an image vector, and the specific process includes:

[0078] S2.1, the image preprocessing network P iThe sending end to be transmitted image X is scaled into a first intermediate image X0∈3×L0×L0 with image width and height of L0 pixels, and is divided into N p p second intermediate images I0∈N p ×(3×L p ×L p ), wherein L0=224, L p =14, The second intermediate image I0 is mapped into a high-dimensional image vector by a linear layer

[0079] S2.2, the image encoder V performs self-attention extraction on the input image vector to obtain an intermediate image feature vector Since the image contains a large amount of spatial information and features, the relationship between these features is crucial for understanding the entire image and subsequent semantic text generation, so the image vector obtained according to the preprocessing Using the residual self-attention network in series in the image encoder V, the system model of the application can adaptively focus on the most important features of the image through the self-attention mechanism, and retain the information flow through the residual connection.

[0080] The self-attention extraction process is as follows:

[0081] The first residual self-attention network performs feature extraction on the input image vector to obtain the first feature map The j∈[2,24]th residual self-attention network performs feature extraction on the (j-1)th feature map to obtain the jth feature map The 24th feature map is taken as the intermediate image feature vector

[0082] The residual self-attention network does not change the size of the input vector, but the intermediate image feature vector contains a large number of key features of the image.

[0083] The pooling layer of the image encoder V performs dimension transformation on the intermediate image feature vector to obtain the image feature vector The dimension is transformed into a fixed size, which is convenient for the use of the subsequent image semantic text extraction subnetwork.

[0084] Step S3, the image semantic text is obtained by performing feature extraction on the image vector through the image semantic text extraction subnetwork, and the specific process includes:

[0085] Loop: n from 1 to n out ​

[0086] S3.1, set the intermediate image semantic text T0 as an n list with a length of n, T0 e n x 1.

[0087] S3.2, text preprocessing network P t Embedding the intermediate image semantic text T0 to get the text vector

[0088] S3.3, the text encoder E extracts the text vector by self-attention to get the text feature vector Since the text contains rich semantic information and structural features, these features and their mutual relationships are crucial for understanding the entire text content and subsequent image semantic generation tasks. Therefore, according to the text vector The system model of the application utilizes a series of residual self-attention networks, which can adaptively focus on the most critical information in the text through the self-attention mechanism. The self-attention mechanism enables the model to capture long-distance dependencies and understand the complex interactions between words, thereby enhancing the ability of semantic expression. At the same time, the use of residual connection ensures the effective flow of information in the model, avoiding the gradient vanishing problem that may occur in deep networks, ensuring the stability and efficiency of the model when processing complex text structures.

[0089] The self-attention extraction process is as follows:

[0090] The first residual self-attention network extracts the input text vector to get the first feature text The k e [2, 12] residual self-attention network extracts the k-1 feature text to get the k feature text The 12th feature text is taken as the text feature vector output.

[0091] The residual self-attention network does not change the size of the input vector, and at this time the text feature vector contains a large number of key features of the text, which is very suitable for combining with the image feature vector to predict and generate new image semantic words.

[0092] S3.4, the text decoder D extracts the text feature vector and the image feature vector by multi-modal attention to get the n image semantic word y n The residual self-attention network in the text decoder D is responsible for extracting the features of the semantic text, and the residual multi-modal attention network is responsible for combining the text feature vector and image feature vector to extract higher level features. By alternating the use of residual self-attention network and residual multi-modal attention network, the model is able to extract the most important and rich features, and the final mapping layer maps the features to the newly generated nth image semantic word y n .

[0093] The multi-modal attention extraction process is as follows:

[0094] The first residual self-attention network extracts features from the input text feature vector to obtain the first intermediate feature vector

[0095] The first residual multi-modal attention network extracts features from the input first intermediate feature vector and image feature vector to obtain the first intermediate mixed feature vector

[0096] The input of the l-th residual self-attention network (l ∈ [2, 12]) is the (l-1)-th intermediate mixed feature vector and the output is the l-th intermediate feature vector

[0097] The input of the l-th residual multi-modal attention network (l ∈ [2, 12]) is the (l-1)-th intermediate feature vector and the output is the l-th intermediate mixed feature vector

[0098] The final mapping layer of the text decoder D maps the 12th intermediate mixed feature vector to obtain the nth image semantic word y n .

[0099] S3.5, the nth image semantic word y n is spliced to the tail of the intermediate image semantic text T0 for the next loop, and steps S3.2 to S3.5 are repeated until the termination condition is reached. Here, the termination condition is set as: the generated nth image semantic word y n is a special termination symbol s t .

[0100] In this embodiment, n out cycles generate y nThe image semantic text is 1237, 16871, 631, 1629, 593, 2416, 1010, 530, 320, 1570, 269, 49407 respectively. Therefore, the image semantic text T is obtained as [1237, 16871, 631, 1629, 593, 2416, 1010, 530, 320, 1570, 269, 49407]. According to the set dictionary, the image semantic text T is translated into "two elephants are playing with each other in a field.", which is consistent with people's understanding of this. Figure 3 Therefore, transmitting only this text can effectively convey the semantics of the image to the receiver.

[0101] Notably, compared to the traditional JPEG algorithm, which achieves a minimum transmission data size of 5.12KB for the sample image, this communication method requires only 55B to transmit image semantic text, effectively minimizing communication resource requirements and enabling basic image semantic communication even when communication resources are scarce. Furthermore, as discrete information, text is highly convenient to transmit using existing digital communication systems, avoiding conflicts with existing communication systems when semantic communication is deployed in real-world systems and reducing the cost of semantic communication applications.

[0102] Step S4: sending the image semantic text from the transmitting end to the receiving end via wireless communication. The specific process includes:

[0103] S4.1. After the image semantic text T is UTF-8 encoded at the sending end, it is converted into a binary code M using arithmetic coding, but is not limited to this encoding method.

[0104] S4.2. After the binary code m is modulated at the transmitting end, a wireless signal S is obtained and sent to the receiving end through a wireless channel.

[0105] S4.3. At the receiving end, the received wireless signal S is demodulated to obtain a binary code m.

[0106] S4.4. At the receiving end, the binary code M is decoded to obtain the image semantic text T.

[0107] Step S5: Perform reasoning on downstream tasks based on the image semantic text received by the receiving end.

[0108] At the receiving end, the received image semantic text T is used to reason about various visual text subtasks, including image caption generation and image text question answering tasks.

[0109] The received image semantic text T is an image caption, which is used directly as the output of the image caption generation subtask "two elephants are playing with each other in a field.", which is consistent with human perception.

[0110] In the image-text question-answering subtask, the receiver's pre-trained large language model L uses the QA-BERT model as the receiver's image-text question-answering inference model. QA-BERT (Question Answering with BERT) is an application based on the BERT (Bidirectional Encoder Representations from Transformers) model, specifically designed for question-answering (QA) tasks and widely used in various question-answering systems. BERT is a pre-trained language representation model proposed by Google in 2018. It learns deep language features through pre-training on large amounts of text data, which can then be applied to downstream NLP tasks such as text classification, named entity recognition, sentiment analysis, and question-answering through fine-tuning.

[0111] The QA-BERT model connects the received image semantic text T with the question about the image content Q1 = "What animal is in the picture?" to obtain an intermediate feature vector, and performs textual reasoning on the intermediate feature vector to obtain the answer A1 = "elephants" corresponding to the image X to be transmitted at the sender and question Q1, which is consistent with human perception.

[0112] The QA-BERT model connects the received image semantic text T with the question about the image content Q2 = "How many elephants are in the picture?" to obtain an intermediate feature vector. It then performs textual reasoning on the intermediate feature vector and obtains the answer A2 = "two" corresponding to the image X to be transmitted at the sender and question Q2, which is consistent with human perception.

[0113] This embodiment demonstrates that under conditions of scarce communication resources, the semantic communication method of this embodiment only transmits image semantic text. While saving communication resources, it can still effectively ensure the execution of two text-based subtasks: downstream image subtitle generation and image-text question-answering. Figure 4 The effect of the communication method of this embodiment is demonstrated: image semantic text is extracted from the image, and then the image semantic text is used for downstream text-based subtasks to obtain correct results.

[0114] It is worth noting that, as in the above steps S1 to S4, the communication method of this embodiment is effective for images of any content and at any scale.

[0115] Example 2

[0116] like Figure 5 As shown, this embodiment provides a semantic communication system based on an image-text association knowledge network. The semantic communication system includes: an image-text association knowledge network construction module 501, an image feature extraction module 502, an image semantic text extraction module 503, a wireless communication module 504, and a downstream task reasoning module 505. The specific functions of each module are as follows:

[0117] Constructing an image-text association knowledge network module 501, which is used to construct an image-text association knowledge network consisting of an image preprocessing and encoding subnetwork A and an image semantic text extraction subnetwork B connected in sequence;

[0118] An image feature extraction module 502 is configured to pre-process the image to be transmitted through an image pre-processing and encoding sub-network to obtain an image vector;

[0119] An image semantic text extraction module 503 is configured to extract features from image vectors using an image semantic text extraction sub-network to obtain image semantic text.

[0120] Wireless communication module 504, used to send the image semantic text from the sending end to the receiving end via wireless communication;

[0121] The downstream task reasoning module 505 is used to perform reasoning on downstream tasks based on the image semantic text received by the receiving end.

[0122] The specific implementation of each module in this embodiment can be found in the above-mentioned embodiment 1, and will not be described one by one here; it should be noted that the device provided in this embodiment is only illustrated by the division of the above-mentioned functional modules. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure can be divided into different functional modules to complete all or part of the functions described above.

[0123] Example 3

[0124] This embodiment provides a computer device, which can be a computer, such as Figure 6As shown, the processor 602, the memory, the input device 603, the display 604 and the network interface 605 connected by the system bus 601, the processor is used to provide computing and control ability, the memory includes non-volatile storage medium 606 and internal memory 607, the non-volatile storage medium 606 stores operating system, computer program and database, the internal memory 607 provides environment for the running of operating system and computer program in non-volatile storage medium, when the processor 602 executes the computer program stored in the memory, the above-mentioned embodiment 1 proposes a kind of semantic communication method based on image-text association knowledge network, and semantic communication method includes the following steps:

[0125] S1, construct the image-text association knowledge network composed of image pre-processing and coding sub-network A and image semantic text extraction sub-network B connected in sequence;

[0126] S2, pre-process the required transmission image by image pre-processing and coding sub-network to obtain image vector;

[0127] S3, image semantic text is obtained by feature extraction to image vector by image semantic text extraction sub-network;

[0128] S4, image semantic text is sent from sending end to receiving end by wireless communication;

[0129] S5, inference of downstream task is carried out based on the image semantic text received by receiving end.

[0130] Embodiment 4

[0131] The embodiment provides a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the semantic communication method based on the image-text association knowledge network provided in the above-mentioned embodiment 1, and the semantic communication method includes the following steps:

[0132] S1, construct the image-text association knowledge network composed of image pre-processing and coding sub-network A and image semantic text extraction sub-network B connected in sequence;

[0133] S2, pre-process the required transmission image by image pre-processing and coding sub-network to obtain image vector;

[0134] S3, image semantic text is obtained by feature extraction to image vector by image semantic text extraction sub-network;

[0135] S4, image semantic text is sent from sending end to receiving end by wireless communication;

[0136] S5, inference of downstream task is carried out based on the image semantic text received by receiving end.

[0137] The storage medium described in the embodiment can be a magnetic disk, an optical disk, a computer memory, a random access memory (RAM), a U disk, a mobile hard disk, and the like.

[0138] The above embodiments are preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations, simplifications, and the like made without departing from the spirit and principles of the present application should be equivalent replacement methods and should be included in the protection scope of the present application.

Claims

1. A semantic communication method based on a graph-text association knowledge network, characterized in that: The semantic communication method comprises the following steps: S1, construct an image-text association knowledge network consisting of an image preprocessing and encoding subnetwork A and an image semantic text extraction subnetwork B connected in sequence; the image preprocessing and encoding subnetwork A includes an image preprocessing network P i and image encoder V, the image to be transmitted X passes through the image preprocessing network P i After scaling, embed the image vector Then the image vector The image feature vector is extracted by the image encoder V in: Image preprocessing network P i It consists of a segmentation layer and an embedding layer. The input is the image to be transmitted X∈3×w×h. The image width w and height h are determined by the image. The output is an image vector N p is the image vector Length, L ei The set image embedding dimension; The image encoder V is composed of N v It consists of a residual self-attention network and a pooling layer connected in sequence, and the input is an image vector The output is the extracted image feature vector L i is the dimension of the image feature vector set; The image semantic text extraction sub-network B includes a text preprocessing network P t , text encoder E and text decoder D, used to transform the image feature vector Convert image to semantic text T∈n through loop iteration out ×1,n out The length of the final output image semantic text depends on the image feature vector The text preprocessing network P t It consists of an embedding layer, the input is the intermediate image semantic text T0∈n×1, and the output is a text vector n is the current loop number, n∈[1,n out ],n out To output the text length, the image feature vector Decision, L et Embed the text to set the dimension; The text encoder E consists of N e The residual self-attention network is sequentially composed, and the input is the text vector Output is text feature vector L t is the dimension of the set text feature vector; The text decoder D consists of N d residual self-attention networks and N d The residual multimodal attention network is constructed alternately in sequence and cascaded with a mapping layer at the end. The input is the text feature vector and image feature vector The output is the nth image semantic word y n ; S2, preprocessing the image to be transmitted through the image preprocessing and encoding sub-network to obtain an image vector; S3, extract features from the image vector through the image semantic text extraction sub-network to obtain image semantic text; S4, sending the image semantic text from the sending end to the receiving end via wireless communication; S5. Perform reasoning on downstream tasks based on the image semantic text received at the receiving end.

2. The semantic communication method based on the image-text association knowledge network according to claim 1 is characterized in that: The process of step S2 is as follows: S2.1, Image Preprocessing Network P i The image X to be transmitted at the sending end is scaled to the first intermediate image X0∈3×L0×L0, where the width and height are both L0 pixels, and then divided into N p The width and height of the image are both L p The second intermediate image of the pixel I0∈N p ×(3×L p ×L p ), where N p =(L0 / L p )*(l0 / L p ), the second intermediate image I0 is mapped to a high-dimensional image vector through a linear layer S2.2, image encoder V for the input image vector Perform self-attention extraction to obtain the intermediate image feature vector The residual self-attention network in the image encoder V does not change the size of the input vector; The self-attention extraction process is as follows: The first residual self-attention network treats the input image vector Perform feature extraction to obtain the first feature map The first j∈[2,N i ] The residual self-attention network is fed into the j-1th feature map of the input Perform feature extraction to obtain the jth feature map The Nth i Feature map as intermediate image feature vector The pooling layer of the image encoder V is applied to the intermediate image feature vector Perform dimension transformation to obtain image feature vector 3. The semantic communication method based on the image-text association knowledge network according to claim 2 is characterized in that: The process of step S3 is as follows: S3.

1. Set the intermediate image semantic text T0 to be a list of length n, T0∈n×1; S3.2 Text Preprocessing Network P t Embed the intermediate image semantic text T0 to obtain the text vector S3.3, text encoder E to text vector Perform self-attention extraction to obtain text feature vector The self-attention extraction is specifically as follows: The first residual self-attention network pays attention to the input text vector Perform feature extraction to obtain the first feature text k∈[2,N e ] The residual self-attention network is fed into the k-1th feature text Perform feature extraction to obtain the kth feature text The Nth e Feature text as text feature vector Output; S3.4, text decoder D for text feature vector and image feature vector Perform multimodal attention extraction to obtain the nth image semantic word y n , wherein the multimodal attention extraction is specifically as follows: The first residual self-attention network treats the input text feature vector Perform feature extraction to obtain the first intermediate feature vector The first residual multimodal attention network takes the first intermediate feature vector of the input and image feature vector Perform feature extraction to obtain the first intermediate mixed feature vector The first l∈[2,N d ]The input of the residual self-attention network is the l-1th intermediate mixed feature vector The output is the lth intermediate eigenvector The first l∈[2,N d ]The input of the residual multimodal attention network is the l-1th intermediate feature vector The output is the lth intermediate mixed feature vector The last cascade mapping layer of the text decoder D converts the Nth d The intermediate mixed feature vector is mapped to obtain the nth image semantic word y n ; S3.5, the nth image semantic word y n Spliced ​​to the end of the intermediate image semantic text T0 until the generated nth image semantic word y n The special termination symbol s is set e , and output the obtained intermediate image semantic text T0 as the image semantic text T.

4. The semantic communication method based on the image-text association knowledge network according to claim 1 is characterized in that: The process of step S4 is as follows: S4.

1. At the sending end, the image semantic text T is Unicode-encoded and converted into a binary code M; S4.

2. After modulating the binary code M at the transmitting end, a wireless signal S is obtained and sent to the receiving end via a wireless channel. S4.

3. Demodulate the received wireless signal S at the receiving end to obtain a binary code M; S4.

4. At the receiving end, the binary code M is decoded to obtain the image semantic text T.

5. The semantic communication method based on the image-text association knowledge network according to claim 1 is characterized in that: The process of step S5 is as follows: At the receiving end, the received image semantic text T is used to reason about various visual text subtasks, including image caption generation and image text question answering tasks. The received image semantic text T is an image caption, which is directly used as the output of the image caption generation subtask; In the image-text question-answering subtask, the pre-trained large language model L at the receiving end connects the received image semantic text T with the question Q∈k×1 about the image content to obtain an intermediate feature vector, and performs text reasoning on the intermediate feature vector to obtain the answer S∈l×1 corresponding to the image X to be transmitted at the sending end and the question Q, where l is the word length of the answer itself.

6. A semantic communication system based on a picture-text association knowledge network, used to implement the semantic communication method based on a picture-text association knowledge network according to any one of claims 1 to 5, characterized in that: The semantic communication system comprises: Construct an image-text association knowledge network module, which is used to construct an image-text association knowledge network consisting of the image preprocessing and encoding subnetwork A and the image semantic text extraction subnetwork B connected in sequence; An image feature extraction module is used to preprocess the image to be transmitted through the image preprocessing and encoding sub-network to obtain an image vector; Image semantic text extraction module, used to extract features from image vectors through the image semantic text extraction sub-network to obtain image semantic text; A wireless communication module, used for transmitting the image semantic text from the transmitting end to the receiving end via wireless communication; The downstream task reasoning module is used to perform reasoning on downstream tasks based on the image semantic text received by the receiving end.

7. A computer device comprising a processor and a memory for storing a program executable by the processor, characterized in that: When the processor executes the program stored in the memory, it implements the semantic communication method based on the image-text association knowledge network described in any one of claims 1 to 5.

8. A storage medium storing a program, characterized in that: When the program is executed by a processor, the semantic communication method based on the image-text association knowledge network described in any one of claims 1 to 5 is implemented.

Images