Semantic communication method, device and system, electronic equipment and computer storage medium

Abstract

The invention provides a semantic communication method and device, and relates to the technical field of communication, in particular to the technical fields of large models, deep learning and the like. According to the specific implementation scheme, the method comprises the steps of obtaining a to-be-processed material comprising at least one type of modal data; generating a transmission problem text based on the to-be-processed material; and the transmission problem text is transmitted to the receiving end, so that the receiving end obtains the to-be-processed material through the transmission problem text, and the data transmission amount is reduced through the scheme.

Description

Technical Field

[0001] The present disclosure relates to the field of communication technologies, specifically to the fields of data processing, computers, etc., and particularly to a semantic communication method, a semantic communication device, a semantic communication system, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] As the capacity of communication systems approaches the Shannon limit, traditional syntactic communication has encountered an efficiency bottleneck. To overcome this bottleneck, a new paradigm of semantic communication has been proposed. Current semantic communication focuses on the semantic content of information, uses artificial intelligence technologies for semantic extraction, compression, and transmission, removes redundant data, reduces the amount of transmitted data, thereby improving the efficiency and depth of communication, and can exhibit better performance than traditional communication systems under bandwidth-limited and noise-limited conditions.

[0003] Since the goal of a semantic communication system is to transmit semantic information, under the condition that the channel capacity is not large enough, the transmission of semantic information will be limited, affecting the semantic recovery performance at the receiving end. At the same time, for different modalities of data to be transmitted, how to transmit the data more effectively and concisely has become a hot issue to be studied in semantic communication. Summary of the Invention

[0004] The present disclosure provides a semantic communication method, a semantic communication device, a semantic communication system, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to a first aspect, there is provided a semantic communication method, the method comprising: obtaining a to-be-processed material including at least one modality of data; generating a transmission problem text based on the to-be-processed material; and transmitting the transmission problem text to a receiving end, so that the receiving end obtains the to-be-processed material through the transmission problem text.

[0006] According to a second aspect, there is provided another semantic communication method, the method comprising: receiving a transmission problem text; generating a reply answer text based on the transmission problem text; and obtaining the to-be-processed material based on the reply answer text.

[0007] According to a third aspect, there is provided a semantic communication device, the device comprising: an obtaining unit configured to obtain a to-be-processed material including at least one modality of data; a problem generation unit configured to generate a transmission problem text based on the to-be-processed material; and a transmission unit configured to transmit the transmission problem text to a receiving end, so that the receiving end obtains the to-be-processed material through the transmission problem text.

[0008] According to a fourth aspect, another semantic communication device is provided, which includes: a receiving unit configured to receive a transmission problem text; an answer generation unit configured to generate a reply answer text based on the transmission problem text; and an obtaining unit configured to obtain a material to be processed based on the reply answer text.

[0009] According to a fifth aspect, a semantic communication system is provided, which includes: a sending end and a receiving end; the sending end is configured to obtain a material to be processed including at least one type of modal data; generate a transmission problem text based on the material to be processed; transmit the transmission problem text to the receiving end; the receiving end is configured to receive the transmission problem text; generate a reply answer text based on the transmission problem text; and obtain a material to be processed based on the reply answer text.

[0010] According to a sixth aspect, an electronic device is provided, which includes: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in any implementation manner of the first aspect or the second aspect.

[0011] According to a seventh aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, and the computer instructions are used to cause a computer to execute the method described in any implementation manner of the first aspect or the second aspect.

[0012] The semantic communication method and device provided by the embodiments of the present disclosure first obtain a material to be processed including at least one type of modal data; then generate a transmission problem text based on the material to be processed; and finally transmit the transmission problem text to the receiving end so that the receiving end obtains the material to be processed through the transmission problem text. Thus, the material to be processed including at least one type of modal data is obtained to generate a transmission problem text and transmitted to the receiving end, and the receiving end obtains the material to be processed through the transmission problem text. Through the efficient semantic information transmission in the form of questions and answers, the multi-modal data transmission efficiency is improved, the amount of transmitted data is reduced, and the comprehensiveness and reliability of the information obtained by the receiving end are improved.

[0013] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understandable through the following description. Description of the Drawings

[0014] The drawings are used to better understand the solution and do not constitute a limitation to the present disclosure. Among them:

[0015] Figure 1 is a flowchart of an embodiment of the semantic communication method according to the present disclosure;

[0016] Figure 2 is a flowchart according to another embodiment of the semantic communication method of the present disclosure

[0017] Figure 3 is a schematic structural diagram of a semantic communication system corresponding to the semantic communication method of the present disclosure;

[0018] Figure 4 is a schematic structural diagram according to an embodiment of the semantic communication device of the present disclosure;

[0019] Figure 5 is a schematic structural diagram according to another embodiment of the semantic communication device of the present disclosure;

[0020] Figure 6 is a schematic structural diagram of a semantic communication system according to the present disclosure;

[0021] Figure 7 is a block diagram of an electronic device for implementing the semantic communication method of the embodiments of the present disclosure. Detailed implementation manners

[0022] The following describes exemplary embodiments of the present disclosure with reference to the accompanying drawings. Various details of the embodiments of the present disclosure are included to facilitate understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted below.

[0023] The goal of a semantic communication system is to transmit semantic information. Under the condition that the channel capacity is not large enough, the transmission of semantic information will be limited, affecting the semantic recovery performance of the receiving end. Moreover, for the transmission of multimodal data, the data transmission efficiency is relatively low. To address this defect, the present disclosure proposes a semantic communication method. Figure 1 shows a process 100 according to an embodiment of the semantic communication method of the present disclosure. The above-mentioned semantic communication method includes the following steps:

[0024] Step 101, obtain a material to be processed including at least one type of modal data.

[0025] In this embodiment, at least one type of modal data refers to one or more types of data in multimodal data. Multimodal data refers to data obtained from different fields or perspectives. These data are called a modality through different forms of existence or information sources, and data composed of two or more modalities is called multimodal data. Multimodal data can include different forms of data such as text, pictures, audio, and video.

[0026] In this embodiment, the material to be processed is the material that needs to be transmitted, and the material includes at least one type of modal data. For example, the material to be processed includes text, or the material to be processed includes audio and text.

[0027] In this embodiment, the material to be processed can be obtained in various ways. For example, by communicating with the terminal, the material to be processed sent by the terminal can be obtained; or by directly accessing the database, the material to be processed can be obtained from the database.

[0028] Step 102: Generate a transmission problem text based on the material to be processed.

[0029] In this embodiment, the transmission problem text is a type of text data. Through transmitting the text data, semantic communication can be carried out simply and quickly, and channel resources can also be saved. The transmission problem text represents a problem related to the material to be processed. By answering this problem, a reply answer text can be obtained, and the material to be processed can be restored through the reply answer text. For example, if the material to be processed includes an image of a famous painting by a certain writer, a question is generated through this image: What is the 20th painting by a certain writer depicting a certain object? By transmitting this question as the transmission problem text to the receiving end, the receiving end can effectively restore the image.

[0030] In this embodiment, the above-mentioned step 102 includes: identifying the multi-modal data in the material to be processed, and converting the multi-modal data into the text to be processed through a general conversion tool; generating a transmission problem text based on the text to be processed. Among them, generating a transmission problem text based on the text to be processed includes: performing semantic recognition on the text to be processed to obtain a semantic text; generating a transmission problem text based on the semantic text. Among them, the above-mentioned text to be processed is a descriptive text that describes the material to be processed. Through a large model or rule matching, the semantic text can be processed into a transmission problem text. The above-mentioned semantic recognition can adopt traditional semantic recognition methods such as natural language processing methods to obtain the semantic text, which will not be elaborated here.

[0031] Optionally, generating a transmission problem text based on the text to be processed further includes: detecting whether the text to be processed conforms to a predefined fixed sentence pattern; in response to detecting that the text to be processed conforms to the predefined fixed sentence pattern, extracting the title of the text to be processed; filling the title into a pre-set question template to obtain the transmission problem text. For example, if the text to be processed is: A poem, then by extracting the title of the poem in the text to be processed and filling the title into the question template of the content of the interrogative poem, the transmission problem text can be obtained.

[0032] Step 103: Transmit the transmission problem text to the receiving end so that the receiving end can obtain the material to be processed through the transmission problem text.

[0033] In this embodiment, the execution entity on which the semantic communication method runs is connected to the receiving end through a transmission channel. First, the execution entity encodes the transmission problem text, thereby converting the transmission problem text into a signal form suitable for transmission through the transmission channel, such as a digital signal or an analog signal. Then, the execution entity sends the encoded information through a transmitter and enters the transmission channel.

[0034] In this embodiment, the receiving end receives the encoded information through the transmission channel, decodes the encoded information to obtain the transmission problem text, replies to the transmission problem text to obtain the reply answer text of the material to be processed, and obtains the material to be processed through the reply answer text.

[0035] The semantic communication method provided by the embodiment of the present disclosure first obtains the material to be processed including at least one type of modal data. Then, based on the material to be processed, a transmission problem text is generated. Finally, the transmission problem text is transmitted to the receiving end so that the receiving end can obtain the material to be processed through the transmission problem text. Thus, the material to be processed including at least one type of modal data is obtained to generate a transmission problem text and transmitted to the receiving end. The receiving end obtains the material to be processed through the transmission problem text. Through the efficient semantic information transmission in the form of question and answer, the multi-modal data transmission efficiency is improved, the amount of transmitted data is reduced, and the comprehensiveness and reliability of the information obtained by the receiving end are improved.

[0036] In some optional implementation manners of the present disclosure, generating the transmission problem text based on the material to be processed includes: generating a text to be processed based on the material to be processed; inputting the text to be processed into a pre-trained question generation model to obtain the transmission problem text output by the question generation model.

[0037] In this optional implementation manner, the text to be processed is a descriptive text that describes the material to be processed, that is, the text to be processed is the text representation of the material to be processed. When the material to be processed only includes text, the text to be processed is obtained by preprocessing the material to be processed. Or when the material to be processed only includes text, the material to be processed is directly used as the text to be processed.

[0038] In this optional implementation manner, when the material to be processed includes multi-modal data other than text, a data-to-text conversion tool corresponding to the multi-modal data is used to process the multi-modal data, and the processed text is combined with the text in the material to be processed to obtain the text to be processed.

[0039] In this optional implementation manner, the question generation model can be a deep learning model that represents the corresponding relationship between the text to be processed and the transmission problem text. The question generation model can adopt various model architectures. For example, the question generation model can adopt transform and vision transform architectures.

[0040] In this alternative implementation, the pre-trained question generation model is a model that has been pre-trained. The specific training process of the question generation model is as follows:

[0041] Collect a sufficient dataset of question generation samples from different data sources. The data sources can include public datasets, enterprise internal databases, or be obtained from the Internet through web crawling technology. The sample dataset includes at least one sample data, and the sample data includes text and the corresponding questions for the text.

[0042] Use the collected data to train the question generation model. Based on the predicted questions output by the question generation model during each iterative training and the sample data input into the question generation model during this iterative training, adjust the parameters of the question generation model to optimize the performance of the question generation model.

[0043] Evaluate the performance of the question generation model through evaluation metrics (such as accuracy, recall rate) to ensure that the trained question generation model can meet the business requirements.

[0044] The method for generating transmission questions provided in this embodiment generates a text to be processed based on the material to be processed; inputs the text to be processed into the pre-trained question generation model to obtain the transmission question text output by the question generation model, thereby improving the reliability of the transmission question text through the question generation model.

[0045] Optionally, the above-mentioned generating the transmission question text based on the material to be processed includes: inputting the material to be processed and the question prompt text into a multi-modal large model to obtain the transmission question text output by the multi-modal large model. Among them, the question prompt text is used to prompt the multi-modal large model to perform different-modal data analysis on the material to be processed, determine the content of the material to be processed, and generate transmission question text for the material content.

[0046] In some alternative implementations of the present disclosure, the above-mentioned generating the text to be processed based on the material to be processed includes: determining the material type based on the material to be processed; determining the material knowledge base based on the material type; and generating the text to be processed based on the material knowledge base and the material to be processed.

[0047] In this alternative implementation, the above-mentioned determining the material type based on the material to be processed includes: detecting whether each modal data in the material to be processed has its corresponding data format or data source. For example, images are usually in data formats such as JPEG, PNG, etc., audio is usually in data formats such as WAV, MP3, etc., and text is usually in data formats such as TXT, CSV, etc. In response to detecting that there is modal data with a corresponding data format, determine the material type of this modal data as the type corresponding to the data format; in response to detecting that the modal data has a corresponding data source, such as metadata, determine the material type of this modal data through the data source.

[0048] In this alternative implementation, determining the material type based on the material to be processed further includes: in response to detecting that modal data does not have a corresponding data format and data source, performing data content detection on the modal data, that is, detecting whether the modal data conforms to a preset content format. If it conforms to the preset content format, determining that the modal data belongs to the material type of this content format. For example, the preset content formats include: visual data composed of pixel matrices, auditory data represented by waveforms or spectrograms, text data composed of characters, words, or sentences, and sensor data composed of time series data including multi-dimensional features, etc.

[0049] In this alternative implementation, there can be multiple material knowledge bases, and each material knowledge base has materials of a corresponding material type, that is, each material knowledge base can store materials of one type of modal data, and each material knowledge base also correspondingly stores the text corresponding to each material. Each material knowledge base has a corresponding material type, and the material knowledge base can be determined through the material type.

[0050] In this alternative implementation, generating the text to be processed based on the material knowledge base and the material to be processed includes: determining the materials in the material knowledge base through the material to be processed; selecting the text to be processed from the material knowledge base based on this material.

[0051] The method for generating the text to be processed provided by this alternative implementation determines the material type based on the material to be processed; determines the material knowledge base based on the material type; generates the text to be processed based on the material knowledge base and the material to be processed, and determines the text to be processed through the material knowledge base, providing a reliable implementation method for obtaining the text to be processed and improving the reliability of obtaining the text to be processed.

[0052] Optionally, generating the text to be processed based on the material to be processed includes: determining the material type based on the material to be processed; determining the text to be processed based on the material type and the material to be processed. Determining the text to be processed based on the material type and the material to be processed includes: selecting a text conversion tool of the corresponding type based on the material type, and converting the material to be processed into the text to be processed through the text conversion tool.

[0053] In some embodiments of the present disclosure, the above semantic communication method further includes: determining the material type based on the material to be processed; transmitting the material type to the receiving end so that the receiving end can obtain the material to be processed by transmitting the problem text and the material type.

[0054] In this embodiment, for the above determination of the material type based on the material to be processed, reference can specifically be made to the content described in the above alternative implementation.

[0055] In this embodiment, the receiving end has a corresponding material knowledge base. Transmitting the material type to the receiving end enables the receiving end to determine the corresponding material knowledge base based on the material type, so that after obtaining the reply answer text, the material to be processed is determined based on the reply answer text and the material knowledge base.

[0056] The semantic communication method provided in this embodiment determines the material type based on the material to be processed; transmits the material type to the receiving end so that the receiving end obtains the material to be processed by transmitting the question text and the material type, and the receiving end can further judge the material to be processed through the material type, improving the reliability of the receiving end to obtain the material to be processed.

[0057] In some alternative implementation manners of the present disclosure, determining the material type based on the material to be processed includes: inputting the material to be processed into a data type recognizer to obtain at least one data type output by the data type recognizer; using the at least one data type as the material type.

[0058] In this alternative implementation manner, the material type may include: text type, video type, audio type, sensor type, etc.

[0059] In this alternative implementation manner, the data type recognizer is a tool for determining multiple data types by recognizing the material content of the material. For example, the data type recognizer can directly recognize the characters and words in the material to be processed, and thus directly determine the data type of the recognized characters and words as the text type; the data type recognizer can also recognize the pixel matrix in the material to be processed, and thus determine the data type of the recognized pixel matrix as the video type; the data type recognizer can also recognize the spectrogram in the material to be processed, and thus determine the data type of the recognized spectrogram as the audio type.

[0060] The method for determining the material type provided in this alternative implementation inputs the material to be processed into a data type recognizer to obtain at least one data type output by the data type recognizer; uses the at least one data type as the material type, providing a reliable implementation manner for obtaining the material type and improving the reliability and accuracy of obtaining the material type.

[0061] Figure 2 Flow 200 of another embodiment of the semantic communication method according to the present disclosure is shown. The above semantic communication method includes the following steps:

[0062] Step 201, receive the transmitted question text.

[0063] In this embodiment, the transmitted question text is a text obtained by the sending end by processing the material to be processed. The transmitted question text represents a question related to the material to be processed, and the reply answer text is obtained by answering this question. The material to be processed can be restored through the reply answer text.

[0064] In this embodiment, the execution entity on which the semantic communication method runs is connected to the sending end through a transmission channel. The execution entity can obtain the transmission problem text sent by the sending end through the transmission channel.

[0065] Step 202: Generate a reply answer text based on the transmission problem text.

[0066] In this embodiment, the reply answer text is the text obtained after answering the transmission problem text. For example, if the transmission problem text is: What's the weather like in xx year and yy month? The reply answer text is: Sunny.

[0067] Step 203: Obtain the material to be processed based on the reply answer text.

[0068] In this embodiment, the material to be processed is the content that the sending end needs to transmit. To simplify the transmission of the material to be processed, the transmission problem text is obtained through the material to be processed; after the execution entity on which the semantic communication method runs obtains the transmission problem text, it answers the transmission problem text to obtain the reply answer text, and the material to be processed is obtained by performing material conversion on the reply answer text.

[0069] In this embodiment, the sending end and the execution entity on which the semantic communication method runs can, through a prior agreement, pre-store the types of the material to be processed in advance. Thus, when the execution entity obtains the transmission problem text or the reply answer text, it can directly determine the material type of the material to be processed through the pre-stored information. For this reason, after the execution entity obtains the reply answer text, it can perform material generation of the corresponding material type on the reply answer text to obtain the material to be processed.

[0070] In this embodiment, the above step 203 includes: inputting the reply answer text, the pre-stored material type, and the material generation prompt words into the multi-modal large model to obtain the material to be processed output by the multi-modal large model.

[0071] Optionally, the execution entity can also pre-agree on the material types to which different types of entities in the text need to be converted. The above step 203 also includes: determining various types of entities based on the reply answer text; determining the corresponding types of materials based on various types of entities; sorting the materials of each type of entity according to the order of different types of entities in the reply answer text to obtain the material to be processed. For example, the sending end and the execution entity agree that the character entity in the text adopts video type material, the sound entity in the text adopts audio type material, and other content in the text adopts text type material. For example, if the reply answer text is: A little girl sang a song named TT, then the converted material to be processed is: An image of a girl, sang a song, TT audio file.

[0072] The semantic communication method provided by the embodiments of the present disclosure first receives a transmission problem text, then generates a reply answer text based on the transmission problem text, and finally obtains the material to be processed based on the reply answer text. Thereby, efficient semantic information transmission in the form of question and answer is realized, the multi-modal data transmission efficiency is improved, the amount of transmitted data is reduced, and the comprehensiveness and reliability of the information obtained at the receiving end are improved.

[0073] In some alternative implementation manners of the present disclosure, the above-mentioned generating a reply answer text based on the transmission problem text includes: inputting the transmission problem text into a pre-trained answer generation model to obtain the reply answer text output by the question and answer generation model.

[0074] In this alternative implementation manner, the reply answer text is a description text for describing the material to be processed, that is, the reply answer text obtained by replying to the transmission problem text is the text representation of the material to be processed.

[0075] In this alternative implementation manner, the answer generation model can be a deep learning model representing the corresponding relationship between the transmission problem text and the reply answer text. The answer generation model can adopt various model architectures, such as the answer generation model can adopt the transform or vision transform architecture.

[0076] In this alternative implementation manner, the pre-trained answer generation model is a model that has been pre-trained. The specific training process of the answer generation model is as follows:

[0077] Collect a sufficient answer generation sample data set from different data sources. The data sources can include public data sets, enterprise internal databases, or be obtained from the Internet through web crawling technology. The sample data set includes at least one sample data, and the sample data includes a question text and the corresponding answer text to the question text.

[0078] Use the collected data to train the answer generation model, and adjust the parameters of the answer generation model based on the predicted question output by the answer generation model during each iterative training and the sample data input into the answer generation model in this iterative training to optimize the performance of the answer generation model.

[0079] Evaluate the performance of the answer generation model through evaluation metrics (such as accuracy rate, recall rate) to ensure that the trained answer generation model can meet the business requirements.

[0080] The method for generating a reply answer text provided by this embodiment inputs the transmission problem text into a pre-trained answer generation model to obtain the reply answer text output by the question and answer generation model. Thereby, through the answer generation model, the reliability of the obtained reply answer text is improved.

[0081] In some alternative implementations of the present disclosure, the above method further includes: receiving the material type of the material to be processed; obtaining the material to be processed based on the reply answer text, including: obtaining the material to be processed based on the reply answer text and the material type.

[0082] In this alternative implementation, the execution entity on which the semantic communication method runs receives the material type sent by the sending end. Through the material type, the type of the material to be processed can be determined. By generating the material corresponding to the material type for the reply answer text, the material to be processed is obtained.

[0083] The method for obtaining the material to be processed provided by this alternative implementation receives the material type of the material to be processed, and obtains the material to be processed based on the reply answer text and the material type, improving the reliability of obtaining the material to be processed.

[0084] In some alternative implementations of the present disclosure, the above obtaining the material to be processed based on the reply answer text and the material type includes: determining a preset material knowledge base based on the material type; obtaining the material to be processed based on the material knowledge base and the reply answer text.

[0085] As Figure 3 shown, the sending end obtains the material to be processed S. In Figure 3 , the material to be processed S includes: initial text data, video data, and audio data. Based on the material to be processed S, the material type L is determined; based on the material to be processed S, the text to be processed is obtained, and the text to be processed is input into the question generation model to obtain the transmission question text output by the question generation model. The transmission question text and the material type L are transmitted to the receiving end. The receiving end receives the material type L and the transmission question text, inputs the transmission question text into the answer generation model to obtain the reply answer text output by the answer generation model, and obtains the corresponding material to be processed S from the material knowledge base based on the material type.

[0086] The method for obtaining the material to be processed provided by this alternative implementation determines a preset material knowledge base based on the material type; obtains the material to be processed based on the material knowledge base and the reply answer text, improving the accuracy and reliability of obtaining the material to be processed.

[0087] Optionally, in this embodiment, the material to be processed may only include: the text data to be transmitted. The sending end obtains the text data to be transmitted, and generates the transmission question text by passing the transmission text data through the question generation model; the receiving end receives the transmission question text and the material type from the sending end, and obtains the reply answer text of the transmission question text based on the answer generation model; the receiving end directly restores the material to be processed according to the reply answer text.

[0088] Optionally, in this embodiment, the material to be processed may only include: image data. The sending end acquires the image data, converts the image data into a description text; passes the description text through a question generation model to generate a transmission question text; the receiving end receives the transmission question text and the material type from the sending end, and obtains a reply answer text for the transmission question text based on an answer generation model; the receiving end directly restores the material to be processed according to the reply answer text.

[0089] Optionally, in this embodiment, the material to be processed may only include: video data. The sending end acquires the video data, converts the video data into a description text; passes the description text through a question generation model to generate a transmission question text; the receiving end receives the transmission question text and the material type from the sending end, and obtains a reply answer text for the transmission question text based on an answer generation model; the receiving end directly restores the material to be processed according to the reply answer text.

[0090] Further referring to Figure 4 , as an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a semantic communication device. This device embodiment corresponds to Figure 1 the method embodiment shown, and this device can be specifically applied to various electronic devices.

[0091] As Figure 4 shown, the semantic communication device 400 provided in this embodiment includes: an acquisition unit 401, a question generation unit 402, and a transmission unit 403. Among them, the above acquisition unit 401 can be configured to acquire the material to be processed including at least one modality data. The above question generation unit 402 can be configured to generate a transmission question text based on the material to be processed. The above transmission unit 403 can be configured to transmit the transmission question text to the receiving end so that the receiving end obtains the material to be processed through the transmission question text.

[0092] In this embodiment, for the specific processing of the acquisition unit 401, the question generation unit 402, and the transmission unit 403 in the semantic communication device 400 and the technical effects brought by them, reference can be respectively made to Figure 1 the relevant descriptions of step 101, step 102, and step 103 in the corresponding embodiment, which will not be elaborated here.

[0093] In some optional implementation manners of this embodiment, the above question generation unit 402 is configured to: generate a text to be processed based on the material to be processed; input the text to be processed into a pre-trained question generation model to obtain the transmission question text output by the question generation model.

[0094] In some alternative implementation manners of this embodiment, the above-mentioned problem generation unit 402 is configured to: determine the material type based on the material to be processed; determine the material knowledge base based on the material type; and generate the text to be processed based on the material knowledge base and the material to be processed.

[0095] In some alternative implementation manners of this embodiment, the above-mentioned device further includes a unit to be processed (not shown in the figure), and the unit to be processed is configured to: determine the material type based on the material to be processed; and transmit the material type to the receiving end, so that the receiving end obtains the material to be processed by transmitting the problem text and the material type.

[0096] In some alternative implementation manners of this embodiment, the above-mentioned unit to be processed is further configured to: input the material to be processed into a data type recognizer to obtain at least one data type output by the data type recognizer; and use the at least one data type as the material type.

[0097] For the semantic communication device provided by an embodiment of the present disclosure, first, an acquisition unit 401 acquires the material to be processed including at least one modality data; then, a problem generation unit 402 generates a transmission problem text based on the material to be processed; and finally, a transmission unit 403 transmits the transmission problem text to the receiving end, so that the receiving end obtains the material to be processed through the transmission problem text. Thus, the material to be processed including at least one modality data is used to generate a transmission problem text and transmitted to the receiving end, and the receiving end obtains the material to be processed through the transmission problem text. Through the efficient semantic information transmission in the form of question and answer, the multi-modal data transmission efficiency is improved, the amount of transmitted data is reduced, and the comprehensiveness and reliability of the information obtained by the receiving end are improved.

[0098] Further referring to Figure 5 , as an implementation of the methods shown in the above figures, the present disclosure provides another embodiment of a semantic communication device. This device embodiment corresponds to the method embodiment shown in Figure 2 , and this device can be specifically applied to various electronic devices.

[0099] As Figure 5 shown, the semantic communication device 500 provided in this embodiment includes: a receiving unit 501, an answer generation unit 502, and an obtaining unit 503. Among them, the above-mentioned receiving unit 501 can be configured to receive the transmission problem text. The above-mentioned answer generation unit 502 can be configured to generate a reply answer text based on the transmission problem text. The above-mentioned obtaining unit 503 can be configured to obtain the material to be processed based on the reply answer text.

[0100] In this embodiment, for the specific processing of the receiving unit 501, the answer generation unit 502, and the obtaining unit 503 in the semantic communication device 500 and the technical effects brought by them, reference can be made to Figure 2The relevant descriptions of steps 201, 202, and 203 in the corresponding embodiments will not be elaborated here.

[0101] In some alternative implementation manners of this embodiment, the above answer generation unit 502 is configured to: input the transmission problem text into a pre-trained answer generation model to obtain the reply answer text output by the Q&A generation model.

[0102] In some alternative implementation manners of this embodiment, the above device further includes: a material processing unit (not shown in the figure), and the material processing unit is configured to: receive the material type of the material to be processed; the above obtaining unit 503 is configured to: obtain the material to be processed based on the reply answer text and the material type.

[0103] In some alternative implementation manners of this embodiment, the above obtaining unit 503 is further configured to: determine a preset material knowledge base based on the material type; obtain the material to be processed based on the material knowledge base and the reply answer text.

[0104] Further referring to Figure 6 , as an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a semantic communication system, and this system embodiment corresponds to Figure 1 、 Figure 2 The method embodiments shown, and this system can be specifically implemented by various electronic devices.

[0105] As Figure 6 shown, the semantic communication system 600 provided in this embodiment includes: a sending end 601 and a receiving end 602. Among them, the above sending end 601 is used to obtain the material to be processed including at least one type of modal data; generate a transmission problem text based on the material to be processed; and transmit the transmission problem text to the receiving end. The above receiving end 602 is used to receive the transmission problem text; generate a reply answer text based on the transmission problem text; and obtain the material to be processed based on the reply answer text.

[0106] In this embodiment, both the sending end 601 and the receiving end 603 can be clients or both can be servers.

[0107] In this embodiment, in the semantic communication system 600: the specific processing of the sending end 601 and the technical effects brought by it can be respectively referred to Figure 1 Steps 101, 102, and 103 in the corresponding embodiments; the specific processing of the receiving end 602 and the technical effects brought by it can be respectively referred to Figure 2 The relevant descriptions of steps 201, 202, and 203 in the corresponding embodiments will not be elaborated here.

[0108] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0109] Figure 7 A schematic block diagram of an exemplary electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as, for example, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, for example, a personal digital processor, a cellular phone, a smartphone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0110] As Figure 7 shown, the device 700 includes a computing unit 701 that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0111] A plurality of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, an optical disc, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0112] The computing unit 701 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 executes the various methods and processes described above, such as the semantic communication method. For example, in some embodiments, the semantic communication method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the semantic communication method described above can be executed. Alternatively, in other embodiments, the computing unit 701 can be configured to execute the semantic communication method in any other suitable manner (e.g., by means of firmware).

[0113] The various embodiments of the systems and techniques described above in this document can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGA), application-specific integrated circuits (ASIC), application-specific standard products (ASSP), systems-on-a-chip (SOC), complex programmable logic devices (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a dedicated or general-purpose programmable processor, and can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0114] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to the processor or controller of a general-purpose computer, a special-purpose computer, or other programmable semantic communication devices, such that when the program codes are executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program codes can be executed entirely on the machine, partially on the machine, executed partially on the machine as an independent software package and partially on a remote machine, or executed entirely on a remote machine or server.

[0115] In the context of this disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0116] In order to provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0117] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an information server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0118] A computer system can include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, a server of a distributed system, or a server incorporating a blockchain.

[0119] It should be understood that the various forms of processes shown above can be used, with steps reordered, added or deleted. For example, the steps recited in this disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution disclosed in this disclosure can be achieved, and no limitation is imposed herein.

[0120] The above specific embodiments do not constitute a limitation on the protection scope of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub - combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the protection scope of this disclosure.

Claims

1. A semantic communication method, the method comprising: Obtaining a to-be-processed material including at least one modality data; Generating a transmission problem text based on the to-be-processed material; Transmitting the transmission problem text to a receiving end, so that the receiving end obtains the to-be-processed material through the transmission problem text.

2. The method according to claim 1, wherein, The generating a transmission problem text based on the to-be-processed material includes: Generating a to-be-processed text based on the to-be-processed material; Inputting the to-be-processed text into a pre-trained problem generation model to obtain the transmission problem text output by the problem generation model.

3. The method according to claim 2, wherein The generating a to-be-processed text based on the to-be-processed material includes: Determining a material type based on the to-be-processed material; Determining a material knowledge base based on the material type; Generating a to-be-processed text based on the material knowledge base and the to-be-processed material.

4. The method according to any one of claims 1-3, the method further comprising: Determining a material type based on the to-be-processed material; Transmitting the material type to the receiving end, so that the receiving end obtains the to-be-processed material through the transmission problem text and the material type.

5. The method according to claim 4, wherein, The determining a material type based on the to-be-processed material includes: Inputting the to-be-processed material into a data type recognizer to obtain at least one data type output by the data type recognizer; Taking the at least one data type as the material type.

6. A semantic communication method, the method comprising: Receiving a transmission problem text; Generating a reply answer text based on the transmission problem text; Obtaining a to-be-processed material based on the reply answer text.

7. The method according to claim 6, wherein, The generating a reply answer text based on the transmission problem text includes: Inputting the transmission problem text into a pre-trained answer generation model to obtain the reply answer text output by the question and answer generation model.

8. The method according to claim 6 or 7, the method further comprising: Receiving the material type of the to-be-processed material; The obtaining a to-be-processed material based on the reply answer text includes: Obtaining a to-be-processed material based on the reply answer text and the material type.

9. The method according to claim 8, wherein, The obtaining a to-be-processed material based on the reply answer text and the material type includes: Determining a preset material knowledge base based on the material type; Obtaining a to-be-processed material based on the material knowledge base and the reply answer text.

10. A semantic communication device, the device comprising: An obtaining unit configured to obtain a to-be-processed material including at least one modality data; A problem generation unit configured to generate a transmission problem text based on the to-be-processed material; A transmission unit configured to transmit the transmission problem text to a receiving end, so that the receiving end obtains the to-be-processed material through the transmission problem text.

11. A semantic communication device, the device comprising: A receiving unit configured to receive a transmission problem text; An answer generation unit configured to generate a reply answer text based on the transmission problem text; An obtaining unit configured to obtain a to-be-processed material based on the reply answer text.

12. A semantic communication system, the system comprising: A sending end and a receiving end; The sending end is used to obtain the material to be processed including at least one modality of data; Based on the material to be processed, generate a transmission problem text; Transmit the transmission problem text to the receiving end; The receiving end is used to receive the transmission problem text; Based on the transmission problem text, generate a reply answer text; Based on the reply answer text, obtain the material to be processed.

13. An electronic device, characterized in that, Comprising: At least one processor; And A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1-9.

14. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the method according to any one of claims 1-9.

Images