Apparatus and method for sequential semantic generation communication in communication system

Abstract

The present disclosure relates generally to a communication system and, more particularly, to an apparatus and a method for sequential semantic generation communication in a communication system. An operation method of a terminal in a communication system may comprise the steps of: receiving a first image; on the basis of a first model, generating a text prompt from the first image; evaluating the importance of a word included in the text prompt; and on the basis of the importance of the word, transmitting the word to another terminal.

Description

Device and method for sequential semantic generation communication in a communication system

[0001] The present disclosure relates generally to communication systems, and more particularly to devices and methods for sequential semantic generation communication in communication systems.

[0002] Recent advances in artificial intelligence (AI) technology are having a significant impact on communications systems. In particular, AI technology is being actively utilized to address challenges such as limited bandwidth, diverse channel conditions, and diverse user requirements.

[0003] 3GPP communication standards are also applying deep learning technology to communication systems, preparing for next-generation 6G communications beyond 5G. These technological advancements demand rapid data exchange and information conversion in diverse industries, including autonomous vehicles, mobile edge computing, and robotics.

[0004] In particular, generative models such as DALL-E, CLIP, and BLIP have recently achieved remarkable results in text-to-image conversion tasks. These multimodal generative models can understand and generate shared meaning across two different data domains: text and images. This plays a crucial role in leveraging the ability to interpret and explain data across various modalities in communication systems and to generate interactive cognitive meaning.

[0005] Based on the discussion as described above, the present disclosure provides a device and method for sequential semantic generation communication in a communication system.

[0006] Additionally, the present disclosure provides a device and method for maximizing communication efficiency through conversion between text and images by utilizing a multimodal generative model in a communication system.

[0007] In addition, the present disclosure provides a device and method for reducing communication load by conveying the meaning of image data using text in a communication system.

[0008] In addition, the present disclosure provides a device and method for fast and efficient communication by sequentially conveying the meaning of an image using text in a communication system.

[0009] Additionally, the present disclosure provides a device and method for optimizing a transmission order according to the information importance of each word of a text in a communication system.

[0010] According to various embodiments of the present disclosure, a method of operating a terminal in a communication system may include a process of receiving a first image, a process of generating a text prompt based on a first model from the first image, a process of evaluating the importance of a word included in the text prompt, and a process of transmitting the word to another terminal based on the importance of the word.

[0011] According to various embodiments of the present disclosure, in a communication system, a terminal includes a transceiver and a control unit operably connected to the transceiver, wherein the control unit receives a first image, generates a text prompt based on a first model from the first image, evaluates the importance of a word included in the text prompt, and transmits the word to another terminal based on the importance of the word.

[0012] Devices and methods according to various embodiments of the present disclosure enable fast and efficient data transmission while maintaining high similarity to the original image by reducing communication load and minimizing information loss through efficient conversion between text and images using a multimodal generative model.

[0013] In addition, the devices and methods according to various embodiments of the present disclosure can significantly improve data transmission efficiency and quality in various environments and conditions by optimizing the transmission order according to the information importance of each word through a sequential semantic transmission method, thereby enabling the target similarity level to be reached with a minimum number of transmission steps.

[0014] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure belongs from the description below.

[0015] FIG. 1 illustrates a schematic diagram for sequential semantic communication according to one embodiment of the present disclosure.

[0016] FIG. 2 illustrates an example of a word transmission method in sequential semantic communication according to one embodiment of the present disclosure.

[0017] FIG. 3 illustrates simulation results according to transmission methods according to various embodiments of the present disclosure.

[0018] FIG. 4 is a drawing showing a device configuration according to various embodiments of the present disclosure.

[0019] FIG. 5 illustrates an example applied to vehicle-to-vehicle communication according to one embodiment of the present disclosure.

[0020] The terms used in this disclosure are used only to describe specific embodiments and may not be intended to limit the scope of other embodiments. The singular expression may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by those of ordinary skill in the art described in this disclosure. Terms defined in general dictionaries among the terms used in this disclosure may be interpreted as having the same or similar meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined in this disclosure. In some cases, even if a term is defined in this disclosure, it cannot be interpreted to exclude embodiments of the present disclosure.

[0021] The terms used in this disclosure are used only to describe specific embodiments and may not be intended to limit the scope of other embodiments. The singular expression may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by those of ordinary skill in the art described in this disclosure. Terms defined in general dictionaries among the terms used in this disclosure may be interpreted as having the same or similar meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined in this disclosure. In some cases, even if a term is defined in this disclosure, it cannot be interpreted to exclude embodiments of the present disclosure.

[0022] The various embodiments of the present disclosure described below illustrate a hardware-based approach as an example. However, since the various embodiments of the present disclosure include techniques utilizing both hardware and software, the various embodiments of the present disclosure do not exclude a software-based approach.

[0023] Additionally, in the detailed description and claims of the present disclosure, “at least one of A, B, and C” can mean “only A,” “only B,” “only C,” or “any combination of A, B, and C.” Additionally, “at least one of A, B, or C” or “at least one of A, B, and / or C” can mean “at least one of A, B, and C.”

[0024] The present disclosure relates to a device and method for sequential semantic generative communication in a communication system. Specifically, the present disclosure describes a technique for reducing communication load and minimizing information loss through efficient conversion between text and images using a multimodal generative model in a communication system, thereby maintaining high similarity to the original image and enabling fast and efficient data transmission.

[0025] The terms used in the following description, including terms referring to signals, channels, control information, network entities, and device components, are provided for convenience of explanation. Therefore, the present disclosure is not limited to the terms described below, and other terms with equivalent technical meanings may be used.

[0026] Additionally, while this disclosure describes various embodiments using terminology used in certain communication standards (e.g., 3rd Generation Partnership Project (3GPP)), these are merely illustrative examples. The various embodiments of this disclosure can be easily modified and applied to other communication systems.

[0027] FIG. 1 illustrates a schematic diagram for sequential semantic communication according to one embodiment of the present disclosure.

[0028] Referring to Fig. 1, a transmitter for sequential semantic communication (hereinafter, referred to as a transmitter) can convert an image into a text prompt (101). Specifically, the transmitter (e.g., Alice) can convert an original image into a text prompt using an image-to-text model ('Img2Txt Gen' model). For example, the transmitter can input 'an image of a cat running through a field of flowers' and generate a text prompt 'A while cat running through a field of flowers'. Operation (101) may be an operation that converts the main visual elements of the image into text, thereby reducing the size of the transmission data and compressing the meaning.

[0029] In one embodiment, the image-to-text model may be an image-based text decoder using a transformer-based text prompt generation technique.

[0030] The transmitter can prioritize words for sequential transmission (103). Specifically, the words in the generated text prompt can be sorted by importance to sequentially transmit them. For example, the transmitter can use the 'Txt2Img Gen' model and the learned perceptual image patch similarity (LPIPS) metric to evaluate the importance of each word for image reconstruction.

[0031] In one embodiment, the LPIPS metric measures the visual similarity between an original image and a reconstructed image, allowing words with high similarity to be transmitted first. For example, in the sentence "A white cat running through a field of flowers," "cat" may be evaluated as the most important word and thus be transmitted first. In operation (103), the transmitter utilizes a "language model (LM)" to analyze the correlation between words and determine the order of words to be transmitted sequentially, starting with the word with the highest information content.

[0032] In one embodiment, the LPIPS metric represents a value from 0 to 1 and can express visual similarity.

[0033] The transmitter can transmit words with higher priority to the receiver in the order of the determined words (105). For example, the word "cat" can be transmitted as the first word.

[0034] In one embodiment, each word is transmitted independently, which reduces communication overhead and allows for the image to be reconstructed incrementally for each action.

[0035] A receiver (a "transmitter for sequential semantic communication") can reconstruct an image based on the received word (107). Specifically, the receiver can generate an image based on the received word using the 'Txt2Img Gen' model. For example, after receiving the word 'cat', the receiver can reconstruct an image of a cat based on the word. In operation (107), the receiver can generate an image visually similar to the original image using the transmitted word.

[0036] According to the embodiment of FIG. 1, the present disclosure enables efficient communication between a transmitter and a receiver, compressively conveying meaningful information through text to reconstruct a high-quality image similar to the original image. Furthermore, it reduces communication load and enables efficient data transmission while maintaining high similarity.

[0037] FIG. 2 illustrates an example of a word transmission method in sequential semantic communication according to an embodiment of the present disclosure. FIG. 2 presents, but is not limited to, the lowest LPIPS transmission (201), the most attentive transmission (202), and the least attentive transmission (203) as examples.

[0038] In the initial communication phase, the transmitter can assume knowledge of the receiver's text-to-image conversion processor to ensure that the transmitter transmits the word the receiver needs to generate as accurately as possible. Therefore, assuming both the transmitter and receiver cache and store the necessary models, the transmitter can store and cache the receiver's model. This allows the transmitter to predict the image Bob will generate based on the transmitted word.

[0039] In some embodiments, it may be appropriate for the transmitter to store only a limited number of additional model parameters in addition to the required image-to-text conversion model. For example, the transmitter may cache or store all or only a portion of the receiver's parameters. Depending on these circumstances, the transmitter may vary in how it selects the order of words to transmit.

[0040] When the transmitter caches (stores) the entire model of the receiver, the transmitter can fully utilize all of the receiver's capabilities. The transmitter can predict the receiver's generated images using the text it generates. Before transmitting a word, the transmitter can predict Bob's expected outcome using all words in the generated text prompt. That is, one embodiment of the present disclosure can be performed according to the lowest LPIPS transmission (201) method of FIG. 2.

[0041] If the transmitter caches (stores) part of the receiver's entire model, the transmitter can store only the language model (LM) and the attention module to determine the importance of words and the order in which they are transmitted.

[0042] Specifically, the receiver's text-to-image model may include text embedding (Language Model (LM)), image generation (U-Net), and post-processing (UAE) processes. The transmitter may cache only the LM among the receiver's models to evaluate the importance of words, and the attention module may interpret the text and evaluate the association between words. That is, using this method, one embodiment of the present disclosure may be performed according to at least one of the most attentive transmission (203) or the least attentive transmission (205) of FIG. 2.

[0043] In one embodiment, the attention value is calculated as the product of the attention weight and the value, and the attention weight can be expressed in the form of a matrix representing the relationship between each word.

[0044] Referring to FIG. 2, the objective image is an image (205), which can be expressed as a generated text prompt as a box (207) ('a white cat running through a field of flowers').

[0045] For Lowest LPIPS Transmission (201), the transmitter can simulate the expected image reconstruction results when each word is transmitted using the receiver's full model. In one embodiment, the process of simulating the image reconstruction results may be a process of performing simulations for all candidate words.

[0046] After simulating all words, the word that produces the image most similar to the original image can be selected and sequentially transmitted. In other words, minimum LPIPS transmission can be achieved by selecting and transmitting the word that minimizes the LPIPS value.

[0047] Afterwards, the transmitter repeatedly transmits words one by one, and the receiver that receives them repeatedly transmits words one by one to reconstruct the final image.

[0048] Through this, the minimum LPIPS transmission can ensure that the reconstructed image becomes progressively more similar to the original image as each word is transmitted.

[0049] For example, a minimal LPIPS transmission might consist of the sequential words "cat," "white," "running," and "flower." That is, by transmitting "cat" as the first word, the primary elements of the cat can be reconstructed. Subsequently, by sequentially transmitting the words "white," "running," and "flower," the cat's color, motion, and background elements can be incrementally added.

[0050] Most attentive transmission (203) corresponds to cases where the transmitter can only cache a portion of the receiver's text-to-image generation model. In such cases, it primarily uses LM to analyze word associations. Specifically, most attentive transmission calculates attention values ​​between each word and selects and transmits the most relevant word. This can be a method for quickly conveying meaning by prioritizing the transmission of words with the most significant meaning in a sentence.

[0051] Therefore, most attentive transmission (203) can calculate the attention value between words using a language model, select the most important word first and transmit it, and then sequentially transmit the word most related to the transmitted word.

[0052] For example, the most attentive transmission (203) may transmit words in the following order: 'cat', 'white', 'running', 'through', 'field', 'flowers', 'of', and 'a'. That is, the most important word, 'cat', may be transmitted first to reconstruct the main elements of the cat, and then the words 'white' and 'running', which are most closely related to 'cat', may be transmitted to add the color and movement of the cat. After that, the remaining words may be sequentially transmitted in order of their high relevance to the transmitted words to add background elements.

[0053] Least attentive transmission (205) corresponds to cases where the transmitter can only cache a portion of the receiver's text-to-image generation model, and in such cases, it primarily uses LM to analyze word-to-word associations. Specifically, most attentive transmission calculates an attention value between each word, transmitting the most significant word first and then selecting and transmitting the word with the least correlation to the first word thereafter. This can be a method of prioritizing the transmission of words with less significant meaning in a sentence, thereby transmitting less important information early in the transmission.

[0054] Therefore, the most attentive transmission (205) can calculate the attention value between words using a language model, select the most important word first and transmit it, and then sequentially transmit the word with the least correlation to the transmitted word.

[0055] For example, the least-attentive transmission might be 'cat', 'of', 'white', 'running', 'through', 'a', 'field', and 'flower'. Specifically, transmitting the most important word 'cat' first can reconstruct the main elements of a cat, and then transmitting less important words like 'of' and 'white' later can slow down the image reconstruction.

[0056] FIG. 3 illustrates simulation results according to transmission methods according to various embodiments of the present disclosure.

[0057] Referring to Figure 3, the lowest LPIPS transmission (301) has the lowest LPIPS value and can reach the target similarity level most quickly. The most attentive transmission (303) has a low LPIPS value, but may have a value slightly higher than the lowest LPIPS transmission method. The least attentive transmission (305) has the highest LPIPS value and may require many communication processes.

[0058] FIG. 4 is a diagram illustrating a device configuration according to various embodiments of the present disclosure. Referring to FIG. 4, the device of the present disclosure may include at least one processor (410), a memory (420), and a communication device (430) that is connected to a network and performs communication. In addition, the communication node (400) may further include an input interface device (440), an output interface device (450), a storage device (460), etc. Each component included in the device (400) may be connected by a bus (470) and communicate with each other.

[0059] However, each component included in the device (400) may be connected through an individual interface or individual bus centered around the processor (410), rather than a common bus (470). For example, the processor (410) may be connected to at least one of a memory (420), a communication device (430), an input interface device (440), an output interface device (450), and a storage device (460) through a dedicated interface.

[0060] The processor (410) can execute program commands stored in at least one of the memory (420) and the storage device (460). The processor (410) may refer to a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor in which methods according to embodiments of the present invention are performed. Each of the memory (420) and the storage device (460) may be configured with at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory (420) may be configured with at least one of a read-only memory (ROM) and a random access memory (RAM).

[0061] FIG. 5 illustrates an example applied to vehicle-to-vehicle communication according to one embodiment of the present disclosure.

[0062] Referring to FIG. 5, vehicles (20, 10, 30) can transmit real-time data through communication with each other.

[0063] In the case of autonomous vehicles, there is a need to quickly exchange information such as road conditions, traffic signals, and obstacles. In this case, converting image data into text and transmitting it can be useful for reducing the amount of data transmitted and increasing communication efficiency. For example, an image describing a specific road condition (e.g., an accident, road construction, etc.) can be converted into text and transmitted. For example, an image can be converted into text and sent as "A car accident at the intersection of Main St. and 1 st You can send text like "Ave".

[0064] In addition, important information in vehicle-to-vehicle communication needs to be transmitted with priority. According to an embodiment of the present disclosure, the most important information can be transmitted first, and secondary information can be transmitted later. For example, the first text can be "Accident Ahead," followed by secondary information such as "At the intersection of Main St. and 1 st Ave", "Two vehicles involved" can be transmitted.

[0065] Furthermore, in vehicle-to-vehicle communication, each vehicle may be of a different make and model, and may use a variety of sensors and data formats. In such cases, using a common text-based semantic representation can enhance interoperability.

[0066] In summary, the present disclosure can be applied to vehicle-to-vehicle communication, offering significant advantages in terms of data transmission efficiency, real-time information sharing, and improved interoperability. Applying this technology to vehicle-to-vehicle communication will significantly improve the safety and efficiency of autonomous vehicles.

[0067] The methods according to the embodiments described in the claims or specification of the disclosure may be implemented in the form of hardware, software, or a combination of hardware and software.

[0068] When implemented in software, a computer-readable storage medium storing one or more programs (software modules) may be provided. The one or more programs stored in the computer-readable storage medium are configured for execution by one or more processors within an electronic device. The one or more programs include instructions that cause the electronic device to execute methods according to the embodiments described in the claims or specification of the present disclosure.

[0069] These programs (software modules, software) may be stored in random access memory, non-volatile memory including flash memory, read only memory (ROM), electrically erasable programmable read only memory (EEPROM), magnetic disc storage devices, compact disc-ROM (CD-ROM), digital versatile discs (DVDs) or other forms of optical storage devices, magnetic cassettes, or may be stored in memories formed by a combination of some or all of these. In addition, each configuration memory may include multiple copies.

[0070] Additionally, the program may be stored on an attachable storage device that is accessible via a communication network, such as the Internet, an intranet, a local area network (LAN), a wide area network (WAN), a storage area network (SAN), or a combination thereof. Such a storage device may be connected to a device performing an embodiment of the present disclosure via an external port. Additionally, a separate storage device on the communication network may be connected to a device performing an embodiment of the present disclosure.

[0071] In the specific embodiments of the present disclosure described above, components included in the disclosure are expressed in the singular or plural form, depending on the specific embodiment presented. However, the singular or plural expressions are selected to suit the presented situation for convenience of explanation, and the present disclosure is not limited to singular or plural components. Components expressed in the plural form may be composed of singular elements, or components expressed in the singular form may be composed of plural elements.

[0072] While the detailed description of this disclosure has described specific embodiments, it should be understood that various modifications are possible without departing from the scope of this disclosure. Therefore, the scope of this disclosure should not be limited to the described embodiments, but should be defined not only by the scope of the claims described below, but also by equivalents thereof.

Claims

1. In the method of operating a terminal in a communication system, The process of receiving the first image, A process of generating a text prompt based on the first model from the first image, The process of evaluating the importance of words included in the above text prompt, A method comprising a process of transmitting the word to another terminal based on the importance of the word.

2. In claim 1, the second model is a model for predicting the first image by the word by the other terminal, The above second model is partially or completely stored in the terminal, 3. In claim 2, A process of predicting a second image based on the second model stored in the terminal, and A process for identifying the similarity between the first image and the second image, The process of selecting the word with the highest similarity, A method comprising a process of sequentially transmitting the highest word to the other terminal.

4. In claim 2, The process of predicting a second image by the word based on the second model stored in the terminal is as follows: The process of selecting and transmitting the most important word from the first image above, A method comprising a process of sequentially transmitting the most important words in order of high relevance.

5. In claim 2, The process of predicting a second image by the word based on the second model stored in the terminal is as follows: The process of selecting and transmitting the most important word from the first image above, A method comprising a process of sequentially transmitting the most important words in order of decreasing relevance.

6. In a terminal in a communication system, Transmitter and receiver, A control unit operably connected to the above transmitter and receiver, The above control unit, Receive the first image, Generate a text prompt based on the first model from the first image, Evaluate the importance of the words contained in the above text prompt, A device that transmits the word to another terminal based on the importance of the word.

7. In claim 6, the second model is a model for predicting the first image by the word by the other terminal, The above second model is a device that is partially or completely stored in the terminal.

8. In claim 7, the control unit predicts the second image by the word based on the second model stored in the terminal. Identifying the similarity between the first image and the second image, Select the word with the highest similarity, A device that sequentially transmits the highest word to the other terminal.

9. In claim 7, The control unit, based on the second model stored in the terminal, predicts a second image by the word. Select the most important word from the first image above and send it, A device comprising a process of sequentially transmitting the most important words in order of high relevance.

10. In claim 7, the control unit predicts the second image by the word based on the second model stored in the terminal. Select the most important word from the first image above and send it, A device that sequentially transmits the most important words in descending order of relevance.

Images