Device and method for orthogonal multiple access semantic communication via low-rank adaptation based fine-tuning

Abstract

An orthogonal multiple access deep semantic communication device and method through low-order adaptation-based fine-tuning are disclosed. A semantic communication method between a first terminal and a second terminal comprises the steps of: the first terminal extracting features (E(x)) from input data (x); the first terminal performing a predetermined operation on the extracted features (E(x)) to generate transmission data (); and the first terminal transmitting the transmission data through a wireless channel, wherein U1 is an orthonormal matrix stored in the first terminal, ∑A is a diagonal matrix generated as a result of Singular Value Decomposition on a fine-tuned weight matrix (A) of a pre-trained neural network model, and VA is an orthogonal matrix generated as a result of Singular Value Decomposition on the weight matrix (A), wherein the encoder of the fine-tuned neural network model is stored in the first terminal and the decoder of the fine-tuned neural network model is stored in the second terminal, and the extracted features (E(x)) are the It is characterized by being the output of an encoder.

Description

Technology Field

[0001] The present invention relates to wireless communication, and more specifically, to an apparatus and method for orthogonal multiple access deep semantic communication through fine-tuning based on low-rank adaptation (LoRA). Background Technology

[0002] Deep Semantic Communication, which is widely researched today, is based on Artificial Intelligence (AI)-based encoders and decoders. In this process, the problem of AI models being biased by their learning environments is a significant obstacle preventing the widespread adoption of such AI-based encoders and decoders. This invention aims to address this learning environment bias problem by analyzing the maximum size of learning environments that an AI model can accommodate based on mathematical modeling, and by configuring the model so that different learning environments do not influence each other, thereby creating an environment where AI-based encoders and decoders within the network can achieve optimal performance in real-world scenarios.

[0003] Through this invention, operation in multi-user networks, which is cited as a limitation of deep semantic communication, becomes possible. Furthermore, by enabling the derivation of a mathematical relationship between the parameter design of AI-based encoder / decoder models and the number of users that can be accommodated within the network, AI-based communication can be designed more systematically in the future. Prior art literature

[0004] Republic of Korea Published Patent No. 2024-0012829 (Published on Jan. 30, 2024)

[0005] Kwon Dong-seung, Na Ji-hyun, "Research Trends in Semantic Communications", ETRI Electronics and Telecommunications Trends, DOI: https: / / doi.org / 10.22648 / ETRI.2022.J.370608 The problem to be solved

[0006] The technical problem that the present invention aims to solve is to provide an apparatus and method for orthogonal multiple access deep semantic communication through fine-tuning based on Low-Rank Adaptation (LoRA). means of solving the problem

[0007] A semantic communication method between a first terminal and a second terminal according to an embodiment of the present invention comprises the step of the first terminal extracting a feature (E(x)) from input data (x), and the first terminal performing a predetermined operation on the extracted feature (E(x)) to transmit data ( The method includes the step of generating ), and the step of the first terminal transmitting the transmission data through a wireless channel, wherein U1 is an orthonormal matrix stored in the first terminal, and ∑ A is a diagonal matrix generated as a result of Singular Value Decomposition on the fine-tuned weight matrix (A) of a pre-trained neural network model, and said V A is an orthogonal matrix generated as a result of singular value decomposition for the weight matrix (A), and the first terminal stores an encoder of the finely tuned neural network model, and the second terminal stores a decoder of the finely tuned neural network model, and the extracted feature (E(x)) is the output of the encoder. Effects of the invention

[0008] According to the apparatus and method for orthogonal multiple access deep semantic communication through fine-tuning based on low-order adaptation according to an embodiment of the present invention, deep semantic communication in a multi-user network can be operated.

[0009] In addition, according to the embodiments of the present invention, a technique is proposed to offset signal interference between multiple users through the mathematical design of transmitted data, thereby maximizing the efficiency of communication resources within the network. Brief explanation of the drawing

[0010] Detailed descriptions of each drawing are provided to help to more fully understand the drawings cited in the detailed description of the present invention. FIG. 1 illustrates a communication system according to one embodiment of the present invention. FIGS. 2a to 2d are diagrams for explaining the setup process for semantic communication. FIG. 3 is a diagram for explaining the semantic communication process between the first terminal and the second terminal illustrated in FIG. 1. Figure 4 is a diagram illustrating a fine-tuning technique. Figure 5 is a diagram illustrating the singular value decomposition technique. Specific details for implementing the invention

[0011] Specific structural or functional descriptions of embodiments according to the concept of the present invention disclosed herein are provided merely for the purpose of explaining embodiments according to the concept of the present invention, and embodiments according to the concept of the present invention may be implemented in various forms and are not limited to the embodiments described herein.

[0012] Since embodiments according to the concept of the present invention may be subject to various modifications and may take various forms, embodiments are illustrated in the drawings and described in detail in this specification. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, and includes all modifications, equivalents, or substitutions that fall within the spirit and scope of the present invention.

[0013] Terms such as "first" or "second" may be used to describe various components, but said components should not be limited by said terms. For the sole purpose of distinguishing one component from another, for example, without departing from the scope of rights according to the concept of the present invention, the first component may be named the second component and similarly the second component may be named the first component.

[0014] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. Conversely, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Other expressions describing the relationships between components, such as "between" and "exactly between," or "adjacent to" and "directly adjacent to," should be interpreted in the same way.

[0015] The terms used herein are used merely to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “comprising” or “having” are intended to indicate the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described herein, and should not be understood as precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0016] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined herein.

[0017] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings attached to this specification. However, the scope of the patent application is not limited or restricted by these embodiments. Identical reference numerals in each drawing indicate identical components.

[0018] FIG. 1 illustrates a communication system according to one embodiment of the present invention.

[0019] Referring to FIG. 1, the communication system (10) may mean a wireless communication system for semantic communication and includes at least one first terminal (100), at least one second terminal (200), and a server (300). At least two of the first terminal (100), the second terminal (200), and the server (300) may exchange data or messages through a wired or wireless communication network. That is, the first terminal (100), the second terminal (200), and the server (300) may perform wired or wireless communication with each other. Additionally, the first terminal (100), the second terminal (200), and / or the server (300) may be implemented as a computing device including at least a processor and / or memory. Computing devices include PCs (Personal Computers), servers, smartphones, mobile phones, smart watches, smart glasses, etc.

[0020] The first terminal (100) and the second terminal (200) can obtain information for semantic communication (which may be referred to as background knowledge depending on the embodiment) through communication with the server (300). After the information for semantic communication is obtained, the first terminal (100) and the second terminal (200) can perform semantic communication via the server (300) according to the embodiment. Below, the configuration process for sharing information for semantic communication and the communication process for performing semantic communication are described separately.

[0021] First, we will explain the setup process for sharing information for semantic communication. This setup process may also be referred to as the wireless fine-tuning process.

[0022] FIGS. 2a to 2d are drawings for explaining the setup process for semantic communication. In the following, it is assumed that Alice and Carol are included in the first terminal (100), and Bob and David are included in the second terminal (200). Additionally, a situation is considered in which Alice transmits a predetermined message to David and Carol transmits a predetermined message to Bob. However, the scope of the present invention is not limited thereto, and the number of the first terminal (100) and the number of the second terminal (200) may vary.

[0023] Additionally, it is assumed that each terminal stores an encoder or a decoder included in a neural network model that has been fine-tuned with respect to a pre-trained encoder / decoder structure. More specifically, the encoder may be stored in a first terminal (100) that transmits a message, and the decoder may be stored in a second terminal (200) that receives a message. Furthermore, among the parties exchanging messages, the encoder is stored in the sender and the decoder is stored in the receiver, wherein, according to the embodiment, the encoder and the decoder may refer to a model that has been fine-tuned after pre-training in an end-to-end manner. Thus, the encoder stored in Alice and the decoder stored in David correspond to each other, and the encoder stored in Carol and the decoder stored in Bob correspond to each other.

[0024] Referring to FIG. 2a, the server (300) generates arbitrary U and V matrices in the form of orthonormal, and generates U1, U2, V1, and V2 by separating rows equal to the number of users within the generated U and V matrices. In this example, since the number of first terminals (100) participating in semantic communication is 2, the server (300) separates each of the U and V matrices into 2 matrices. Additionally, the server (300) transmits U1 and V1 to Alice and transmits U2 and V2 to Carol.

[0025] Afterward, Alice and Carol perform calculations on the data received from the server (300) and then send the calculation result to the server (300). Specifically, Alice Sends to the server (300), and Carol The signal transmitted by Alice and Carol through the wireless channel is added wirelessly, and this process is called the Air-computation process.

[0026] When looking at Fig. 2b, the server (300) transmits the signal received from Alice and Carol (which may mean a signal formed by combining the signal transmitted by Alice and the signal transmitted by Carol wirelessly) back to Alice and Carol.

[0027] Afterwards, Alice goes to the server (300). Carol is the server (300). Transmits.

[0028] Referring to FIG. 2c, the server (300) can mathematically calculate each key value based on the received data and deliver it to each user. Specifically, the server (300) to Alice to Carol to Bob and to David and Delivers.

[0029] In FIG. 2d, data (which may be named as information or key value) obtained by the first terminal (100) and the second terminal (200), respectively, is illustrated.

[0030] FIG. 3 is a diagram illustrating the semantic communication process between the first terminal and the second terminal illustrated in FIG. 1. In FIG. 3, the process of Alice transmitting certain data to David is described.

[0031] In ①, Alice can extract features ((x)) corresponding to input data (x). The extracted features may be named as feature vectors, latent vectors, or latent variables, etc., as the output of an encoder stored in Alice. Additionally, the input data may include at least one of voice data, image data, video data, and text data.

[0032] In ② and ③, Alice performs calculations using the data she possesses. First, regarding the feature (E(x)): The result of multiplying by Multiply by. The result of the operation in ③ is am.

[0033] In ④, since the signal transmitted by Alice through the wireless channel is added to the signal transmitted by Carol through the wireless channel, This becomes. Here, E(y) can represent the features extracted by Carol.

[0034] In ⑤, David [is on] the received signal (the signal derived in ④). Multiply by. The result of the multiplication operation is It becomes.

[0035] Therefore, David can recover the data through the process of ⑥. That is, David can obtain the inference result by inputting the result of ⑤ into the decoder.

[0036] Figure 4 is a diagram illustrating a fine-tuning technique.

[0037] In the present invention, a pre-trained neural network model is fine-tuned using any one of the Parameter Efficient Fine-Tuning (PEFT) techniques. An exemplary PEFT technique may be Low-Rank Adaptation (LoRA). Although LoRA is a technique generally utilized in Large Language Models (LLMs), in the present invention, LoRA is applied to a general neural network model to learn only specific parameters. Through this, the linear algebraic properties exhibited by the fine-tuned specific parameters can be utilized.

[0038] As illustrated in FIG. 4, the neural network model may include multiple neural network layers and multiple activation functions (e.g., RELu). In this case, fine-tuning can be performed by combining a LoRA layer in parallel with any one of the multiple neural network layers included in the pre-trained neural network model and learning only the weights constituting the LoRA layer. Additionally, the neural network layer to which the LoRA layer is combined may be a linear layer.

[0039] If PEFT is not applied, the learning parameters are as follows, and therefore m×n parameters must be learned.

[0040]

[0041] However, in the case of PEFT, the learning parameters are as follows, and thus r × (m + n) parameters are learned. Here, if r is smaller than m, the learning parameters can be reduced.

[0042]

[0043] Next, the Singular Value Decomposition (SVD) technique is explained.

[0044] Figure 5 is a diagram illustrating the singular value decomposition technique.

[0045] Referring to FIG. 5, an arbitrary matrix (A) can be separated into three matrices U, ∑, and V through Singular Value Decomposition (SVD). That is, when separated into the form shown in FIG. 5, Singular Value Decomposition (SVD) can be understood as expressing a single matrix as the product of several orthogonal matrices. Through this, there are multiple spaces within a single matrix, and when data is inserted into these spaces, the data can be configured so that they do not affect each other. In the present invention, we propose an orthogonal multiple access (semantic) communication technique utilizing these features.

[0046] In Fig. 5, A can represent an arbitrary rectangular matrix of size (m×n), U can represent an orthogonal matrix of size (m×m), ∑ can represent a diagonal matrix of size (m×n), and V can represent an orthogonal matrix of size (n×n). In short, through singular value decomposition, matrix A can be expressed as the sum of m orthogonal matrices. Additionally, matrix A can represent the weight matrix of a LoRA layer.

[0047] The device described above may be implemented as a set of hardware components, software components, and / or a combination of hardware components and software components. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an Arithmetic Logic Unit (ALU), a Digital Signal Processor, a microcomputer, a Field Programmable Array (FPA), a Programmable Logic Unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0048] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave in order to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0049] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either individually or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; ROMs; RAMs; flash memory; and hardware devices specifically configured to store and execute program instructions. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0050] The present invention has been described with reference to the embodiments illustrated in the drawings, but this is merely illustrative, and those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents. Accordingly, the true technical scope of protection of the present invention should be determined by the technical spirit of the appended claims. Explanation of the symbols

[0051] 10: Communication System 100 : First terminal 200 : Second terminal 300 : Server

Claims

Claim 1 In a method for semantic communication between a first terminal and a second terminal, the first terminal extracts a feature (E(x)) from input data (x); the first terminal performs a predetermined operation on the extracted feature (E(x)) to transmit data ( The step of generating ); and the step of the first terminal transmitting the transmission data through a wireless channel, wherein U1 is an orthonormal matrix stored in the first terminal, and the ∑ A is a diagonal matrix generated as a result of Singular Value Decomposition on the fine-tuned weight matrix (A) of a pre-trained neural network model, and the above V A A semantic communication method in which is an orthogonal matrix generated as a result of singular value decomposition of the weight matrix (A), an encoder of a finely tuned neural network model is stored in the first terminal, a decoder of the finely tuned neural network model is stored in the second terminal, and the extracted feature (E(x)) is the output of the encoder. Claim 2 In claim 1, the fine-tuned neural network model is a semantic communication method fine-tuned through PEFT (Parameter Efficient Fine-Tuning). Claim 3 In paragraph 2, the above PEFT is a semantic communication method that is a Low-Rank Adaptation (LoRA). Claim 4 In paragraph 3, the finely tuned weight matrix (A) is the weight matrix of the LoRA layer, a semantic communication method. Claim 5 In paragraph 4, the above-mentioned predetermined operation is And, the above U2 is a matrix in the form of an orthonormal matrix stored in a third terminal that is multiplely connected to the above first terminal and transmits a predetermined message, and the above ∑ C is a diagonal matrix generated as a result of singular value decomposition for the fine-tuned weight matrix (C) of a pre-trained neural network model stored in the third terminal, and V C A semantic communication method in which is an orthogonal matrix generated as a result of singular value decomposition for the weight matrix (C), V1 is an orthonormal matrix stored in the first terminal, and V2 is an orthonormal matrix stored in the third terminal. Claim 6 In paragraph 5, the step of the second terminal receiving a signal in which the signal transmitted by the third terminal is added to the signal transmitted by the first terminal due to air-computation; and the second terminal, to the received signal A semantic communication method further comprising: a step of generating an input signal of a decoder by multiplying; and a step in which the second terminal decodes the input signal of the decoder using the decoder.

Images