Method and apparatus for responding to transmission error in split processing
The method addresses transmission errors in split processing by using error detection codes and error learning layers to ensure accurate neural network operations in communication systems, effectively maintaining data integrity and accuracy.
Patent Information
- Application Number
- PCT/KR2024/004306
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-12
- Filing Date
- 2024-04-03
- Publication Date
- 2024-11-21
AI Technical Summary
In split processing between a terminal and a server in communication systems, transmission errors in intermediate data during neural network learning or inference processes are not effectively detected and mitigated, leading to data distortion and loss of information.
A method and device that obtain setting information about the transmission format of intermediate data, attach error detection codes to result values, and transmit these to a server for error checking, where errors are detected and processed using learned layers to derive final results, ensuring data integrity and accuracy.
This approach effectively detects and mitigates transmission errors in split processing, maintaining data integrity and ensuring accurate neural network operations by using error detection codes and error learning layers to correct and process distorted data.
Smart Images

Figure KR2024004306_21112024_PF_FP_ABST
Abstract
Description
Method and device for responding to transmission errors in split processing
[0001] Embodiments of the present disclosure relate to a method and apparatus for responding to a transmission error in split processing between a user equipment (UE) and a server in a communication system.
[0002] 5G mobile communication technology defines a wide frequency band to enable fast transmission speeds and new services, and can be implemented not only in the sub-6GHz frequency band such as 3.5 gigahertz (3.5GHz), but also in the ultra-high frequency band called millimeter wave (mmWave) such as 28GHz and 39GHz ('Above 6GHz'). In addition, for 6G mobile communication technology, which is called the system after 5G communication (Beyond 5G), implementation in the terahertz band (for example, the 3 terahertz (3THz) band at 95GHz) is being considered to achieve a transmission speed that is 50 times faster than 5G mobile communication technology and an ultra-low latency time that is reduced to one-tenth.
[0003] In the early stages of 5G mobile communication technology, the goal was to support services and satisfy performance requirements for enhanced Mobile Broadband (eMBB), Ultra-Reliable Low-Latency Communications (URLLC), and massive Machine-Type Communications (mMTC), including beamforming and massive MIMO to mitigate path loss of radio waves in ultra-high frequency bands and increase the transmission distance of radio waves, support for various numerologies for efficient use of ultra-high frequency resources (such as operation of multiple subcarrier intervals), dynamic operation of slot formats, initial access technology to support multi-beam transmission and wideband, definition and operation of BWP (Bidth Part), new channel coding methods such as LDPC (Low Density Parity Check) codes for large-capacity data transmission and Polar Code for reliable transmission of control information, and L2 pre-processing (L2 Standardization has been made for network slicing, which provides dedicated networks specialized for specific services, and pre-processing.
[0004] Currently, discussions are underway to improve and enhance the initial 5G mobile communication technology in consideration of the services that 5G mobile communication technology was intended to support, and physical layer standardization is in progress for technologies such as V2X (Vehicle-to-Everything) to help autonomous vehicles make driving decisions and increase user convenience based on their own location and status information transmitted by vehicles, NR-U (New Radio Unlicensed) for the purpose of system operation that complies with various regulatory requirements in unlicensed bands, NR terminal low power consumption technology (UE Power Saving), Non-Terrestrial Network (NTN), which is direct terminal-satellite communication to secure coverage in areas where communication with terrestrial networks is impossible, and Positioning.
[0005] In addition, standardization of wireless interface architecture / protocols is in progress for technologies such as intelligent factories (Industrial Internet of Things, IIoT) to support new services through linkage and convergence with other industries, Integrated Access and Backhaul (IAB) that provides nodes for expanding network service areas by integrating wireless backhaul links and access links, Mobility Enhancement technology including Conditional Handover and Dual Active Protocol Stack (DAPS) handover, and 2-step random access (2-step RACH for NR) that simplifies random access procedures. Standardization is also in progress for system architecture / services such as 5G baseline architecture (e.g., Service-based Architecture, Service-based Interface) for grafting Network Functions Virtualization (NFV) and Software-Defined Networking (SDN) technologies, and Mobile Edge Computing (MEC) that provides services based on the location of the terminal.
[0006] Once these 5G mobile communication systems are commercialized, an explosive increase in connected devices will be connected to the communication network, necessitating enhanced functionality and performance of 5G mobile communication systems and integrated operation of these connected devices. To this end, new research will be conducted on improving 5G performance and reducing complexity, supporting AI services, supporting metaverse services, and drone communications by utilizing eXtended Reality (XR), Artificial Intelligence (AI), and Machine Learning (ML) to efficiently support Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR).
[0007] In addition, the development of these 5G mobile communication systems includes new waveforms to ensure coverage in the terahertz band of 6G mobile communication technology, multi-antenna transmission technologies such as Full Dimensional MIMO (FD-MIMO), Array Antenna, and Large Scale Antenna, metamaterial-based lenses and antennas to improve the coverage of terahertz band signals, high-dimensional spatial multiplexing technology using Orbital Angular Momentum (OAM), Reconfigurable Intelligent Surface (RIS) technology, as well as full duplex technology to improve the frequency efficiency and system network of 6G mobile communication technology, satellite, AI (Artificial Intelligence) from the design stage and AI-based communication technology that realizes system optimization by internalizing end-to-end AI support functions, and ultra-high-performance communication and computing resources to provide services with complexity that exceeds the limits of terminal computing capabilities. It can serve as a basis for the development of next-generation distributed computing technologies that can be realized by utilizing them.
[0008] Meanwhile, as the network to which the neural network model is applied expands, split processing can be applied between multiple processing devices, for example, terminals and servers, in learning and inference of the neural network.
[0009] Embodiments of the present disclosure may relate to a method and apparatus for providing split processing between a terminal and a server in a communication system.
[0010] Embodiments of the present disclosure can provide a method and device for dealing with transmission errors between split processing devices.
[0011] Embodiments of the present disclosure may provide a method and device for detecting and / or mitigating transmission errors in split processing between a terminal and a server.
[0012] Embodiments of the present disclosure may provide a method and device for detecting and mitigating transmission errors of intermediate data during a learning or inference process between split processing devices.
[0013] In a communication system according to one embodiment of the present disclosure, a method of operating a terminal may include an operation of obtaining configuration information on a transmission format of intermediate data by segmented processing of a neural network, an operation of generating result values through processing of at least one first layer among a plurality of layers constituting the neural network, an operation of generating the intermediate data by attaching error detection codes for the result values to the result values based on the configuration information, and an operation of transmitting the generated intermediate data to a server including at least one subsequent second layer among the plurality of layers.
[0014] In a communication system according to one embodiment of the present disclosure, a method of operating a server may include: an operation of obtaining configuration information on a transmission format of intermediate data by segmented processing of a neural network; an operation of receiving intermediate data including result values through processing of at least one first layer among a plurality of layers constituting the neural network and error detection codes for the result values from a terminal; an operation of checking whether the intermediate data contains an error based on the error detection codes; an operation of processing the result values of the intermediate data by a second layer learned to correspond to the first layer when an error is detected in the intermediate data; and an operation of processing an output of the second layer by at least one third layer subsequent to the first layer among the plurality of layers to derive a final result.
[0015] A terminal of a communication system according to one embodiment of the present disclosure may include a transceiver and a processor connected to the transceiver. The processor may be configured to obtain configuration information regarding a transmission format of intermediate data through segmented processing of a neural network, generate result values through processing of at least one first layer among a plurality of layers constituting the neural network, attach error detection codes for the result values to the result values based on the configuration information to generate the intermediate data, and transmit the generated intermediate data to a server including at least one subsequent second layer among the plurality of layers through the transceiver.
[0016] A server of a communication system according to one embodiment of the present disclosure may include a network interface and a processor connected to the network interface. The processor may be configured to obtain configuration information on a transmission format of intermediate data by segmented processing of a neural network, receive intermediate data including result values through processing of at least one first layer among a plurality of layers constituting the neural network and error detection codes for the result values from a terminal, determine whether the intermediate data contains an error based on the error detection codes, and, if an error is detected in the intermediate data, process the result values of the intermediate data by a second layer trained to correspond to the first layer, and process the output of the second layer by at least one third layer subsequent to the first layer among the plurality of layers to derive a final result.
[0017] FIG. 1 illustrates the structure of a communication system according to one embodiment of the present disclosure.
[0018] FIG. 2 is a diagram for explaining split processing between a terminal and a server according to one embodiment.
[0019] FIG. 3 illustrates a sequence flow diagram illustrating a procedure for dividing a neural network model and processing it in multiple processing devices according to one embodiment of the present disclosure.
[0020] FIG. 4 is a diagram for explaining a system structure based on 5GMS (media streaming) according to one embodiment of the present disclosure.
[0021] FIG. 5 illustrates a 5G-based real-time media communication structure supporting segmented processing according to one embodiment of the present disclosure.
[0022] FIG. 6A and FIG. 6B are diagrams for explaining a data structure according to one embodiment of the present disclosure.
[0023] FIGS. 7A and 7B are diagrams illustrating a data structure including a partition element according to one embodiment.
[0024] FIG. 8a, FIG. 8b, FIG. 8c, and FIG. 8d are diagrams illustrating transmission of serialized data according to one embodiment of the present disclosure.
[0025] FIG. 9 illustrates multiple layers of a neural network model according to one embodiment of the present disclosure.
[0026] FIG. 10 illustrates split processing in multiple processing devices according to one embodiment of the present disclosure.
[0027] FIG. 11 is a diagram for explaining a transmission error response in split processing according to one embodiment of the present disclosure.
[0028] FIG. 12 is a diagram illustrating an error learning layer learned with normal values, error values, and network status according to one embodiment.
[0029] FIG. 13 is a block diagram showing a configuration of a terminal in a communication system according to one embodiment of the present disclosure.
[0030] FIG. 14 is a block diagram showing a configuration of a server in a communication system according to one embodiment of the present disclosure.
[0031] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.
[0032] In describing this disclosure, descriptions of technical details that are well-known in the technical field to which this disclosure pertains and are not directly related to this disclosure will be omitted. This is to avoid obscuring the gist of this disclosure by omitting unnecessary explanations and to convey it more clearly. Furthermore, the terms described below are defined based on their functions in this disclosure and may vary depending on the intent or custom of the user or operator. Therefore, their definitions should be based on the contents of this specification as a whole.
[0033] For the same reason, some components in the attached drawings are exaggerated, omitted, or schematically depicted. Furthermore, the dimensions of each component do not entirely reflect its actual size. Identical or corresponding components in each drawing are assigned the same reference numbers.
[0034] The advantages and features of the present disclosure, and methods for achieving them, will become clearer with reference to the embodiments described below in detail together with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms. These embodiments are provided only to ensure that the disclosure of the present disclosure is complete and to fully inform those skilled in the art of the scope of the disclosure, and the present disclosure is defined only by the scope of the claims. Like reference numerals designate like elements throughout the specification. In addition, when describing the present disclosure, if a specific description of a related function or configuration is determined to unnecessarily obscure the gist of the present disclosure, the detailed description thereof will be omitted. In addition, the terms described below are terms defined in consideration of the functions of the present disclosure, and may vary depending on the intention or custom of the user or operator. Therefore, the definitions should be made based on the contents throughout the specification.
[0035] In addition, although LTE, LTE-A, or 5G systems may be described below as examples, embodiments of the present disclosure may also be applied to other communication systems having similar technical backgrounds or channel types. For example, this may include 5G-Advance or NR-Advance, or 6th generation mobile communication technology (6G) developed after 5G mobile communication technology (or new radio, NR), and the 5G described below may also include existing LTE, LTE-A, and other similar services. In addition, the present disclosure may be applied to other communication systems with some modifications within a range that does not significantly deviate from the scope of the present disclosure, as determined by a person having skilled technical knowledge.
[0036] At this time, it will be understood that each block of the processing flowchart drawings and combinations of the flowchart drawings can be performed by computer program instructions. These computer program instructions can be installed in a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing equipment, so that the instructions executed by the processor of the computer or other programmable data processing equipment create a means for performing the functions described in the flowchart block(s). These computer program instructions can also be stored in a computer-available or computer-readable memory that can direct a computer or other programmable data processing equipment to implement the functions in a specific manner, so that the instructions stored in the computer-available or computer-readable memory can also produce a manufactured item that includes an instruction means for performing the functions described in the flowchart block(s). Since the computer program instructions may be installed on a computer or other programmable data processing device, a series of operational steps may be performed on the computer or other programmable data processing device to create a computer-executable process, and the instructions that cause the computer or other programmable data processing device to perform the steps for performing the functions described in the flowchart block(s) may also provide steps for performing the functions described in the flowchart block(s).
[0037] Additionally, each block may represent a module, segment, or portion of code that contains one or more executable instructions for performing a specific logical function(s). It should also be noted that in some alternative implementation examples, the functions described in the blocks may occur out of order. For example, two blocks depicted in succession may actually be executed substantially concurrently, or the blocks may sometimes be executed in reverse order, depending on their respective functions.
[0038] Here, the term '~ part' used in this embodiment means software or hardware components such as FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit), and the '~ part' performs certain roles. However, the '~ part' is not limited to software or hardware. The '~ part' may be configured to be on an addressable storage medium or may be configured to play one or more processors. Therefore, as an example, the '~ part' includes components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and '~ parts' may be combined into a smaller number of components and '~ parts' or further separated into additional components and '~ parts'. Additionally, the components and '~parts' may be implemented to activate one or more CPUs within a device or secure multimedia card. In addition, in an embodiment, the '~parts' may include one or more processors.
[0039] FIG. 1 illustrates the structure of a communication system according to one embodiment of the present disclosure.
[0040] Referring to FIG. 1, a communication system may include a user equipment (UE) (110) configured to support wireless or wired communication and at least one server (120) that provides communication services to the UE (110) via a network (N / W) (100) (e.g., a 5G network). The UE (110) may access a wireless network (e.g., the network (100)) via a base station (e.g., node B) (not shown). The server (120) may provide services to the UE (110) via at least one network entity of a core network (e.g., the network (100)).
[0041] FIG. 2 is a diagram for explaining split processing between a terminal and a server according to one embodiment.
[0042] Referring to FIG. 2, terminal processing (210) within terminal (110) may include one or more layers (e.g., a first layer and a second layer), and each layer may include a plurality of nodes. A result of terminal processing (210) (e.g., a result value (202)) may be transmitted to a server (120) via a network (200). The result may be input to server processing (220) that includes one or more layers (e.g., a third layer, a fourth layer, and a fifth layer). The result may be processed by a plurality of nodes within each layer.
[0043] After terminal processing (210), when the result (e.g., result value (202)) is transmitted to the server (120) via the network (200), data distortion (e.g., error value (206)) due to a transmission error (204) may occur. For example, when the result (e.g., result value (202)) performed on the terminal (110) is transmitted to the server (120), the calculation result values of some nodes (e.g., at least one node in the second layer) may not be transmitted, or may be changed to a completely different value (e.g., error value (206)) due to distortion of some bits. When the distorted result is applied to nodes in subsequent layers (e.g., third to fifth layers), it will affect the overall calculation value and derived result in the server processing (220).
[0044] In a neural network, if nodes in some layers are disconnected, the performance of the model network can be affected. When a node is disconnected, it no longer receives input from nodes in previous layers and no longer contributes to the output of the current layer. This leads to information loss and an inability to learn the underlying patterns in the data.
[0045] As illustrated in FIG. 2, if a transmission error (204) occurs in the result (e.g., result value (202)) transmitted after terminal processing (210), the output data (e.g., error value (206)) of the node where the transmission error (204) occurred will affect all subsequent layers (e.g., the third to fifth layers of server processing (220).
[0046] De-Facto specifications (e.g. H5) and standard documents (e.g. neural network exchange format (NNEF) and open neural network exchange (ONNX)) define how to store the layer structure and node-specific weight values of a neural network model. Intermediate data between layers may require a standard for storing it because it is generated and consumed during computation within a processor (e.g., a GPU (graphics processing unit). Network models such as NNEF can store data in two file formats: graph and dat. Graph can describe the structure of the data. Dat can be composed of data values in a format that can be dumped to memory for fast loading. However, NNEF does not support the transmission of intermediate data as mentioned above. Since the Dat format describes data values consecutively, when the Dat format is used, the receiving device cannot know which part of the intermediate data has an error. When the data of a node detected as having an error has a certain value, the value may be a value that has not been learned yet, or it may act as an excessive outlier in inference and learning, distorting the overall result.
[0047] In neural network training and inference, since the connections between layers are made within the GPU and its memory area, storage methods during transmission or methods for responding to transmission errors may not be considered. However, when the model is divided and processed and intermediate results (e.g., intermediate data) are transmitted through a network (e.g., network (100 or 200)), a method for transferring data (e.g., intermediate data) stored in the memory of one processing unit (e.g., GPU) to another processing unit (e.g., CPU (central processing unit)) for storage, a method for transmitting the data, and a method for verifying the integrity of the received data and handling errors may be required.
[0048] In embodiments of the present disclosure, a neural network model may be defined as a set of multiple parallel or serially connected layers. A layer may be defined as including one or more nodes. A node may be defined as including one or more component(s) of a multidimensional array. A component may be defined as having a weight. A weight may be defined as a value that is initialized to a random or specified value and then modified through learning. The output value of a layer (e.g., intermediate data) may be defined as a calculated value derived through an operation between the output values of the nodes of the previous layer and the weights of the nodes of the layer.
[0049] Embodiments of the present disclosure provide synchronization and serialization for transmitting intermediate data, which is a result of processing by one processing device, to another processing device when processing in a network of a neural network model is split into one or more and performed by each of a plurality of processing devices (e.g., a terminal and a server), and the connection between the devices is based on a mobile communication network.
[0050] Embodiments of the present disclosure provide a method for a processing device that receives intermediate data stored and transmitted serially to determine whether a value described in the intermediate data is normal or has changed due to an error that occurred during transmission.
[0051] Embodiments of the present disclosure provide a method for responding to error values when it is determined that an error has occurred in intermediate data.
[0052] One embodiment of the present disclosure may provide a service negotiation and processing procedure for dividing the processing of a neural network model in a network (e.g., a mobile communication network) and processing it across multiple processing devices.
[0053] In the present disclosure, a network of a neural network model can be defined as a set of multiple parallel or serially connected layers. A layer can be defined as including one or more nodes. A node can be defined as including one or more component(s) of a multidimensional array. A component can be defined as having a bias, an activation function, and / or a weight. A weight can be defined as a value that is initialized to a random or specified value and then modified through learning. The output value of a layer (e.g., intermediate data) can be defined as a calculated value derived through an operation between the output values of the nodes of the previous layer and the weights of the nodes of the layer.
[0054] Split processing for a neural network model can involve splitting a neural network model into one or more pieces, transferring the split model pieces to one or more processing devices, performing processing on the one or more processing devices, and connecting the processing by transferring intermediate data via a local or mobile network. Processing devices supporting the split processing of the present disclosure may include, for example, a mobile terminal or a cloud / edge server.
[0055] FIG. 3 illustrates a sequence flow diagram illustrating a procedure for dividing a neural network model and processing it across multiple processing devices according to one embodiment of the present disclosure. The procedure described below may include service negotiation and processing in a mobile communications network (e.g., a 5G system).
[0056] Referring to FIG. 3, in operation 301, an application provider (AP) may communicate with an application function (AF) of a 5G system to transmit information about a service to be provided. The AF may create an application server (AS) (e.g., server (120)) or prepare resources for service provision in an already created AS. The resources may include, for example, a storage for a neural network model and a computational device for model processing. After the resource allocation and preparation of the AS is completed, the AF may transmit information related to the resources (e.g., provisioning information) to the AP.
[0057] In operation 302, the AP can transmit information related to the neural network model to the AS. The AS can ingest information about a splittable AI model from the information.
[0058] In operation 303, a user equipment (UE) may execute an application ('App'), and the App may perform split AIML (artificial intelligence and machine learning) processing service discovery to find a split processing service for a neural network model for desired learning or inference by communicating with an AP, thereby receiving information for communicating with an AF to receive the service.
[0059] In operation 304, the App invokes the API (application programmable interface) of the AIML processing unit (APU) within the terminal to perform a split neural network processing service. The APU communicates with the AF via the media session handler (MSH) to negotiate with the AF the components of serialization for exchanging intermediate data generated during split learning or split inferencing with the AS, as well as information regarding the transmission session.
[0060] As a result of the above negotiation in operation 305, the MSH may receive information for downloading a neural network model (e.g., downlink position information) and / or information for transmitting serialized data (e.g., uplink position information), and may transmit the received information to the APU.
[0061] In operation 306, the APU may receive (e.g., download) configuration information (e.g., at least some of the AI model information such as ) for a neural network model to be implemented in a terminal (UE) from the AS.
[0062] In operation 307, the App can receive instructions from the user for learning or inference (e.g., 'Request query').
[0063] At operation 308, the App can instruct the APU to process the neural network model on the terminal (e.g., the first and second layers (L1, L2)).
[0064] In operation 309, the APU may transmit intermediate data containing result values, which are outputs of processing, to a serializer. The serializer may serialize the intermediate data to generate serialized data (e.g., intermediate data in a serialized format). In one embodiment, the serialized data may include the result values of the processing and corresponding error detection codes.
[0065] The serialized data generated by the serializer in operation 310 can be transmitted to the AS via the mobile communication network.
[0066] In operation 311, the AS may deserialize the received serialized data. Deserialization may include parsing the result value according to the layer structure of the network and error detection using an error detection code.
[0067] In operation 312, if an error is found through the deserialization, the AS may receive status information related to the network status from the 5GC (5G core network) (e.g., at least one network entity) through the AF. The status information (e.g., NWDAF report and / or EVEX report) related to the network status provided by the 5GC of 3GPP according to a standard such as NWDAF (network data analytics function) or EVEX (event exposure) may include at least one of user data traffic and patterns, location, mobility and usage of user terminals (e.g., terminals (110)), network infrastructure and service status, network and service bottlenecks, QoS (Quality of Service) information, RAN (Radio Access Network) and core network status, real-time traffic analysis data, network security status, failure diagnosis and prevention data, or data analysis information for a machine learning algorithm. An error learning layer learned based on at least one of the information included in the status information may be selected for neural network model processing in the AS.
[0068] In operation 313, the selected error learning layer is used, and the output of the error learning layer can be applied as the input of the next layer. In one example, when the first layer (L1) and the second layer (L2) of the neural network model are executed in the terminal (110), the error learning layer used in the AS can be the L2-prime layer that predicts the normal value for the second layer (L2). The L2-prime layer can process the result values included in the serialized data before L3. The output of the L2-prime layer can be passed on to the next layers (for example, the fourth layer (L4) and the fifth layer (L5)) through the third layer (L3), and the final result can be derived by the layers (L3, L4, L5).
[0069] If it is determined that no error occurred in the transmission process of the serialized data received in the deserialization of operation 311, the deserialized data (e.g., output data of L2) is transmitted to the next layers (L4 and L5) through the third layer (L3) in operation 314, and the final result can be derived.
[0070] The final result (e.g., inference result) derived through operation 313 or operation 314 in operation 315 can be returned to the APU of the terminal (110) by the AS.
[0071] The above operations may be based on the system architecture of TS 26.501, which defines 5G media streaming (5GMS) of 3GPP system aspects 4 (SA4).
[0072] FIG. 4 is a diagram for explaining a 5GMS-based system structure according to one embodiment of the present disclosure.
[0073] Referring to FIG. 4, a first processing device (410) (e.g., terminal (110)) may include at least one of an App, an MSH, an APU in charge of neural network processing, a serializer, an error detection code generation unit, or a segmented neural network model storage. A second processing device (420) (e.g., server (120) or AS) may include at least one of an APU in charge of neural network processing, a deserializer, an error detection code determination unit, or a segmented neural network model storage.
[0074] Processing devices supporting segmented processing (e.g., the first processing device (410) and the second processing device (420)) can share information about the structure and weight values of the segmented neural network model that the device is in charge of processing among the entire neural network model through service provisioning by the App or session configuration via MSH. Service provisioning can be performed between the service provider (430) and the App of the first processing device (410). Session configuration can be performed between the AF (450) (e.g., at least one network entity of 5GC) and the MSH of the first processing device (410).
[0075] FIG. 5 illustrates a 5G-based real-time media communication architecture supporting segmented processing according to one embodiment of the present disclosure. The illustrated real-time media communication architecture may be based on, for example, TS 26.506.
[0076] Referring to FIG. 5, a terminal (UE) (510) (e.g., terminal (110)) may include an App (e.g., a native WebRTC App) that performs web RTC (real-time communication), a web App, and a media access function. The media access function may include an MSH, a web RTC framework, and a codec. The MSH may communicate with an AF (540) that supports 5G-RTC via an RTC-5 interface.
[0077] The 5G-RTC AF (540) may include a network support (NS) function (NS-AF), a configuration function, and a provisioning function, and may communicate with a policy control function (PCF), a network exposure function (NEF), and a session management function (SMF) of a 5G network. The provisioning function may communicate with a 5G RTC application provider (530) (e.g., an AP) via an RTC-1 interface.
[0078] The Web RTC framework can communicate with an AS (520) supporting 5G-RTC (e.g., server (120)) via an RTC-4 interface. The 5G-RTC AS (520) can include an interactive connectivity establishment (ICE) function, a media function, a transport gateway function, a WebRTC signaling function, an application support Web function, and an interworking function.
[0079] The target with which the APU of the terminal (510) negotiates for service preparation may not be the 5GMS AF (not shown), but the WebRTC signaling function of the 5G-RTC AF (540) or 5G-RTC AS (520), and the reception and processing of the neural network model may be handled by the media function of the 5G-RTC AS (520).
[0080] One embodiment of the present disclosure may provide a method for representing a layer in which error-responsive learning is performed in a storage format of a neural network model supporting segmented processing.
[0081] NNEF, an open source standard for storing neural network models, can use a file format (e.g., *.graph) representing model network structure information and a file format (e.g., *.dat) representing layer node weight information to store neural network model data. The graph information can include structure information of the dat information. A processing device can use the graph information to generate a data structure of the model network and then map the tensor values of the dat file to the data structure. Embodiments of the present disclosure can provide a method for indicating a split point of a layer for split processing and a method for identifying an error-responsive learning layer that is used only when an error occurs.
[0082] FIG. 6A and FIG. 6B are diagrams for explaining a data structure according to one embodiment of the present disclosure.
[0083] Referring to Fig. 6a, a data structure (e.g., graph information of NNEF) is in the JSON (JavaScript object notation) format, and elements such as 'input' (610), 'output' (630), and 'operations' (620) can be used to describe the input, output, and one or more operations of the entire model, respectively. 'Operations' (620) can correspond to at least one layer described in the embodiments of the present disclosure. 'input' (610) and 'output' (630) can correspond to the input and output of the layer. Referring to Fig. 6b, the output of one operation can become the input of the next operation.
[0084] The detailed properties of a layer can be indicated by, for example, 'name', 'type', 'shape', and / or 'value'. Name represents the neural network computation of the layer, type represents the data type of the variable, shape represents the form of a multidimensional array, and value can represent the argument value of the operation.
[0085] and below show the data structure of NNEF for setting up a neural network model according to one embodiment.
[0086] input {name: "input_tensor"type: "float32"shape: [1, 28, 28, 3]}output {name: "output_tensor"type: "float32"shape: [1, 10]}
[0087] operation {name: "conv2d"inputs {name: "input_tensor"}outputs {name: " output_tensor"type: "float32"shape: [1, 28, 28, 32]}attributes {name: "filter"type: "float32"shape: [5, 5, 3, 32]value: [0.1, 0.2, ..., 0.9]}}
[0088] As described above, an operation may include at least one input, at least one output, or at least one attribute. At least one input or at least one output may include at least one of a name, a type, or a shape. At least one attribute may include at least one of a name, a type, a shape, or a value.
[0089] The method for describing a neural network model structure according to the present disclosure can display layers that can be separated by taking into account division processing.
[0090] , , and below illustrate data structures defining a neural network model that supports segmentation processing according to one embodiment.
[0091] input {name: "input_tensor"type: "float32"shape: [1, 28, 28, 3]}output {name: "output_tensor"type: "float32"shape: [1, 10]}split {name: "max_pool"attributes {error_prune_input {name: "packet_error_report"type: "float32"shape: [1, 255]}error_prune_operation: "max_pool_error_prune"}}
[0092] operation {name: "conv2d"inputs {name: "input_tensor"}outputs {name: "conv_output"type: "float32"shape: [1, 28, 28, 32]}attributes {name: "filter"type: "float32"shape: [5, 5, 3, 32]value: [0.1, 0.2, ..., 0.9]}}
[0093] operation {name: "max_pool"inputs {name: "conv_output"}outputs {name: "output_tensor"type: "float32"shape: [1, 14, 14, 32]}attributes {name: "window_shape"type: "integer"value: [1, 2, 2, 1]}}
[0094] operation {name: "max_pool_error_prune"inputs {name: "conv_output"}inputs {name: "packet_error_report"}outputs {name: "conv_output"type: "float32"shape: [1, 28, 28, 32]}attributes {name: "filter"type: "float32"shape: [5, 5, 3, 32]value: [0.1, 0.2, ..., 0.9]}}
[0095] As described above, the data structure of the configuration information may include 'input', 'output', 'operations', and 'split'. The 'split' element may include the 'name' of the 'operation' element. The data specified in the 'input' of the 'operation' specified by the 'split' element may be intermediate data. The intermediate data may be output from a previous processing device (e.g., a first processing device), and the 'operation' (e.g., 'max_pool') specified by the 'split' element may be performed in a next processing device (e.g., a second processing device).
[0096] The 'attributes' of 'split' can include error_prune_input and error_prune_operation. If an error is detected in the input of the 'max_pool' operation, error_prune_input can indicate additional input to be authorized, and error_prune_operation can indicate additional operation to be executed.
[0097] For example, the data structure may include 'split', which is the top-level element describing the entire model, and the 'name' of 'split' may point to the second operation 'max_pool'. In addition, the input_tensor, which is the input of the entire model, may be connected to the first operation conv2d, the output of the first operation conv_output may be connected as the input of the second operation max_pool, and the output_tensor, which is the output of the second operation, may be connected as the output of the entire model. The 'split' may indicate that the second operation named max_pool is the location of the split processing. If an error is detected in the output of the second operation, a processing device that supports error_prune_input and error_prune_operation may execute the operation of 'max_pool_error_prune', which receives the packet_error_report and conv_output specified as error_prune_input of 'attributes' as inputs, and connect the output conv_output to the 'max_pool' operation. The 'max_pool' operation can produce the final output of the entire model.
[0098] FIGS. 7A and 7B are diagrams illustrating a data structure including a partition element according to one embodiment.
[0099] Referring to Fig. 7a, the data structure may include elements such as 'input' (710), 'operations' (720), 'output' (730), and 'split' (750). Referring to Fig. 7b, the output of one operation may become the input of the next operation. 'split' may indicate an operation to be subject to split processing (e.g., 'L3 operation'). 'split' may include 'attributes' that indicate an operation to be used as an error learning layer.
[0100] One embodiment of the present disclosure may provide a method for serializing intermediate data.
[0101] FIG. 8a, FIG. 8b, FIG. 8c, and FIG. 8d are diagrams illustrating transmission of serialized data according to one embodiment of the present disclosure.
[0102] Referring to FIG. 8A, processing devices that support split processing (e.g., terminal processing (810) and server processing (820)) can obtain configuration information (e.g., AI model information downloaded in operation 306) for intermediate data in serialized format (e.g., serialized data (800)) received from a previous processing device through network transmission (830) or for transmission to a next processing device.
[0103] The above setting information may further include information indicating at least one of the properties common to intermediate data (e.g., serialized data (800)), such as a header format, the number of times data is transmitted, the total size before data compression, a data compression method, a data compression range, an identifier of a layer within the data, the number of nodes within the data, a data array size per node, a data format per node, a storage size of a data format, an error detection method for data, a storage size of an error detection code, a description order of data and error detection code, or a node initial value.
[0104] The header format can indicate whether the header is in json format, bit allocation format, or type-length-value format.
[0105] The number of data transfers can indicate the number of times intermediate data is transferred per second.
[0106] The total size before data compression can be expressed in bytes as the size of the intermediate data before compression.
[0107] The data compression method can indicate whether the intermediate data is compressed and the compression method used. A value of 0 means no compression, while values greater than 1 can indicate the identifier of the compression method used.
[0108] The compression range of the data indicates whether the intermediate data is compressed per node or as a whole.
[0109] The identifier of the layer in the data is an identifier that identifies which layer in the neural network model the intermediate data was generated from.
[0110] The number of nodes in the data can indicate the number of nodes included in the intermediate data. The number of nodes is already determined in the model network, but can be included in the intermediate data for faster processing.
[0111] The size of the data array per node can be expressed as an array of the sizes of each dimension in an n-dimensional array. For example, when each dimension of an n-dimensional array has sizes j, k, and l, it is expressed as an array in the order [j, k, l].
[0112] The data type per node can indicate the type of data held by the node, such as an integer, a real number, or a 32-bit floating-point representation of a real number (big-endian float32). For an n-dimensional array, the variable type can be expressed as array.
[0113] The storage size of a data type can be expressed in bytes as the size in which the data type is stored. In the case of an n-dimensional array, the storage size of the data type can be expressed as array.
[0114] The data error detection method may indicate an identifier of the method used to detect data errors. The detection method may be described for each node and / or each component described below.
[0115] The storage size of the error detection code can be expressed in bytes when the error detection code is stored. The storage size can be specified for each node and / or each component described below.
[0116] The order in which data and error detection codes are described can be specified, for example, by indicating whether a node identifier and error detection code are described for each node, an error detection code for each component, or all nodes are described and then the error detection codes for each node and component are combined. The combination of nodes and their components, and the node error detection codes and component error detection codes, is not limited to the examples.
[0117] FIGS. 8b, 8c, and 8d illustrate examples of serialized payloads within serialized data (800) according to settings according to one embodiment of the present disclosure.
[0118] Referring to FIG. 8b, the serialization payload (802) may include result values of nodes, for example, result value of node #1, result value of node #2, ..., and result value of node #m.
[0119] Referring to FIG. 8c, the serialization payload (804) may include result values of nodes and error detection codes for each result value of nodes, for example, node #1 result value, node #1 error detection code, node #2 result value, node #2 error detection code, ... and node #m result value and node #m error detection code.
[0120] Referring to FIG. 8d, the serialization payload (806) may include error detection codes for each element of the array of nodes, and error detection codes for each result value of the nodes, for example, node#1 result value, error detection code for each node#1 array element, node#1 error detection code, node#2 result value, error detection code for each node#2 array element, node#2 error detection code, ... and node#m result value, error detection code for each node#m array element, and node#m error detection code.
[0121] The initial value of a node is a value used to initialize the node's data when an error is determined to have occurred in the node. Depending on the type, characteristics, and purpose of the layer, the initial value of the node may be determined as 0, 1, 0.5, or an eigen matrix. After receiving serialized data from a device in a previous stage (e.g., a previous layer), a device in a subsequent stage can initialize the corresponding node or component using the initial value and then perform processing in the subsequent stage if an error is detected in the serialized data.
[0122] As illustrated in FIG. 8a, serialized intermediate data, for example, serialized data (800), may be composed of a serialized header and a serialized payload.
[0123] The serialization header can be in JSON format, bit allocation format, or type-length-value format. A JSON serialization header can contain the field name and field value, separated by a semicolon. A bit allocation serialization header can toggle a flag bit to indicate which field values are written, and then include the specified field values. A type-length-value serialization header can contain the length and value for fields of the specified type.
[0124] In one embodiment, if the header of intermediate data has the same attribute values as those described in the above settings, it may not be duplicated in the header, or if it is separately described, it may indicate a different setting. For example, after setting "no compression" in the configuration step, the header information may indicate that compression has been applied in the nth transmission, and the intermediate data may be compressed and transmitted, and the intermediate data may be transmitted without compression in the n+1st transmission.
[0125] In addition to the aforementioned settings, the header of the intermediate data may further include specific values that vary for each intermediate data, such as an identifier of the data to be inferred or learned, or at least one of a query and a query identifier. If the size of the intermediate data is relatively small, a single serialized data may include multiple intermediate data for multiple data (e.g., frames).
[0126] Data identifiers can be input to a neural network model to indicate the identifier of the data being inferred or learned. Examples of data include video media or images, and identifiers can include additional attribute information such as frame numbers, image identifiers, or the location where the image was captured. When receiving processing results from a server, the terminal can also receive the data identifiers, allowing it to determine which data the terminal has learned.
[0127] The query identifier is included when transmitting the entire query along with intermediate data. The server, in parallel with the processing results on the terminal, can receive the query from a second or third alternative neural network model and return the resulting inference results from the alternative neural network model, along with the model identifier and query identifier. If the terminal determines that the result is more appropriate, it can receive the model via the model identifier and use it for future learning or inference.
[0128] As described above, serialized data (e.g., serialized data (800)) may include calculated values (e.g., result values) that are calculation results of nodes belonging to a layer and error detection code for detecting whether there is an error in the calculated values.
[0129] The intermediate data resulting from the layer can be generated as an n-th order matrix for each node by m nodes. Multidimensional array data storage in languages like C and Python follows row-major order, with the innermost dimension element of the multidimensional array being stored first, followed by the next dimension element.
[0130] According to row-major order, if in a three-dimensional matrix where n is 3, each dimension has sizes j, k, and l, the storage order of the elements of the array is as follows.
[0131] arr[0,0,0], arr[0,0,1], arr[0,0,2], ..., arr[0,0,l-1], arr[0,1,0], ..., arr[1,k-1,2], arr[j-1,k-1,l-1]
[0132] In the example above, the number of nodes is m, and the number of components of intermediate data is m × j × k × l.
[0133] Embodiments of the present disclosure can detect errors and store error detection codes in different ways for each node and each element.
[0134] If the error detection code is generated for each node, m node error detection codes can be written in the above example. If the error detection code is generated for each component, m × j × k × l error detection codes can be written.
[0135] Error detection methods may include parity bits, checksums, CRC (cyclic redundancy code check), or FEC (forward error check), and the types of error detection codes of nodes and components are described in the properties of the header of the aforementioned settings or intermediate data.
[0136] Error detection of node data can be performed on the entire n-dimensional array of data for each node. For example, the serialized data for a layer consisting of m nodes may include m node error detection codes. For example, if CRC-32 is used, the 32-bit code generated after applying the CRC-32 code to all data in each node becomes the CRC-32 code.
[0137] Error detection of components of node data can be performed on all components of the n-dimensional array data of each node. For example, for an array consisting of three dimensions and having sizes j, k, and l, a parity bit can be calculated for each component. For example, a floating-point value expressed as 32 bits is expressed as a combination of 1s and 0s, and if the number of 1s is even, it is represented as 1, and if it is odd, it is represented as 0. The receiving device can determine whether an error has occurred for each component by comparing the array component value with the parity bit.
[0138] In the above example, for a node having a 3D array where j, k, and l are 32, 32, and 32, respectively, the 32x32x32 float32 type floating point data requires 4 bytes for each component, resulting in a size of 131,072 KB. If a parity bit is used as an error detection code for each component, the size of the error detection code becomes 4 KB of parity bits for 131,072 KB of data. If CRC-32 is used as an error detection code for each node, the size of the error detection code becomes 4 bytes.
[0139] Node data may be compressed for transmission efficiency, either for all node data for higher transmission efficiency or for individual node data to prevent transmission errors from affecting the entire serialized data.
[0140] When compression is performed on the entire node data, serialized data including at least one of the above-mentioned node data, error detection code for the node data, or error detection code for the component may be compressed.
[0141] When compression is performed on individual node data, the error detection code for the node data and node components described above may be compressed, and an identifier and node compression error detection code may be added.
[0142] One embodiment of the present disclosure can provide a method for detecting errors in node data and initializing node data.
[0143] A processing device supporting split processing can receive serialized intermediate data (e.g., serialized data) from a previous device, and then, based on the configuration and serialization header, check the type and range (node, component) of the error detection code used in the received data and whether it is compressed, and, if necessary, decrypt the received data. The processing device can generate an error detection code for the received data and compare whether the generated value is the same as the value of the error detection code included in the received data.
[0144] If the generated value and the received value do not match, the processing device may determine that an error (e.g., a transmission error) has occurred during the transmission of data to the corresponding node or component.
[0145] If the processing device determines that an error has occurred in the transmission process of the received data (e.g., serialized data), before transmitting the values of the received data to the remaining layers of the neural network model, the processing device may initialize the data values of the node where the error occurred without using the values of the node where the error is presumed to have occurred.
[0146] Initializing data values for nodes in a neural network model can include setting the initial values at which nodes begin learning for each layer's feature. For example, if a layer uses 0 to represent an unlearned, neutral state, the data value of a node can be initialized to 0. If the neutral state is 0.5 or 1, the data value of a node can be initialized to 0.5 or 1. For example, if eigenvectors are specified, the matrix of array data can be initialized to the eigenvectors.
[0147] Data values for initialization for each layer can be described in the model graph or serialization header, or determined through communication between the terminal and the server.
[0148] One embodiment of the present disclosure may provide a method for overcoming transmission errors by error learning.
[0149] Once the data for a node determined to have a transmission error is initialized, the possibility of that node being an outlier is eliminated, and the node operates as if it were in its initial learning state. From the perspective of learning or inference of the overall model, this can be interpreted as one or more nodes responding to a given input, but a specific node (e.g., the node that experienced an error) does not respond.
[0150] To reinterpret this phenomenon, if there is a combination of a series of nodes that show large and small reactions to a certain input pattern, and learning is performed on this combination, if the combination of a series of nodes shows a reaction, it can be expected that the output value of the node can be inferred even if the output value of some nodes is not given.
[0151] Therefore, the method for overcoming transmission errors proposed in the present disclosure can detect errors in received serialized data during segmentation processing of a neural network model network, change the data in which the error is detected to an initial value, apply the output of the last layer of the segmented previous stage to an error learning layer, and apply the output of the error learning layer as an input to a subsequent layer of the segmented neural network model.
[0152] FIG. 9 illustrates multiple layers of a neural network model according to one embodiment of the present disclosure.
[0153] Referring to FIG. 9, the neural network model may be composed of five layers, for example, a first layer (L1) (901), a second layer (L2) (902), a third layer (L3) (903), a fourth layer (L4) (904), and a fifth layer (L5) (905).
[0154] FIG. 10 illustrates split processing in multiple processing devices according to one embodiment of the present disclosure.
[0155] Referring to FIG. 10, among the plurality of layers, the first and second layers (L1, L2) (1001, 1002) and the third to fifth layers (L3, L4, L5) (1003, 1004, 1005) may be divided and processed by a terminal (1010) (e.g., terminal (110)) and a server (1020) (e.g., server (120)). The output (e.g., intermediate data) of the last layer of the divided previous stage (e.g., the second layer (L2)) may be transmitted from the terminal (1010) to the server (1020) via a mobile communication network (1030) (e.g., network (100)).
[0156] FIG. 11 is a diagram for explaining a transmission error response in split processing according to one embodiment of the present disclosure.
[0157] Referring to FIG. 11, intermediate data (e.g., L2 output) which is the output of the first layer (L1) (1110) and the second layer (L2) (1102) of the terminal (1110) may include transmission errors while being transmitted through the mobile communication network (1130). The intermediate data including transmission errors may be connected to the first layer of a subsequent step (e.g., the tpqjsWo layer (L3) (1103) of the server (1120)) after passing through the error learning layer (e.g., L2-prime) (1102a)) of the server (1120).
[0158] The last layer of the previous stage, L2 (1102), L2-prime (1102a) that learns the output of L2, and L3 (1103) that receives the output of L2-prime as L2 input are illustrated. When the neural network model is divided between the second layer, L2 (1102), and the third layer, L3 (1103), the output of L2 (1102) can be serialized and transmitted from the terminal (1110) to the server (1120). The server (1120) can process the serialized data by L2-prime (1102a) that learns the output pattern of L2 (1102) before receiving the serialized data and applying it to L3 (1103) as L3 input, and can apply the output of L2-prime (1102a) to L3 (1103). The output of L3 (1103) can be processed through subsequent layers (e.g., L4 (1104) and L5 (1105)) to produce the final output.
[0159] The L2-prime layer (1102a) can be configured to learn to take as input the normal output (normal value) of L2 (1102), the output (error value) mixed with randomly initialized node data, and the status information of the mobile communication network (1130) (e.g., a report related to the network status), and to generate the expected value of the L2 layer (1102) for a node identified as having an error.
[0160] FIG. 12 is a diagram illustrating an error learning layer learned with normal values, error values, and network status according to one embodiment.
[0161] Referring to FIG. 12, status information indicating the network status of a mobile communication network (1130) (e.g., 3GPP NWDAF report and / or EVEX report) may include at least one of the following information.
[0162] - User data traffic and patterns;
[0163] - Location, mobility and usage of your device;
[0164] - Network infrastructure and service status;
[0165] - Network and service bottlenecks;
[0166] - QoS (Quality of Service) information;
[0167] - RAN (Radio Access Network) and Core Network status;
[0168] - Real-time traffic analysis data;
[0169] - Network security status;
[0170] - Disability diagnosis and prevention data; and / or
[0171] - Data analysis information for machine learning algorithms.
[0172] In the split processing of a neural network model, at least one processing device (e.g., server (120)) may additionally include an error learning layer (e.g., L2-prime (1220)) that assumes a transmission error and learns the transmission error together with the transmission status in the serialization and transmission section for intermediate data between a split point reviewed as an appropriate branch point, i.e., a layer (e.g., L2 (1210)) and a layer (e.g., L3 (1230)). The error learning layer (e.g., L2-prime (1220)) may be configured to learn about the output of a previous layer, or additionally to learn to maintain the accuracy of the overall inference result.
[0173] When a neural network model is configured to include segmented processing, if a transmission error occurs in a result value received from a previous processing device (e.g., terminal (110)), a subsequent processing device (e.g., server (120)) can generate the normal value through a layer (e.g., error learning layer) that infers the normal value before an error occurs by the network (e.g., network (100)) by using the network status as an additional input.
[0174] Neural network models may further include layers trained for special purposes (e.g., error mitigation) (e.g., error-learning layers, L2-prime) between intermediate layers for purposes such as transfer learning. These error-learning layers may be added between layers (e.g., L2) and layers (e.g., L3) based on the point at which processing is to be performed, and then connected to the layers (e.g., L2 and L3).
[0175] In one embodiment, in a neural network model that receives an image as input and provides inference, the basic model may be composed of five layers (e.g., L1, L2, L3, L4, and L5), and for segmentation processing, the point between the second layer, L2, and the third layer, L3, of the basic model may be determined as a segmentation point.
[0176] below shows an example of code that loads the trained model, trained_model.h5, and inputs it into the base_model variable in neural network technology such as Keras.
[0177] from keras.models import Modelfrom keras.layers import Input, Dense, Dropout, LSTMbase_model = load_model('trained_model.h5')
[0178] As shown in Table 8 below, the base_model variables can be stored as x1 and x2 respectively to allow some layers to be processed in the first processing unit.
[0179] x1 = base_model.layers[:2].outputx2 = base_model.layers[2:].output
[0180] An example of a neural network model including an error learning layer for transmission errors can be configured as shown in Table 9 below. This example illustrates an error learning layer that receives a packet error rate as input and predicts a normal value through long-term and / or short-term learning. A neural network model including an error learning layer can be further enhanced by learning the accuracy, complexity, and diversity of state information received from the transmission network, and through learning, can use more and more complex parameters to improve the accuracy of error prediction.
[0181] packetError_input = Input(shape=(1,))y = Dense(32, activation='relu')(packetError_input)y = Dropout(0.2)(y)y = Dense(64, activation='relu')(y)y = Dropout(0.2)(y)y = LSTM(32)(y)y = Dense(1, activation='sigmoid')(y)
[0182] The first part of base_model, x1, the error learning model y, and the remainder of base_model, x2, can be combined as shown in . A new integrated model, model, can be created, and the weights of the base model can be inherited and used as is.
[0183] z = concatenate([x1, y, x2])model = Model(inputs=[base_model.input, packetError_input], outputs=z)for i in range(0,2): # copy x1 weightsmodel.layers[i].set_weights(base_model.layers[i].get_weights())for i in range(2,5): # x2 weights Copyj=i+len(y.layer)model.layers[j].set_weights(base_model.layers[i].get_weights())
[0184] The integrated model, after its configuration is completed, is compiled for use and then receives the image and network transmission status as input to perform inference as shown in below.
[0185] model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])output = model.predict([image_input, packetError_test])
[0186] Embodiments of the present disclosure can divide an already learned network through a method such as the above example, and continue to infer the remaining network by receiving a packet error rate that may occur during the transmission process as an additional factor.
[0187] FIG. 13 is a block diagram showing a configuration of a terminal in a communication system according to one embodiment of the present disclosure.
[0188] Referring to FIG. 13, a terminal (e.g., terminal (110)) may include a processor (1310), a transceiver (1320), and a memory (1330). Depending on the communication method of the terminal (110) described above, the processor (1310), the transceiver (1320), and the memory (1330) may operate. However, the components of the terminal (110) are not limited to the examples described above. For example, the terminal (110) may include more or fewer components than the components described above. In addition, at least one of the processor (1310), the transceiver (1320), and the memory (1330) may be implemented in the form of a single chip. The processor (1310) may include one or more processing circuitry.
[0189] The transceiver (1320) includes a receiving circuit and a transmitting circuit of the terminal (110), and can transmit and receive signals with other devices (e.g., a base station or a server (120)). To this end, the transceiver (1320) may include an RF (radio frequency) transmitter that up-converts and amplifies the frequency of a transmitted signal, and an RF receiver that low-noise amplifies and frequency-converts a received signal. However, this is only one example of the configuration of the transceiver (1320), and the components of the transceiver (1320) are not limited to the RF transmitter and RF receiver.
[0190] The transceiver (1320) can receive a signal through a wireless channel and output it to the processor (1310), and transmit the signal output from the processor (1310) through the wireless channel.
[0191] The memory (1330) can store programs and data required for the operation of the terminal (110). In addition, the memory (1330) can store control information or data included in a signal acquired from the terminal (110). The memory (1330) can be configured as a storage medium or a combination of storage media, such as a read-only memory (ROM), a random access memory (RAM), a hard disk, a compact disk ROM (CD-ROM), and a digital video disc (DVD). At least a portion of the memory (1330) can also be configured to be included in the processor (1310).
[0192] The processor (1310) may control a series of operations so that the terminal (110) may operate according to the embodiments of the present disclosure described above. In one embodiment, the processor (1310) may be configured to obtain configuration information on a transmission format of intermediate data by segmented processing of a neural network, generate result values through processing of at least one first layer (e.g., L2) among a plurality of layers constituting the neural network, attach error detection codes for the result values to the result values based on the configuration information to generate the intermediate data (e.g., serialized data (800)), and transmit the generated intermediate data to a server (120) including at least one subsequent second layer (e.g., L3) among the plurality of layers through the transceiver (1320).
[0193] FIG. 14 is a block diagram showing a configuration of a server in a communication system according to one embodiment of the present disclosure.
[0194] Referring to FIG. 13, a server (e.g., server (120)) may include a processor (1410), a network interface (I / F) (1420), and a memory (1430). Depending on the communication method of the server (120) described above, the processor (1410), the network interface (1420), and the memory (1430) may operate. However, the components of the server (120) are not limited to the examples described above. For example, the server (120) may include more or fewer components than the components described above. In addition, at least one of the processor (1410), the network interface (1420), and the memory (1430) may be implemented in the form of a single chip. The processor (1410) may include one or more processing circuitry.
[0195] The network interface (1420) includes a receiver and a transmitter of the server (120) and can transmit and receive signals with other devices (e.g., terminals (110) or network entities).
[0196] The memory (1430) can store programs and data required for the operation of the server (120). In addition, the memory (1430) can store control information or data included in signals acquired from the server (120). The memory (1430) can be configured as a storage medium or a combination of storage media, such as a ROM, a RAM, a hard disk, a CD-ROM, and a DVD. At least a portion of the memory (1430) can also be configured to be included in the processor (1410).
[0197] The processor (1410) may control a series of operations so that the server (120) may operate according to the embodiments of the present disclosure described above. In one embodiment, the processor (1410) may be configured to obtain setting information on a transmission format of intermediate data by segmented processing of a neural network, receive intermediate data (e.g., serialized data (800)) including result values through processing of at least one first layer (e.g., L2) among a plurality of layers constituting the neural network and error detection codes for the result values, from the terminal (110), determine whether the intermediate data contains an error based on the error detection codes, and, if an error is detected in the intermediate data, process the result values of the intermediate data by a second layer (e.g., L2-prime) trained to correspond to the first layer, and process the output of the second layer by at least one third layer (e.g., L3) following the first layer among the plurality of layers to derive a final result.
[0198] In the present disclosure described above, components are expressed singularly or plurally, depending on the specific embodiments presented. However, the singular or plural expressions are selected to suit the presented situation for convenience of explanation, and the present disclosure is not limited to singular or plural components. Components expressed in plural may be composed of singular elements, or components expressed in singular may be composed of plural elements.
[0199] While specific embodiments have been described in the detailed description of this disclosure, it should be understood that various modifications are possible without departing from the scope of this disclosure. Therefore, the scope of this disclosure should not be limited to the described embodiments, but should be defined not only by the scope of the claims described below, but also by equivalents thereof.
Claims
1. In the method of operating a terminal in a communication system, An operation for obtaining configuration information on the transmission format of intermediate data by segmentation processing of a neural network; An operation of generating result values through processing of at least one first layer among a plurality of layers constituting the above neural network; An operation of generating the intermediate data by attaching error detection codes for the result values to the result values based on the above setting information; and A method characterized by comprising an action of transmitting the generated intermediate data to a server including at least one subsequent second layer among the plurality of layers.
2. In paragraph 1, the setting information is: A method characterized by including information indicating properties of the intermediate data.
3. In the second paragraph, the properties are: A method characterized by including at least one of a header format, a number of times data is transmitted, a total size before data compression, a data compression method, a data compression range, an identifier of a layer within the data, a number of nodes within the data, a data array size per node, a data format per node, a storage size of a data format, an error detection method for data, a storage size of an error detection code, a description order of data and error detection code, or a node initial value.
4. In the method of operating a server in a communication system, An operation for obtaining configuration information on the transmission format of intermediate data by segmentation processing of a neural network; An operation of receiving, from a terminal, intermediate data including result values through processing of at least one first layer among a plurality of layers constituting the neural network and error detection codes for the result values; An operation of checking whether there is an error in the intermediate data based on the above error detection codes; If an error is detected in the intermediate data, an operation of processing the result values of the intermediate data by a second layer learned to correspond to the first layer; and A method characterized by including an operation of processing the output of the second layer by at least one third layer subsequent to the first layer among the plurality of layers to derive a final result.
5. In paragraph 4, the setting information is, A method characterized by including information indicating properties of the intermediate data.
6. In paragraph 5, the properties are: A method characterized by including at least one of a header format, a number of times data is transmitted, a total size before data compression, a data compression method, a data compression range, an identifier of a layer within the data, a number of nodes within the data, a data array size per node, a data format per node, a storage size of a data format, an error detection method for data, a storage size of an error detection code, a description order of data and error detection code, or a node initial value.
7. In the fourth paragraph, the third layer, A method characterized in that it is configured to predict normal values by the first layer based on the result values included in the intermediate data.
8. At the terminal of the communication system, Transmitter and receiver; and Including a processor connected to the above transceiver, The above processor, Obtain setting information on the transmission format of intermediate data by segmentation processing of a neural network, Generating result values through processing of at least one first layer among the plurality of layers constituting the above neural network, Based on the above setting information, error detection codes for the above result values are attached to the above result values to generate the intermediate data, and A terminal characterized in that it is configured to transmit the generated intermediate data to a server including at least one subsequent second layer among the plurality of layers through the transceiver.
9. In paragraph 8, the setting information is, Contains information indicating the properties of the above intermediate data, 10. In paragraph 9, the properties are: A method characterized by including at least one of a header format, a number of times data is transmitted, a total size before data compression, a data compression method, a data compression range, an identifier of a layer within the data, a number of nodes within the data, a data array size per node, a data format per node, a storage size of a data format, an error detection method for data, a storage size of an error detection code, a description order of data and error detection code, or a node initial value.
11. In the server of the communication system, network interface; and comprising a processor connected to the above network interface; The above processor, Obtain setting information on the transmission format of intermediate data by segmentation processing of a neural network, Receiving intermediate data including result values through processing of at least one first layer among the plurality of layers constituting the neural network and error detection codes for the result values from the terminal, Based on the above error detection codes, check whether there is an error in the intermediate data, If an error is detected in the above intermediate data, the result values of the above intermediate data are processed by the second layer learned to correspond to the first layer, A server characterized in that the output of the second layer is processed by at least one third layer subsequent to the first layer among the plurality of layers to produce a final result.
12. In paragraph 11, the setting information is: A server characterized by including information indicating properties of the intermediate data.
13. In paragraph 12, the properties are: A server characterized by including at least one of a header format, a number of times data is transmitted, a total size before data compression, a data compression method, a data compression range, an identifier of a layer in the data, a number of nodes in the data, a data array size per node, a data format per node, a storage size of a data format, an error detection method in the data, a storage size of an error detection code, a description order of data and error detection code, or a node initial value.
14. In the 11th paragraph, the third layer, A server characterized in that it is configured to predict normal values by the first layer based on the result values included in the intermediate data.
Citation Information
Patent Citations
Discharge opening and closing control valve of high-pressure hydrogen tank
KR102722753B1
Data stream fault detection mechanism in an artificial neural network processor
US11221929B1
Neural network intermediate results safety mechanism in an artificial neural network processor
US11263077B1
Software Defined Redundant Allocation Safety Mechanism In An Artificial Neural Network Processor
US20220100601A1
Weights Safety Mechanism In An Artificial Neural Network Processor
US20220103186A1
