Client audio data transmission and processing system based on multi-protocol compatibility

Abstract

The client audio data transmission and processing system based on multi-protocol compatibility according to the present application is characterized by comprising a data processing device including: a communication unit that transmits and receives audio data to and from at least one of a first service provider based on IP and a second service provider based on a protocol of HTTP; a storage unit that stores audio segment files received by the communication unit in a queue (first-in first-out, FIFO) structure; and a control unit that outputs audio data by decoding and transcoding the audio segment files stored in the storage unit.

Description

Technical Field

[0001] This invention relates to a client-side audio data transmission and processing system based on multi-protocol compatibility. Background Technology

[0002] Recently, client systems supporting various audio data transmission and processing methods are under development. In particular, the use of IP-based (Internet Protocol) services and HTTP-based (Hypertext Transfer Protocol) protocols to stream audio data in real time or transmit stored data is widely used. Although such systems are relatively effective in providing users with audio content such as music, podcasts, and audiobooks, they suffer from limitations such as unstable network environments, data loss, and degraded audio quality, resulting in a poor user experience.

[0003] Typically, existing technologies are designed with a fixed network environment in mind, making them ill-suited for handling variations in network bandwidth or data loss, and lacking proactive solutions to prevent audio quality degradation during real-time streaming. Furthermore, the lack of consideration for compatibility between multiple protocols results in limitations in collaboration with specific service providers. In particular, data loss or inconsistency transmitted over HTTP can significantly degrade the user experience during recovery and reprocessing.

[0004] Furthermore, there are issues with the inability to adjust sampling frequency, bit rate, and synchronization status in real time during audio data processing, or the burden on system performance caused by inefficient transcoding and decoding algorithms used to maintain data continuity. Therefore, the need for systems that can flexibly adapt to network conditions and audio data characteristics is becoming increasingly apparent. Summary of the Invention

[0005] This invention addresses the aforementioned problems by providing a client-side audio data transmission and processing system based on multi-protocol compatibility. Specifically, it provides a technical foundation for stable audio data transmission and processing in various environments by ensuring compatibility between IP-based and HTTP-based service providers. This is achieved through Cyclic Redundancy Check (CRC) to identify whether audio data transmitted via HTTP is lost, and by re-requesting and reprocessing lost data to ensure data integrity and minimize audio quality degradation. The system can respond in real-time to potential data loss during transmission. By analyzing the interactions between various variables such as network bandwidth, average sample energy, and user terminal battery status, the system optimizes audio data processing, maximizes energy efficiency, and maintains data processing stability. Utilizing audio data metadata, the system merges and sorts data in MP4 container format, maintaining timeline continuity and resolving synchronization issues, thereby providing users with a high-quality audio experience. The system can also resolve errors or inconsistencies that arise during data processing in real-time, maximizing user experience while maintaining data integrity and improving the performance and stability of the client system.

[0006] The present invention is characterized in that, according to an embodiment of the present invention, a client audio data transmission and processing system based on multi-protocol compatibility includes a data processing device, the data processing device comprising: a communication unit for sending and receiving audio data with at least one of an IP-based first service provider and a second service provider based on an HTTP protocol; a storage unit for storing audio segment files received through the communication unit in a queue (first-in-first-out, FIFO) structure; and a control unit for outputting audio data by decoding and transcoding the audio segment files stored in the storage unit.

[0007] The invention is characterized by including a user terminal connected to the data processing device via a wired or wireless network to output sound; a communication unit requesting audio segment files from a second service provider at a first time interval based on HTTP to receive multiple audio segment files corresponding to the first time interval; a control unit retrieving the multiple audio segment files from the storage unit in a FIFO manner; after identifying the metadata corresponding to the multiple audio segment files to confirm continuity, merging them according to the MP4 container format; and sorting and synchronizing the segments within the MP4 file according to time information.

[0008] The invention is characterized in that the control unit generates a transcoded file that transcodes the MP4 file according to the user terminal's sound output requirements. It extracts the audio stream from the transcoded file to generate decoded data that is restored to PCM (Pulse Code Modulation) data. The control unit adjusts the sampling frequency and bit rate of the decoded data in real time according to the user terminal's output requirements and then sends the decoded data. During the transmission of the decoded data, the control unit extracts new audio segment files from the storage unit to maintain the continuity of the MP4 file and updates it in real time. The control unit adjusts the transcoding settings according to the user terminal's network bandwidth. If the network bandwidth decreases from 10Mbps to 5Mbps, the sampling frequency is adjusted from 48kHz to 32kHz according to the inverse relationship of the bandwidth reduction, and the bit rate is adjusted from 320kbps to 192kbps according to the logarithmic ratio of the bandwidth reduction.

[0009] The invention is characterized in that the control unit uses the synchronization delay time generated when sending decoded data as a benchmark to set the sending priority order. If the synchronization delay time increases from 50ms to 100ms, the segment sending interval is adjusted from 2ms to 4ms according to the exponential ratio of the increase in the synchronization delay time. The segments are reordered based on metadata to maintain synchronization. The control unit identifies inconsistencies in the time information and sampling information contained in the audio segments by analyzing the metadata of the audio segments in the MP4 file and generates inconsistency information. The inconsistency information is sorted and reconstructed based on the queue (first-in-first-out, FIFO) structure stored in the storage unit to generate a new MP4 file that maintains the continuity of the timeline. The new MP4 file is transcoded and sent to the user terminal. If network delay or data loss that may occur during the sending process is identified, the control unit re-requests and reprocesses the lost data according to the metadata.

[0010] The invention is characterized in that the control unit adjusts the audio output volume based on the PCM data proportionally according to the user terminal's usage environment, and adopts an automatic gain control (AGC) algorithm to maintain the audio output volume between 70dB and 90dB. The control unit identifies whether the audio segment file sent via the HTTP-based protocol is lost through Cyclic Redundancy Check (CRC). If the identification result indicates that the audio segment file sent via the HTTP-based protocol is lost, the control unit requests the lost audio segment file from the second service provider again until the audio segment file obtained through the request is received. This process is repeated to maintain the continuity of the MP4 file by repeatedly outputting existing segments or inserting temporary replacement data. After the loss is repaired, the MP4 file is updated and sent to the user terminal in real time.

[0011] The invention is characterized in that the control unit monitors the integrity of the transmitted decoded data. If the data loss rate increases from 1% to 5%, the data repair request frequency is increased from 5 times to 10 times in accordance with the inverse relationship of the increase in the loss rate. The repaired data is integrated in real time and sent to the user terminal. The control unit identifies the audio quality of the audio output at a second time interval based on the PCM data of the decoded MP4 file. If the identification result is audio distortion or noise is detected, the audio data corresponding to the corresponding time interval is reprocessed by comparing it with the existing segments in the storage unit. If the cause of the audio distortion is network latency, the network status is monitored, and audio data with transcoding settings that include a new sampling frequency and bit rate corresponding to the network conditions is sent.

[0012] This invention provides a client-side audio data transmission and processing system based on multi-protocol compatibility. Specifically, it provides the technical foundation for stable transmission and processing of audio data in various environments by ensuring compatibility between IP-based and HTTP-based service providers. To this end, it uses Cyclic Redundancy Check (CRC) to identify whether audio data transmitted via HTTP is lost, and performs re-request and reprocessing on lost data to ensure data integrity and minimize audio quality degradation. It can respond in real-time to potential data loss during transmission. By analyzing the interaction between various variables such as network bandwidth, average sample energy, and user terminal battery status, it optimizes audio data processing, maximizes energy efficiency, and maintains data processing stability. Utilizing the metadata of the audio data, it merges and sorts data according to the MP4 container format, maintaining timeline continuity and resolving synchronization issues, thereby providing users with a high-quality audio experience. It can resolve errors or inconsistencies generated during data processing in real-time, maximizing user experience while maintaining data integrity and improving the performance and stability of the client system. Attached Figure Description

[0013] Figure 1 This diagram illustrates a simplified structure of a client audio data transmission and processing system based on multi-protocol compatibility according to an embodiment of the present invention.

[0014] Figure 2 The diagram illustrates the specific structure of a client audio data transmission and processing system based on multi-protocol compatibility according to an embodiment of the present invention.

[0015] Figure 3 This is a flowchart illustrating a client audio data transmission and processing system based on multi-protocol compatibility according to an embodiment of the present invention.

[0016] Explanation of reference numerals in the attached figures

[0017] 100: User terminal

[0018] 110: User Input Department

[0019] 120: Display Department

[0020] 200: Data processing device

[0021] 210: Ministry of Communications

[0022] 220: Storage Department

[0023] 230: Control Department Detailed Implementation

[0024] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals or symbols refer to structural elements that actually perform the same function. For clarity and convenience of explanation, the size of each structural element in the drawings may be exaggerated. However, the technical concept, core structure, and function of the present invention are not limited to the structures or functions described below. In the process of describing the present invention, if it is determined that a detailed description of a known technology or structure related to the present invention may unnecessarily obscure the main idea of ​​the present invention, its detailed description will be omitted.

[0025] In embodiments of the present invention, terms such as "first," "second," etc., indicating sequence, are used only to distinguish one structural element from other structural elements. Unless otherwise explicitly stated in the context, singular expressions include plural expressions. Furthermore, in embodiments of the present invention, terms such as "constituting," "including," and "possessing" should be understood as not precluding the existence or additional possibility of one or more other features, numbers, steps, actions, structural elements, components, or combinations thereof. Furthermore, in embodiments of the present invention, a "module" or "part" performs at least one function or action, which may be implemented by hardware or software, or by a combination of hardware and software, and may be integrated into at least one module implemented by at least one processor. Furthermore, in embodiments of the present invention, "at least one of a plurality of elements" refers not only to all elements among the plurality of elements, but also to each of the remaining parts of the plurality of elements or all combinations thereof. Furthermore, "configured to" can be replaced as follows, depending on the context: "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." "Configured to" does not simply mean "specifically designed to" in terms of hardware. Additionally, in some cases, the expression "device configured to" can mean that the device, along with other devices or components, "performs" the corresponding actions. For example, the statement "processor configured to perform A, B, and C" can mean a dedicated processor (e.g., an embedded processor) used to perform the corresponding actions, or a generic-purpose processor (e.g., a CPU or application processor) that can perform the corresponding actions through one or more software programs stored in a storage device.

[0026] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. This detailed description is intended to enable those skilled in the art to easily implement the present invention, and the technical concept and scope of the present invention are not limited thereto.

[0027] Figure 1 This diagram illustrates a simplified structure of a client-side audio data transmission and processing system based on multi-protocol compatibility according to an embodiment of the present invention. Figure 2 The diagram illustrates the specific structure of a client-side audio data transmission and processing system based on multi-protocol compatibility according to an embodiment of the present invention. Figure 3 This is a flowchart illustrating a client audio data transmission and processing system based on multi-protocol compatibility according to an embodiment of the present invention.

[0028] Reference Figures 1 to 3 An embodiment of the present invention provides a client audio data transmission and processing system based on multi-protocol compatibility, which includes a data processing device 200 and a user terminal 100.

[0029] Examples of user terminals 100 that connect to the data processing apparatus 200 of an embodiment of the present invention via wired or wireless networks to output sound include personal computers, server computers, handheld devices or laptops, mobile devices (mobile phones, PDAs, media players, etc.), multiprocessor systems, consumer electronic devices, minicomputers, mainframe computers, distributed computing including any of the systems or devices described above, edge computing environments that process data at the edge where data is generated, which is not a central server, etc. The structure is not limited to the structure described.

[0030] User terminal 100 may include at least one processor and memory. Examples of processors may include central processing unit (CPU), graphics processing unit (GPU), microprocessor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), etc., and may have multiple cores.

[0031] The memory may be temporary memory (e.g., RAM (Random Access Memory)), non-temporary memory (e.g., ROM (Read-Only Memory), Flash memory), or a combination thereof. Furthermore, the user terminal 100 may also include additional storage devices. While storage devices include magnetic storage devices, optical storage devices, etc., they are not limited to these. The storage device may store computer-readable instructions for implementing one or more embodiments improved in this specification, and may also store other computer-readable instructions for implementing operating systems, application programs, etc. The computer-readable instructions stored in the storage device may be loaded into memory for execution by a processor.

[0032] Furthermore, the user terminal 100 may include a user input unit 110 and an output device. Examples of the user input unit 110 may include a keyboard, mouse, pen, voice input device, touch input device, infrared camera, video input device, or any other input device. Examples of the output device may include one or more displays, speakers, printers, or any other output device. The computing device may also use input devices or output devices provided on other computing devices as the user input unit 110 or output device. The computing device may also include a communication module that enables communication between the computing device and other devices. This communication module may include a modem, network interface card (NIC), integrated network interface, radio frequency transmitter / receiver, infrared port, USB (Universal Serial Bus) connection, or other interfaces for connecting the computing device to other computing devices. The communication module may include wired or wireless connections.

[0033] The various structural elements of the user terminal 100 can be connected via various interconnections such as buses (e.g., peripheral interconnection (PCI), USB, firmware (IEEE 1394), optical bus structure, etc.), or via networks. In this specification, terms such as "structural element" and "system" generally refer to hardware, a combination of hardware and software, software, or computer-related entities that are executing software.

[0034] A user terminal 100 according to an embodiment of the present invention may include a display unit 120. The display unit 120 is not limited in its display method, and may be implemented using various display methods such as liquid crystal, plasma, light-emitting diode, organic light-emitting diode (OLED), surface-conduction electron-emitter, carbon nanotube, and nanocrystal. If the display unit 120 uses liquid crystal, it includes: a liquid crystal display panel; a backlight unit for supplying light to the liquid crystal display panel; and a panel driving unit for driving the liquid crystal display panel. On the other hand, the display unit 120 may be an OLED panel that is a self-emissive device without a backlight unit.

[0035] A data processing apparatus 200 according to an embodiment of the present invention may include a communication unit 210, a storage unit 220 and a control unit 230.

[0036] The data processing device 200 of one embodiment of the present invention is a device for receiving, sending and processing audio data. It is configured to cope with multiple protocols and network environments. The data processing device can provide compatibility between IP-based service providers and service providers that support HTTP-based protocols, and can automate all processes such as receiving, storing, processing and sending audio data.

[0037] The data processing apparatus 200 comprises a communication unit 210, a storage unit 220, and a control unit 230. The communication unit 210 supports IP-based networks and HTTP-based protocols to stably send and receive audio data with service providers. For example, the communication unit 210 can receive data in real time from a streaming service or download audio data in file format from a cloud-based database. The storage unit 220 performs the function of temporarily storing audio segment files received using a First-In-First-Out (FIFO) structure and acts as a queue for sequentially processing data. The storage unit 220 may include: a high-speed memory for effectively managing large amounts of data; and a file system, configured for stability.

[0038] The control unit 230 can merge data received from the data processing device in MP4 container format, transforming it into Pulse Code Modulation (PCM) data through transcoding and decoding processes. For example, the control unit 230 can sort audio segment files received in an HTTP-based streaming environment according to time information, and transform them into an audio format that meets the output requirements of the user terminal 100 through transcoding. Furthermore, the control unit 230 can monitor errors or network latency that may occur during data processing in real time, and can execute re-request and repair algorithms to correct them.

[0039] The data processing apparatus 200 of one embodiment of the present invention can maintain the quality of audio data and improve transmission efficiency. For example, by dynamically adjusting the sampling frequency and bit rate during transcoding, it minimizes data loss based on network conditions and maintains the temporal continuity of audio data based on the decoded PCM data. Furthermore, by analyzing the user's remaining battery power or network status to select the optimal processing mode, energy efficiency can be improved.

[0040] The data processing device of this invention can be used in various application fields such as real-time streaming, cloud-based data processing, and user-customized audio services, and can achieve stable audio data transmission and processing under multiple protocols and network environments. Ultimately, the data processing device of this invention provides excellent technical advantages in improving user experience and effectively utilizing network and system resources.

[0041] The IP-based first service provider and the HTTP-based second service provider in one embodiment of the present invention represent systems or service providers capable of sending and receiving audio data in various network environments. The IP-based first service provider may primarily include providers of VoIP (Voice over IP) services, SIP (Session Initiation Protocol)-based communication services, or platforms providing real-time streaming. This type of first service provider can transmit data in packet format over IP networks, making it suitable for environments requiring low latency and real-time data transmission. For example, VoIP services effectively compress voice data for real-time transmission, while SIP-based services are used in applications such as multi-person calls and video conferencing.

[0042] On the other hand, a second service provider based on the HTTP protocol is a system that primarily delivers audio data through web servers. This can include various applications such as streaming platforms, cloud storage services, or podcast hosting services. Such second service providers use both HTTP and HTTPS protocols to transmit data and are typically used in application areas where stability and compatibility are paramount. For example, streaming platforms use technologies such as DASH (Dynamic Adaptive Streaming over HTTP) or HLS (HTTP Live Streaming) to provide audio data to user terminals at adaptive bitrates, while cloud storage services support the uploading and downloading of large audio files.

[0043] Although the first and second service providers operate on different protocols, the system of this invention allows users to smoothly send and receive audio data without being limited by specific network environments or service providers by providing mutual compatibility between them. Furthermore, the IP-based first service provider is primarily used in applications that prioritize real-time performance and low latency, while the HTTP-based second service provider is suitable for environments requiring stability and versatility; this is the key difference. Ultimately, the system of this invention maximizes user flexibility and service coverage by supporting both types of service providers.

[0044] An audio segment file, as described in one embodiment of the present invention, refers to a file format used for dividing, storing, and transmitting audio data according to specified time intervals or data sizes. This segment file divides and manages raw audio data into small units, thereby providing real-time streaming, optimized data transmission, and ease of file repair. Each audio segment file may contain metadata for the corresponding segment, including the segment's start time, end time, bit rate, sampling frequency, and encoding / decoding information.

[0045] Audio segment files can be in various formats, using container formats such as MP4, TS (Transport Stream), or AAC (Advanced Audio Coding). For example, in HLS (HTTP Live Streaming) technology, segments are stored in TS format and linked to an m3u8 playlist file that defines the playback order of the segments. In DASH (Dynamic Adaptive Streaming over HTTP), MP4-based segment files are used, and each segment can be designed to support adaptive bitrate streaming and be decoded independently.

[0046] Furthermore, audio segment files are not only used in real-time streaming environments, but also in applications that require storage and offline playback. For example, podcasts or audiobooks can be segmented and transmitted to client terminals, and then the corresponding segments can be played back sequentially to continuously provide the user with an experience.

[0047] The audio segment files of this invention are designed to leverage these common characteristics to improve transmission efficiency in both IP-based and HTTP-based protocols. This allows the audio segment files to flexibly adapt to changes in network bandwidth, and lost or missing segments can be repaired through individual requests. Ultimately, the audio segment files of this invention simultaneously guarantee real-time performance and stability, providing optimal audio quality across various network environments.

[0048] The queue (FIFO) structure of one embodiment of the present invention means a data structure that manages data according to the "First-In, First-Out" principle. The queue operates in a first-in, first-out manner, presenting data in a way that processes and stores data first. This queue structure is very effective in application scenarios where data is input sequentially and processed in the order of input.

[0049] A FIFO (First-In, First-Out) queue structure typically involves two main operations: the "enqueue" operation, which adds data, and the "dequeue" operation, which removes data from the front of the queue. This structure can be used in various systems where maintaining data order and consistency is crucial.

[0050] Queuing structures are particularly effective in real-time data processing, such as memory management, job scheduling, network packet processing, and audio and video streaming. For example, in an audio streaming environment, a FIFO (First-In, First-Out) queue structure can continuously stream audio data by storing it sequentially and providing the corresponding data in order according to playback requests. Furthermore, in HTTP-based streaming protocols, received audio segments are stored in a queue, processed sequentially according to playback order, and provided to the user with stable audio quality.

[0051] In this invention, a FIFO (First-In, First-Out) queue structure is used to improve data processing efficiency and ensure real-time performance and continuity. For example, after storing audio segment files sequentially in the storage unit using a queue structure, the control unit executes a process of decoding and transcoding the corresponding data according to the queue order. Thus, even if network latency or data loss occurs, continuity is maintained using the data stored in the queue, ensuring the stability of the data processing process.

[0052] Furthermore, the FIFO (First-In, First-Out) queue structure simplifies complex data flow and maintains consistency in processing order, helping to prevent data collisions and improve the overall reliability of the system. This feature is an important factor in the audio data processing system of this invention and can be effectively used in various network and data processing environments.

[0053] The audio segment file decoding and transcoding in one embodiment of the present invention is a process of converting audio data in a manner that conforms to multiple formats and playback environments, achieved through a combination of decoding and transcoding. Decoding is the operation of restoring the original audio data by decoding the compressed audio segment file, for example, the operation of restoring it to PCM (Pulse Code Modulation) data. Transcoding includes the operation of re-encoding the decoded audio data according to specific output requirements and converting it into a new format or quality.

[0054] Decoding is performed using a codec (encoder-decoder), which selects the appropriate decoder based on the data compression algorithm contained in the audio segment file. For example, AAC (Advanced Audio Coding) or MP3 (MPEG Audio Layer-3) audio segment files can be decoded using their respective codecs, thereby restoring the compressed audio data to a format close to the original. The decoding process includes removing unnecessary metadata from the audio stream and extracting the pure audio signal.

[0055] Transcoding is the process of re-encoding decoded audio data to convert it into an audio file format optimized for playback devices or network environments. During this process, audio attributes such as sampling frequency, bit rate, and number of channels are adjusted to generate audio data that meets the user's output requirements. For example, to transmit high-resolution audio data in environments with limited network bandwidth, data optimization can be achieved by transcoding at a low bit rate.

[0056] In this invention, the decoding and transcoding of audio segment files are dynamically performed according to the playback requirements of the user terminal and network conditions. During this process, metadata containing time information is used to maintain synchronization with the continuity of the audio data. For example, after merging audio segment files received via an HTTP-based protocol in MP4 container format, the control unit decodes them and extracts the original audio data. By performing transcoding according to user requirements, playable audio data is finally generated.

[0057] This decoding and transcoding process is a crucial component of multi-protocol compatible systems that support multiple protocols and audio formats, effectively addressing issues such as network latency, data loss, and the diversity of user environments. The decoding and transcoding method of this invention maintains the quality of audio data while enabling flexible data conversion to meet user requirements, providing a stable and consistent user experience.

[0058] In one embodiment of the present invention, the communication unit requests audio segment files from the second service provider at a first time interval based on HTTP, so as to receive multiple audio segment files corresponding to the first time interval.

[0059] In one embodiment of the present invention, the communication unit performs the function of requesting and receiving audio segment files from a second service provider via an HTTP-based protocol. The communication unit is designed to operate effectively according to the network environment, requesting audio data and receiving multiple audio segment files corresponding to specific time intervals at preset first time intervals. The first time interval can be adjusted according to user requirements or network conditions, and is typically set in seconds to ensure the continuity of the real-time stream.

[0060] The Communications Department uses an HTTP request mechanism to request each segment file individually, utilizing metadata including the unique identifier (ID) of each segment during the request. For example, for an audio segment file divided into 5-second segments, the Communications Department requests it in the format "segment_1.mp4" on the first request, and in the format "segment_2.mp4" on the second request, and receives them sequentially. The received segment file consists of multiple small audio data fragments, each containing metadata including timing information for continuous playback.

[0061] To account for network latency or data loss, the communication unit includes a retry mechanism. For example, if the reception of a specific segment file fails, the communication unit can retry the same file after a specified waiting time (e.g., 2 seconds) to complete the missing data. In this case, requesting and receiving multiple segments in parallel improves data reception efficiency and ensures smooth operation even in real-time data streaming environments.

[0062] As an implemented embodiment, the communication unit requests audio segment files divided into 3-second units from the second service provider and stores them in the storage unit in a FIFO manner. For example, in the first request, "segment_001.mp4" will be received, and in subsequent requests, "segment_002.mp4" and "segment_003.mp4" will be received sequentially. In this case, the communication unit will confirm the order based on the time information of each segment and store them in the storage unit in the sorted format. This method maintains the continuity of audio data in a real-time streaming environment and provides users with an uninterrupted playback experience.

[0063] Furthermore, the communication unit continuously monitors the network status. If network bandwidth decreases or latency occurs, adaptive strategies such as temporarily increasing the request interval of audio segments or reducing the segment size can be used. For example, under bandwidth-limited conditions, segments of 3 seconds can be combined into 6-second segments for request, thereby reducing network load and ensuring the stability of data transmission. This operation of the communication unit ensures that the system of the present invention can operate stably in various network environments.

[0064] In one embodiment of the present invention, the control unit extracts the plurality of audio segment files from the storage unit in a FIFO manner. After identifying the metadata corresponding to the plurality of audio segment files and confirming their continuity, the control unit merges them in MP4 container format. The segments in the MP4 file are sorted and synchronized in accordance with the time information.

[0065] In one embodiment of the present invention, the control unit sequentially retrieves multiple audio segment files from the storage unit in a FIFO (First-In, First-Out) manner, and confirms the continuity of the data by analyzing the metadata contained in each segment file. The metadata includes the start and end times, sampling rate, bit rate, and encoder information of each segment file, thereby ensuring that the segments are connected in the correct order.

[0066] The control unit can analyze the timeline information of each segment based on metadata to identify missing or duplicate segments. For example, if the end time of "segment_001.mp4" is 10 seconds and the start time of "segment_002.mp4" is 10 seconds, it can be confirmed that the two segments can be connected consecutively. Conversely, if the start time of "segment_002.mp4" is identified as 12 seconds, it is considered that 2 seconds of data has been missed. In this case, in order to fill the missed interval, the control unit will generate replacement data or request the missed segment again through the communication unit.

[0067] If multiple audio segments are confirmed to be consecutively connected, the control unit can merge them using an MP4 container. During the merging process, the audio data of each segment is integrated through a single line within the MP4 container, and the segments are sorted according to a precise timeline based on metadata. For example, the data of "segment_001.mp4" will be assigned to the interval from 0 seconds to 10 seconds within the MP4 container, and the data of "segment_002.mp4" will be assigned to the interval from 10 seconds to 20 seconds. In this way, MP4 files sorted according to the timeline will maintain continuity, ensuring uninterrupted playback on the user's terminal.

[0068] Furthermore, to ensure synchronization of segments within a file merged in MP4 container format, the control unit utilizes timestamp information. Each segment will be accurately replayed and the decoding order maintained within the MP4 file based on the PTS (Presentation Time Stamp) and DTS (Decoding Time Stamp). For example, if the PTS at the end time of "segment_001.mp4" is inconsistent with the start time of "segment_002.mp4", the control unit will correct the interval between the timestamps to maintain the consistency of the playback order.

[0069] As one embodiment, the control unit retrieves five audio segment files (from "segment_001.mp4" to "segment_005.mp4") received from the network from the storage unit using a FIFO method. After confirming that all segments are contiguous by analyzing their respective metadata, the control unit merges them into a single MP4 file. The merged MP4 file contains 50 seconds of audio data, with each segment precisely ordered in 10-second increments on the timeline. This file is then transmitted to the user terminal and played back, providing the user with a continuous, uninterrupted audio experience.

[0070] The control unit continuously monitors the synchronization status of the merged MP4 files. If synchronization fails, the segments are reordered using a reprocessing algorithm or the merging method is adjusted to resolve the issue. This process ensures the quality and continuity of the audio data, providing a stable playback experience even when the network environment changes.

[0071] In one embodiment of the present invention, the control unit generates a transcoded file that transcodes the MP4 file according to the sound output requirements of the user terminal, and generates decoded data that is restored to PCM (Pulse Code Modulation) data by extracting the audio stream of the transcoded file.

[0072] In one embodiment of the present invention, the control unit performs transcoding on an MP4 file based on the user terminal's audio output requirement information. This audio output requirement information includes the user terminal's supported codecs, sampling frequency, bit rate, and audio channel settings. The control unit analyzes the audio stream within the MP4 file to perform a conversion operation that conforms to the output format required by the user terminal; the resulting file is called a transcoded file.

[0073] The transcoding process involves converting the audio data within the original MP4 file into a new format. For example, if the audio stream in an MP4 file is encoded in AAC format and the user terminal requests PCM data, the control unit decodes the AAC data to convert it into original PCM data. This process aims for lossless audio data conversion and uses algorithms that minimize data quality degradation.

[0074] During transcoding, the control unit adjusts the sampling frequency and bit rate of the audio stream. For example, if the user terminal requires a sampling frequency of 44.1kHz and a bit rate of 128kbps, even if the existing audio stream has a sampling frequency of 48kHz and a bit rate of 256kbps, the control unit will lower these values ​​to convert it into a format that meets the user terminal's requirements. During the conversion process, high-quality resampling and bit rate conversion algorithms are employed to minimize audio quality loss and generate optimal data.

[0075] After the transcoding process is complete, the control unit extracts the audio stream from the transcoded file to reconstruct PCM data. PCM data is the raw data of the digitized audio signal and is the basic format for audio playback on user terminals. The process of reconstructing PCM data is performed through a decoding algorithm, maintaining signal accuracy and timeline synchronization throughout the process. For example, if an audio stream encoded with an AAC codec is decoded, the control unit will use the decoder to reconstruct the signal in the time region and convert it into PCM data.

[0076] As an example, if a 1-minute audio stream within an MP4 file is stored in AAC format, the control unit will perform a transcoding operation by converting it into 44.1kHz, 16-bit, stereo PCM data. The resulting PCM data will then be transmitted according to the user terminal's audio output requirements, providing the user with a smooth audio experience.

[0077] Furthermore, to ensure the continuity of audio data during transcoding and decoding, the control unit employs a synchronization algorithm. Based on timeline information (presentation time stamp and decoding time stamp), it ensures that data is processed in the correct order. If any data is missed during transmission, it is identified, and a repair operation is performed to generate complete PCM data. This implementation method enables uninterrupted audio playback at the user terminal and supports stable operation in various network environments.

[0078] In one embodiment of the present invention, the control unit adjusts the sampling frequency and bit rate of the decoded data in accordance with the output requirement information of the user terminal, and then sends the decoded data in real time. During the sending of the decoded data, new audio segment files are extracted from the storage unit to maintain the continuity of the MP4 file and are updated in real time.

[0079] In one embodiment of the present invention, the control unit adjusts and sends the decoded data in real time based on the output requirement information of the user terminal. The output requirement information may vary depending on the playback environment supported by the user terminal, such as sampling frequency, bit rate, and audio channel structure. The control unit can use this as a basis to optimize the decoded data. For example, if the user terminal requires a sampling frequency of 48kHz and a bit rate of 192kbps, the control unit can convert the decoded PCM data into the corresponding format. During this conversion process, high-quality resampling algorithms and bit rate adjustment algorithms are employed to minimize sound quality loss.

[0080] Sampling frequency adjustment is the process of reconstructing PCM data. It adjusts the interval between samples while maintaining the time axis information of the data to meet output requirements. Bit rate adjustment is the process of controlling the amount of data transmitted. Data size can be adjusted through compression algorithms to effectively utilize network bandwidth. For example, if network speed is limited or the terminal configuration is low, the control unit can provide a smooth streaming environment by reducing the bit rate.

[0081] During the transmission of decoded data, the control unit can retrieve new audio segment files from the storage unit in real time, thereby updating the MP4 file. This is an important operation for maintaining the continuity of audio data, sequentially retrieving new segment files stored in the storage unit in a queue-based (FIFO) structure and adding them to the existing MP4 file. The added data is synchronized with the time information of the MP4 container format, thus achieving uninterrupted transmission of the audio stream. For example, when a 2-second audio segment newly added to the storage unit is merged into the MP4, the control unit verifies the timeline continuity with the existing data and uses a synchronization algorithm for sorting if necessary.

[0082] During real-time updates, the control unit checks the metadata of the MP4 files to confirm continuity and integrity. The metadata includes information such as the start and end times of segment files, sampling frequency, and bit rate. The control unit uses this information to correct inconsistencies between files. For example, if the start time of a new segment file differs from the end time of an existing file, the control unit will reset the corresponding segment and, if necessary, request any missing data to maintain file consistency.

[0083] Furthermore, during audio data transmission, the control unit monitors the network status in real time. If the network is unstable or experiences delays, the control unit stores the data being transmitted in a temporary buffer to prevent data loss. Once the network stabilizes, transmission will resume. This ensures uninterrupted audio playback for users, and the control unit operates stably in various network environments.

[0084] This implementation goes beyond simply adjusting and sending decoded data; it provides users with an optimal audio experience through real-time data updates and continuous maintenance, as well as flexible responses based on network conditions. Ultimately, this embodiment helps to effectively overcome network and terminal limitations while ensuring the quality and continuity of audio data.

[0085] In one embodiment of the present invention, the control unit adjusts the transcoding setting value according to the network bandwidth of the user terminal. If the network bandwidth decreases from 10Mbps to 5Mbps, the sampling frequency is adjusted from 48kHz to 32kHz according to the inverse relationship of the bandwidth reduction, and the bit rate is adjusted from 320kbps to 192kbps according to the logarithmic ratio of the bandwidth reduction.

[0086] In one embodiment of the present invention, the control unit monitors the network bandwidth status of the user terminal in real time and adjusts the transcoding settings according to changes in network bandwidth to optimize the quality of audio data. For example, if the network bandwidth decreases from 10 Mbps to 5 Mbps, the control unit adjusts the sampling frequency from 48 kHz to 32 kHz based on the inverse relationship between the bandwidth reduction and the change. This is to maintain data transmission efficiency by reducing the size of transmitted data according to the bandwidth reduction, thereby minimizing latency or data loss caused by network congestion.

[0087] Adjusting the sampling frequency in an inverse proportional relationship can control the amount of data transmission while minimizing the degradation of sound quality. If the sampling frequency is reduced from 48kHz to 32kHz, the amount of data transmission will be reduced by about 33%, which can minimize the perceptible degradation of sound quality. This helps to prevent interruptions that may occur due to reduced network bandwidth and optimizes the user experience.

[0088] Furthermore, the control unit adjusts the bit rate based on a logarithmic ratio of the bandwidth reduction. For example, if the bandwidth is reduced from 10 Mbps to 5 Mbps, the bit rate is adjusted from 320 kbps to 192 kbps. The rationale for adjusting the bit rate logarithmically is that there is no linear relationship between bit rate and audio quality. At bit rates above a specified level, the improvement in audio quality is minimal, while network bandwidth consumption increases proportionally. Therefore, by using a logarithmic ratio, the bit rate can be effectively reduced according to the bandwidth reduction, minimizing the degradation in audio quality delivered to the user.

[0089] For example, if the bitrate is adjusted from 320kbps to 192kbps, the data size is reduced by about 40%, while the audio quality provided to most users remains at a satisfactory level. This logarithmic scaling adjustment plays a particularly important role in environments with chaotic networks or limited bandwidth.

[0090] The control unit flexibly responds to changes in network conditions by adjusting these settings, ensuring smooth audio data reproduction at the user terminal. Furthermore, simultaneous adjustment of the sampling frequency and bit rate maximizes data transmission efficiency and provides sound quality that considers user experience. Ultimately, according to an embodiment of the present invention, maintaining a stable data stream even under changes in the network environment effectively prevents potential quality degradation during audio data transmission, which is its technical significance.

[0091] In one embodiment of the present invention, the control unit sets the transmission priority based on the synchronization delay time generated when transmitting decoded data. If the synchronization delay time increases from 50ms to 100ms, the segment transmission interval is adjusted from 2ms to 4ms according to the exponential ratio of the increase in the synchronization delay time, so as to reorder the segments based on metadata in a way that maintains synchronization.

[0092] In one embodiment of the present invention, the control unit monitors the synchronization delay time generated during the transmission of decoded data in real time and uses this information as a basis to dynamically set the transmission priority. The synchronization delay time represents the time difference between when audio data arrives at the user terminal and is played back. If this value increases, the quality of the audio transmitted to the user may degrade. The control unit uses this delay time as a basis to adjust the data transmission method to minimize synchronization problems.

[0093] For example, if the synchronization delay increases from 50ms to 100ms, the control unit calculates the increase in delay (50ms) exponentially to adjust the transmission interval from the current 2ms to 4ms. The rationale for this exponential adjustment is that as the synchronization delay increases above a predetermined level, the effectiveness of adjusting the data transmission speed increases proportionally. For instance, if the delay initially increases slightly, the effect of adjusting the data transmission interval may be minimal, but if the delay increases geometrically, a larger adjustment is needed to offset the impact.

[0094] During the adjustment of the transmission interval from 2ms to 4ms, the control unit analyzes the timing information of each segment using metadata to reorder the segments so that the data is accurately arranged along the timeline. This process effectively prevents potential interruptions or discontinuities between audio segments when the user terminal replays the data. For example, if a specific segment is out of sequence on the timeline, the control unit will refer to the metadata to reset it to the correct position. This ensures that synchronization problems do not occur.

[0095] Compared to simply adjusting the transmission interval linearly, adjusting the transmission interval exponentially is more effective in solving synchronization problems. This is because the greater the synchronization delay, the more aggressively the transmission interval can be adjusted to optimize data processing speed. For example, an increase of 100ms in delay has a greater impact than an increase of 50ms; the importance of adjusting the transmission interval increases exponentially. This approach is beneficial for maintaining audio data quality even in environments with unstable network conditions or high data processing volumes.

[0096] Ultimately, the control unit of this invention utilizes the exponential relationship between synchronization delay time and transmission interval to optimize data in real time. This prevents a decline in user experience due to network latency, providing a stable audio data transmission and playback environment. This technology is of great significance in designing systems with highly variable network conditions, maximizing the efficiency of synchronization problem resolution to simultaneously improve project reliability and user satisfaction.

[0097] In one embodiment of the present invention, the control unit identifies inconsistencies in the timing and sampling information contained in the audio segments by analyzing the metadata of the audio segments in the MP4 file, sorts and reconstructs the inconsistencies based on the queue (first-in-first-out, FIFO) structure stored in the storage unit to generate a new MP4 file that maintains the continuity of the timeline, and sends the new MP4 file to the user terminal by transcoding the new MP4 file.

[0098] In one embodiment of the present invention, the control unit precisely identifies inconsistencies between the time information and sampling information of each audio segment by analyzing the metadata contained in each audio segment within the MP4 file. The time information represents the start and end times of each segment, and the sampling information includes the sampling frequency and bit rate of the audio data recorded for that segment. For example, even if the time information of a specific segment is recorded as 00:01:00 instead of 00:01:05, the sampling information is not fixed at 48kHz but becomes 44.1kHz. In this case, there is a possibility that the playback might hinder the continuity of the timeline. If such an inconsistency is identified, the control unit sorts and reconstructs segments based on the queue (FIFO) structure stored in the storage unit to maintain the continuity of the timeline. In this process, the control unit first reorders the order within the queue based on the time information of each segment. If an inconsistency in the sampling information is found, it performs a conversion to the same sampling frequency. For example, to merge segments recorded at 48kHz and segments recorded at 44.1kHz, the control unit upsamples the data recorded at 44.1kHz to 48kHz. In this transformation process, a filtering algorithm will be used to minimize the distortion between samples.

[0099] When merging segments reconstructed based on inconsistent information to generate a new MP4 file, the control unit adjusts the time intervals of each segment proportionally to maintain timeline continuity. For example, if the time interval between two segments should be set to 100ms, but is measured to be 120ms, the control unit adjusts the time interval to 100ms to restore continuity. During this process, to minimize the impact of time interval adjustments on latency, real-time correction is performed based on metadata.

[0100] Afterwards, the control unit transcodes the reconstructed MP4 file and sends it to the user terminal. During transcoding, the sampling frequency and bit rate are adjusted according to the user terminal's output requirements, using optimal settings based on network bandwidth and user needs. For example, if the network bandwidth decreases to 5Mbps, the bit rate is adjusted to 192kbps and the sampling frequency is adjusted to 44.1kHz to optimize data transmission speed and maintain audio quality.

[0101] This effectively solves the timeline distortion problem caused by inconsistencies in timing and sampling information between audio segments. In particular, by utilizing a FIFO (First-In, First-Out) queue structure in the storage unit to sort and reconstruct data in real time, a stable audio stream can be provided even under changing network conditions. This represents a significant advancement beyond simply transmitting audio data, offering a sophisticated processing mechanism that takes into account dynamic data variability. This technological advancement provides users with an uninterrupted audio playback environment, a stability that is particularly crucial for live streaming services or environments requiring high-quality audio transmission.

[0102] If the control unit of an embodiment of the present invention identifies network delays or data omissions that may occur during the transmission process, it performs a re-request and reprocessing of the omissions based on the corresponding metadata.

[0103] In one embodiment of the present invention, the control unit identifies in real time any network latency or data loss that may occur during audio data transmission. For example, if a specific audio segment file is not received due to reduced network bandwidth or increased packet loss rate, the control unit confirms the missing segment by comparing it with metadata. The metadata includes information such as the unique ID of each audio segment, start and end times, sampling frequency, and bit rate, which allows for accurate identification of missing data.

[0104] If missing data is identified, the control unit uses the address and request parameters of the service provider recorded in the metadata to re-request the missing data. For example, if the ID of a specific segment is identified as "SEG_12345" and the corresponding data is missing, the control unit can send an HTTP request message in the format of "GET / audio / SEG_12345" to request to receive the data again. If the network latency increases to more than 100ms, the continuity of real-time transmission is maintained by setting data re-requests according to priority.

[0105] During the process of receiving missed data, the control unit generates temporary data based on metadata to prevent transmission interruptions. This temporary data is constructed by repeatedly processing existing segment data, inserting blank data, and generating replacement audio data (e.g., for mute processing). During this temporary data insertion process, the timeline continuity is maintained based on existing metadata to minimize interruptions in the user's audio stream.

[0106] If a re-requested data is received, the control unit dynamically integrates the missing data into the existing audio stream. During this process, the missing segments are sorted based on the time information of the metadata to adjust the timeline of the existing stream in real time. For example, if a missing segment corresponds to the interval 00:01:10 to 00:01:15, it is inserted into the corresponding time interval of the MP4 file after receiving the corresponding data to restore continuity.

[0107] This goes beyond simple data re-requests, providing a mechanism to stably maintain audio data quality under uncertain conditions such as network latency and data loss. For example, if the packet loss rate increases from 5% to 10%, the control unit can proportionally increase the data re-request frequency, adjusting it to maintain a data recovery rate above 95%. This action provides real-time response capabilities based on changes in network conditions, especially in environments such as live streaming services, ensuring the quality of the user experience.

[0108] Ultimately, the control unit of one embodiment of the present invention can flexibly respond to changes in network conditions and optimize the continuity and stability of the audio stream through precise data management utilizing metadata. This enables the provision of high-quality audio services even in unstable network environments, which is of both technical and practical significance.

[0109] In one embodiment of the present invention, the control unit adjusts the audio output volume based on the PCM data proportionally according to the user terminal's usage environment, and uses an automatic gain control (AGC) algorithm to maintain the audio output volume between 70dB and 90dB.

[0110] In one embodiment of the present invention, the control unit automatically adjusts the audio output volume generated based on PCM data according to the user terminal's usage environment. It is designed to use an Automatic Gain Control (AGC) algorithm to stably maintain the user terminal's audio output volume between 70dB and 90dB during this process. The AGC algorithm can proportionally adjust the volume when the audio output volume is too low or too high to ensure the quality of the user experience.

[0111] Specifically, the control unit calculates the current volume level by analyzing the RMS (Root Mean Square) value of the audio signal sampled from the PCM data. If the calculated RMS value exceeds the volume reference range (70dB to 90dB), the AGC algorithm will adjust the gain value in real time to achieve volume adjustment. For example, if the current output volume is measured to be 65dB, the AGC will proportionally increase the gain value to compensate for the insufficient volume, thereby adjusting it to the target volume of 75dB. Conversely, if the output volume is measured to be 95dB, the AGC will decrease the gain value to reduce the excessive volume, thereby reducing it to the target volume range of 85dB.

[0112] During the AGC algorithm adjustment process, the gain value is adjusted proportionally based on how much the output volume exceeds the reference range. For example, if the output volume exceeds or falls below the reference range by more than 5dB, AGC will adjust the gain value proportionally. On the other hand, if the volume exceeds or falls below the reference range by less than 5dB, fine-tuning is used to maintain volume stability. This adjustment method prevents distortion or inconvenience caused by excessive volume changes, achieving natural volume variations.

[0113] Furthermore, the control unit adjusts the sensitivity of the AGC algorithm according to the usage environment. For example, if the user terminal is used outdoors, the ambient noise may necessitate increasing the required volume. Taking this into account, the control unit measures the ambient noise level and adjusts the reference volume range to 75dB to 95dB. Conversely, if used indoors, where ambient noise is low, the reference volume range is set to 65dB to 85dB to prevent inconvenience caused by excessive volume output.

[0114] The adoption of this AGC algorithm plays a crucial role in meeting user-environment-based volume requirements and maintaining output volume quality. In particular, by performing volume adjustments proportionally, it minimizes audio distortion or output signal clipping. For example, if the output volume increases from 80dB to 90dB, AGC proportionally increases the gain by 1.2 times, thereby amplifying the output signal without audio distortion. This effectively prevents potential signal quality degradation during volume adjustments, providing users with an optimal listening environment.

[0115] Ultimately, the control unit of one embodiment of the present invention can optimize the quality of audio output by adjusting the volume in accordance with the user terminal's usage environment. In particular, by performing proportional volume adjustments based on the AGC algorithm and real-time environmental analysis, the consistency of the user experience and ease of listening are ensured. This is a necessary factor in providing high-quality audio services and has technical significance in achieving both reliability and flexibility in network streaming-based audio data processing systems.

[0116] In one embodiment of the present invention, the control unit identifies whether the data of an audio segment file sent via an HTTP-based protocol is lost using a Cyclic Redundancy Check (CRC). If the identification result shows that the data of the audio segment file sent via the HTTP-based protocol is lost, the unit requests the lost audio segment file from the second service provider. The unit repeatedly outputs existing segments or inserts temporary replacement data in a manner that maintains the continuity of the MP4 file until the requested audio segment file is received. After the loss is recovered, the unit updates the MP4 file and sends it to the user terminal in real time.

[0117] In one embodiment of the present invention, the control unit identifies whether data in audio segment files sent via an HTTP-based protocol is lost using Cyclic Redundancy Check (CRC). CRC is used to verify the integrity of transmitted data and can quickly detect data errors using a very small verification code. The control unit calculates the CRC value for each received audio segment file and compares it with the CRC value contained in the original data, thereby identifying whether data is lost or altered.

[0118] If data loss is detected, the control unit immediately requests the missing audio segment file from the second service provider. In this case, the frequency or timing of the re-request is adjusted based on network conditions and the importance of the data. For example, if the lost data accounts for more than 5% of the entire stream, the number of re-requests is increased to improve the speed of data recovery. This achieves the effect of ensuring both effective use of network resources and reliable data recovery by adjusting the re-request frequency in proportion to the proportion of data loss.

[0119] Until a requested audio segment is received, the control unit repeatedly outputs existing segments or inserts temporary replacement data to maintain the continuity of the MP4 file. For example, if the lost segment is 2 seconds long, the previous segment is repeatedly played to fill the gap, or temporary data that has been muted is inserted to minimize the interruption of the audio stream. This method helps prevent a decline in user experience due to data loss and maintains the continuity of the audio stream.

[0120] If the loss recovery process is complete and a re-requested audio segment file is received, the control unit updates the MP4 file. During this process, temporarily inserted data or repeatedly output segments are replaced with the original data. The updated MP4 file ensures audio stream consistency by reordering timeline information and metadata. Afterward, the control unit sends the updated MP4 file to the user terminal in real time to achieve smooth audio playback.

[0121] This effectively prevents audio quality degradation due to data loss, providing users with a stable streaming service. In particular, data loss detection via CRC can quickly and accurately verify the integrity of audio segment files, which is essential in environments where real-time data transmission is crucial. Furthermore, using temporary replacement data in the event of data loss can improve user experience while effectively utilizing network resources.

[0122] In one embodiment of the present invention, the control unit monitors the integrity of the transmitted decoded data. If the data loss rate increases from 1% to 5%, the data repair request frequency is increased from 5 times to 10 times in a corresponding inverse relationship with the increase in the loss rate. The repaired data is then integrated in real time and sent to the user terminal.

[0123] In one embodiment of the present invention, the control unit continuously monitors the integrity of the transmitted decoded data and analyzes the data loss rate in real time. The data loss rate is defined by the proportion of transmitted data that is not acknowledged by the receiving terminal or is lost, which directly affects the quality of the audio stream. Therefore, if the data loss rate increases from 1% to 5%, the control unit dynamically adjusts the data repair request frequency according to the inverse proportional relationship of the increase in the loss rate. For example, if the data loss rate increases from 1% to 5%, the repair request frequency is increased from the existing 5 times to 10 times, thereby accelerating the data repair speed.

[0124] This inverse relationship considers that the higher the data loss rate, the more immediate the response is required. If the data loss rate is low, data integrity can be maintained with relatively fewer repair requests. The higher the loss rate, the more frequent the requests for repairing lost data must be to ensure the continuity of the audio stream. Therefore, instead of simply increasing the number of repair requests linearly in proportion to the loss rate, the frequency is adjusted inversely to optimize network load and enhance real-time repair capabilities.

[0125] The repair request will be optimized taking into account network bandwidth and response time. If the lost data is repaired, the control unit will integrate it into the existing decoded data in real time. During this process, the repaired data will be inserted into the appropriate position within the original audio stream based on the location and time information of the lost data. The inserted data maintains timeline continuity by synchronizing with other audio segments. This synchronization process will be achieved using metadata without degrading audio quality.

[0126] For example, if a user experiences a 5% data loss rate due to network latency, the control unit can adjust the repair request frequency to 10 times to ensure the lost data is repaired as quickly as possible. The repaired data will be integrated into the existing decoded data, allowing the user to experience uninterrupted audio streaming. If the loss rate decreases to 2%, the control unit will readjust the repair request frequency to 7 times to improve network resource efficiency.

[0127] As a crucial aspect and effect, adjusting the repair request frequency inversely to the data loss rate is essential for maintaining the quality of the audio stream in real time. Increasing the repair request frequency geometrically with increasing data loss could lead to excessive network resource consumption. Therefore, adjusting the frequency inversely to the increment of the loss rate allows for both real-time response and resource optimization. Furthermore, by synchronously inserting real-time repaired data into the existing stream, users can minimize the quality degradation caused by data loss and experience stable audio playback.

[0128] Ultimately, this embodiment can maintain the quality and continuity of the audio stream even in the event of data loss, enabling effective use of network resources and real-time data repair. This significantly improves user satisfaction and makes a substantial contribution to providing stable service across various network environments.

[0129] In one embodiment of the present invention, the control unit identifies the audio quality of the audio output at a preset second time interval based on the PCM data of the decoded MP4 file. If the identification result is that audio distortion or noise is detected, the audio data corresponding to the corresponding time interval is compared with the existing segments in the storage unit for further processing. If the cause of the audio distortion is network latency, the network status is monitored, and audio data using transcoding settings that include a new sampling frequency and bit rate corresponding to the network conditions is sent.

[0130] In one embodiment of the present invention, the control unit monitors the audio quality of the audio stream in real time based on the PCM data generated by decoding the MP4 file and thereby identifies potential audio distortion or noise. The audio quality evaluation benchmark is the sampling frequency and bit rate of the PCM data. If frequency distortion, clipping, or the signal-to-noise ratio (SNR) exceeds the benchmark value within a specific time interval, it is determined that the corresponding audio data is distorted. For example, if the SNR decreases to less than 30 dB or the distortion rate increases by more than 5% in a specific frequency band, the control unit compares the PCM data corresponding to the corresponding time interval with existing audio segments in the storage unit.

[0131] During this process, the control unit analyzes the cause of distortion by comparing the PCM data with the original data in the same interval within the storage unit. For example, if irregularities in the timeline continuity or sampling frequency of the PCM data are detected, it is considered a problem caused by network latency. If network latency is determined to be the cause, the control unit monitors the network status in real time, identifying situations where latency exceeds 100ms or the packet loss rate increases by more than 3%. Based on this network status, the control unit adjusts the transcoding settings to reprocess the audio data.

[0132] Specifically, a new sampling frequency and bit rate will be adopted based on network conditions. For example, if the network bandwidth decreases from 5 Mbps to 3 Mbps, the control unit will adjust the sampling frequency from the current 48 kHz to 32 kHz and reduce the bit rate from the current 320 kbps to 192 kbps. This adjustment is performed to optimize data transmission speed based on the reduced network bandwidth, maintaining a balance between audio quality and data transmission by proportionally adjusting the sampling frequency and bit rate.

[0133] The reprocessed audio data is synchronized with existing segment data and metadata in the storage unit. This replaces distorted data with the original data and sends it to the user terminal in real time while maintaining synchronization. For example, if distorted audio data generated in a specific time interval is due to network latency, the control unit will transmit data that has been reprocessed to minimize distortion, thereby allowing the user to experience a stable, distortion-free audio stream.

[0134] As a key feature and benefit, the control unit of this invention monitors the quality of the audio stream in real time and dynamically adjusts the transcoding settings in accordance with the network status, thereby ensuring both data transmission efficiency and audio quality. The audio stream can be stably maintained even with reduced network bandwidth by proportionally adjusting the sampling frequency and bit rate. Furthermore, the reprocessing function of existing segment data can immediately repair data loss or distortion, thus contributing to an improved user experience.

[0135] In one embodiment of the present invention, the control unit adopts a dynamic parameter adjustment algorithm based on the user terminal's usage environment and the characteristics of the audio data. The average sample energy of the audio data, network bandwidth, and the remaining battery capacity of the user terminal are set as key variables, and the audio transmission and processing methods can be optimized based on the relationship between the various variables.

[0136] In one embodiment of the present invention, the control unit employs a dynamic parameter adjustment algorithm based on the user terminal's usage environment and the characteristics of the audio data to optimize the transmission and processing of the audio data. This algorithm sets the average sample energy of the audio data, network bandwidth, and the user terminal's remaining battery power as key variables, and can derive optimized parameters based on the correlation between these variables in real time.

[0137] For example, the average sample energy of audio data represents the output strength of the audio signal. High-energy audio data requires wider bandwidth and processing resources. If the average sample energy increases from -10dBFS to -5dBFS, the control unit will proportionally increase the sampling frequency from 48kHz to 96kHz to prevent signal distortion and will change the transcoding settings accordingly. Conversely, if the average sample energy decreases to below -20dBFS, the bit rate will be adjusted from 128kbps to 192kbps to ensure the signal-to-noise ratio (SNR) and maintain audio clarity.

[0138] Furthermore, network bandwidth directly affects the speed and quality of audio data transmission. If the network bandwidth decreases from 10Mbps to 5Mbps, the control unit will adjust the sampling frequency to 44.1kHz inversely proportional to the bandwidth reduction, and reduce the bit rate from 320kbps to 192kbps based on the logarithmic relationship of the bandwidth reduction. This minimizes data loss and transmission latency while maintaining audio quality to the maximum extent. Conversely, if the network bandwidth increases from 10Mbps to 20Mbps, the control unit will increase the sampling frequency and bit rate to 48kHz and 256kbps respectively, supporting high-quality audio transmission.

[0139] The user terminal's remaining battery level is also an important factor in parameter adjustment. If the battery level drops below 50%, the control unit will adjust the sampling frequency to 32kHz and reduce the bit rate to 128kbps to improve energy efficiency. Conversely, if the battery level remains above 80%, the sampling frequency and bit rate will be adjusted based on high quality in a manner suitable for the user's environment, thereby optimizing the quality of the audio stream.

[0140] This dynamic parameter adjustment algorithm derives the optimal audio processing method in real time by reflecting the interaction between various variables, thereby maintaining a balance between audio quality and resource utilization in various usage environments. For example, in complex situations such as high average sample energy, reduced network bandwidth, and low battery capacity, the control unit dynamically changes the transcoding settings based on the relationships between various variables to ensure that the user experience does not deteriorate.

[0141] As a significant achievement and benefit, the adjustment benchmarks and proportional relationships of the various variables set in this embodiment significantly improve efficiency and flexibility compared to existing fixed audio processing methods. By adjusting the sampling frequency based on the average sample energy, signal distortion is prevented. The logarithmic proportional relationship between network bandwidth and bit rate maximizes data transmission efficiency. Furthermore, by taking battery capacity into account, optimization is achieved to minimize the energy consumption of the user terminal. This approach maintains the stability and quality of audio transmission and processing even under complex variable environments and is beneficial for optimizing the utilization of network and hardware resources.

[0142] In one embodiment of the present invention, the average sample energy of the audio data is calculated based on the peak amplitude within the data segment. If the average sample energy increases from 20dB to 40dB, the output volume is increased from 5% to 10% according to the logarithmic ratio of the energy increase. If the energy decreases to less than 20dB, the output volume is reduced to 15%, thereby maintaining volume balance.

[0143] In one embodiment of the present invention, the average sample energy of audio data is calculated based on the peak amplitude within a data segment. This serves as a benchmark for dynamically adjusting the output volume by quantitatively analyzing the intensity and dynamic range of the audio signal. The average sample energy is defined as the square root of the value obtained by squaring the peak amplitude of each sample within the audio data and dividing it by the total number of samples in the data segment. This calculation method quantitatively represents the average intensity of the audio signal, providing fundamental data for adjusting the signal volume.

[0144] In this embodiment, if the average sample energy increases from 20dB to 40dB, the output volume will increase from 5% to 10% according to the logarithmic proportion of the energy increase. In this case, the logarithmic proportion is designed to prevent abrupt changes in output volume, thus providing the user with a stable and consistent listening experience. For example, when the average sample energy is 20dB, the output volume is set to 5%; if the average sample energy increases to 40dB, the output volume will increase to 10%. This takes into account the non-linear effects of the energy increase to present a natural volume change for auditory perception.

[0145] Conversely, if the average sample energy decreases to less than 20dB, the output volume decreases by 15% in proportion to the decrease in energy. This is a measure to prevent excessive volume reduction and maintain volume balance when signal strength decreases. For example, if the average sample energy decreases to 15dB, the output volume is reduced by 15% compared to the existing volume, and the reduced signal volume is adjusted to an appropriate level.

[0146] As a key advantage and benefit, employing a logarithmic ratio between average sample energy and output volume allows for more precise and stable volume control compared to simple proportional adjustments. This logarithmic ratio provides natural volume variations even under low or high signal strength, minimizing auditory discomfort that can result from extreme peak amplitude changes. In particular, gradually increasing the output volume from 20dB to 40dB maintains the dynamic range of the audio data and ensures signal clarity.

[0147] Furthermore, even with the average sample energy reduced to less than 20dB, signal distortion is prevented by decreasing the output volume, minimizing the impact of external factors such as background noise. This provides a system that simultaneously improves audio data quality and listening experience while operating stably in various environments. This approach significantly improves the efficiency of audio data processing and user satisfaction, achieving more flexible and precise volume control compared to existing fixed output methods.

[0148] In one embodiment of the present invention, the network bandwidth is measured by the connection speed between the terminal and the data processing device. If the bandwidth decreases from 10Mbps to 5Mbps, the sampling frequency of the transmitted audio data is adjusted from 48kHz to 24kHz according to the inverse relationship of the bandwidth reduction. If the bandwidth increases to 10Mbps or more, the sampling frequency is restored to 48kHz or more.

[0149] In one embodiment of this invention, network bandwidth, measured by the connection speed between the user terminal and the data processing device, serves as a key variable for optimizing the efficiency and quality of audio data transmission and reception. Network bandwidth represents the amount of data that can be transmitted per second, measured in Mbps (Megabits per second). Changes in bandwidth directly impact potential delays, data loss, and distortion during audio data transmission. By monitoring and adaptively processing bandwidth in real time, stable audio quality is ensured.

[0150] In this embodiment, if the network bandwidth decreases from 10 Mbps to 5 Mbps, the sampling frequency of the transmitted audio data is adjusted from 48 kHz to 24 kHz according to the inverse relationship of the bandwidth reduction. This is a measure to minimize network load by reducing the amount of data transmitted when the network bandwidth decreases. By using an inverse proportional relationship, the sampling frequency is gradually reduced in accordance with the network bandwidth reduction, thereby adjusting the network state. For example, if the bandwidth decreases to 7.5 Mbps, the sampling frequency is adjusted to approximately 36 kHz, thereby achieving stable audio transmission without data loss.

[0151] Conversely, if the bandwidth increases to 10 Mbps or higher, the sampling frequency is restored to 48 kHz or higher, allowing the transmission of high-quality audio data. For example, if the bandwidth increases to 15 Mbps, the sampling frequency is set to 96 kHz, enabling the transmission of high-resolution audio data. This approach provides the flexibility to optimize audio quality in real time based on changes in network conditions.

[0152] As a significant advantage and effect, by employing the inverse relationship between network bandwidth and sampling frequency, stable audio data transmission can be achieved even in bandwidth-constrained environments. While existing systems based on fixed sampling frequencies suffer from data loss or quality degradation when bandwidth decreases, this embodiment addresses this issue by dynamically adjusting the sampling frequency according to bandwidth changes. In particular, it effectively reduces network load by decreasing the data transmission volume while simultaneously reducing the sampling frequency when bandwidth decreases.

[0153] Furthermore, with increased bandwidth, high-resolution audio quality can be provided by restoring the sampling frequency, thereby optimizing the user experience. For example, even if network conditions change in a live streaming or video conferencing environment, this embodiment can maintain stable quality and effectively utilize network resources. This approach provides a competitive advantage in delivering high-quality audio services under various network conditions and significantly improves user satisfaction.

[0154] In one embodiment of the present invention, the remaining battery level of the user terminal is monitored in real time by a battery sensor within the terminal. If the remaining battery level decreases from 50% to 20%, the audio data processing speed is reduced by 30% according to the exponential ratio of the decrease in battery level, and the transcoding algorithm is changed to a low power mode. If the remaining battery level decreases to less than 20%, the audio transmission quality is limited to 96kbps, thereby maximizing energy efficiency.

[0155] In one embodiment of the present invention, the remaining battery level of the user terminal is monitored in real time by a battery sensor built into the terminal. The measured battery level is used as a key variable to optimize the efficiency of data processing and audio transmission, and various dynamic adjustment algorithms are employed to minimize battery consumption and maintain user experience.

[0156] First, if the battery level decreases from 50% to 20%, the audio data processing speed is reduced by 30% based on an exponential proportionality to the decrease in battery capacity. This is a measure designed to gradually suppress battery consumption, minimizing data processing speed reduction and audio quality degradation. For example, if the battery level is 40%, the data processing speed is reduced to approximately 85% of its original speed; if the level decreases to 30%, the processing speed is adjusted to approximately 70%. This approach helps extend the device's usage time by effectively suppressing battery consumption based on the decrease in battery capacity.

[0157] Furthermore, if the remaining battery level drops below 20%, the audio data transcoding algorithm will switch to a low-power mode. In low-power mode, power consumption is reduced by simplifying complex transcoding processes or omitting unnecessary data processing. For example, power consumption can be reduced by omitting complex filtering processes for high-resolution audio streams or by maintaining processing speed through simple compression algorithms.

[0158] Additionally, if the battery level drops below 20%, the audio transmission quality will be limited to 96kbps. This setting ensures a minimum acceptable audio quality while reducing battery consumption. 96kbps is a suitable bit rate for maintaining voice data quality and reducing the amount of data transmitted, thus maintaining a balance between network bandwidth and power consumption. For example, when the battery level drops to 10%, limiting the transmission quality to 96kbps can achieve a reduction in battery consumption of more than 40% compared to previous methods.

[0159] The significance of this approach lies in maximizing the energy efficiency of the user terminal by dynamically adjusting data processing and transmission algorithms based on changes in battery level. In existing fixed-setting methods, there is a problem where the terminal suddenly shuts down and data processing and transmission quality deteriorates as battery level decreases. Conversely, this embodiment predictively manages the decrease in battery level and controls battery consumption by adjusting processing speed and transmission quality based on an exponential proportional relationship, thereby solving this problem.

[0160] As a result, stable service can still be provided to users even when battery power is low, and energy can be effectively managed by extending the device's usage time. In particular, when battery power is low, the device's energy efficiency is maximized through a low-power mode, thereby improving user satisfaction and ensuring continuous service. This goes beyond simple power saving; it provides optimal action based on battery usage conditions, thus guaranteeing a differentiated user experience.

[0161] In one embodiment of the present invention, the control unit analyzes the interaction between the average sample energy and the network bandwidth to interrupt real-time transmission and store the corresponding data when the bandwidth decreases to below 5 Mbps and the average sample energy reaches above 30 dB. The control unit then transmits the data in batch transmission mode according to a specified period. It analyzes the impact of the adjustment results of each variable on the audio quality in real time. If the audio quality score decreases to below 80%, a warning message informing the user of the quality degradation is transmitted.

[0162] [The Korean National Research and Development Program supports this invention]

[0163] [Project Number] 2410005793

[0164] [Project Number] 00444700

[0165] [Department Name] Trade, Industry and Resources Department

[0166] [Name of the Project Management (Specialized) Agency] Korea Institute for Industrial Technology Evaluation

[0167] [Research Project Name] Materials and Components Technology Development (R&D)

[0168] [Research Topic Title] Development of a Broadcast and Communication Converged Media System and Subscription Service for Next-Generation Electric and Connected Vehicles Targeting Europe (and China)

[0169] [Name of the organization conducting the project] DENEX Co., Ltd.

[0170] [Research Period] July 1, 2024 – December 31, 2026

Claims

1. A client-side audio data transmission and processing system based on multi-protocol compatibility, characterized in that, Including data processing devices, The data processing device includes: The communications department will send and receive audio data with at least one of a first service provider based on IP and a second service provider based on an HTTP protocol. The storage unit stores audio segment files received through the communication unit in a queue structure, wherein the queue structure is a first-in, first-out (FIFO) structure; and The control unit outputs audio data by decoding and transcoding the audio segment files stored in the storage unit.

2. The client audio data transmission and processing system based on multi-protocol compatibility according to claim 1, characterized in that, This includes a user terminal that connects to the data processing device via a wired or wireless network to output sound. The communication unit requests audio segment files from the second service provider at a first time interval based on HTTP, in order to receive multiple audio segment files corresponding to the first time interval. The control unit retrieves the multiple audio segment files from the storage unit in a first-in-first-out manner. After confirming continuity by identifying the metadata corresponding to each of the multiple audio segment files, it merges them according to the MP4 container format. The segments within the MP4 file are sorted and synchronized according to the corresponding time information.

3. The client audio data transmission and processing system based on multi-protocol compatibility according to claim 2, characterized in that, The control unit generates a transcoded file that corresponds to the user terminal's sound output requirements, and extracts the audio stream from the transcoded file to generate decoded data that is restored to pulse code modulation data. The control unit adjusts the sampling frequency and bit rate of the decoded data in real time according to the output requirements of the user terminal, and then sends the decoded data. During the transmission of the decoded data, the control unit retrieves new audio segment files from the storage unit to maintain the continuity of the MP4 file and updates it in real time. The control unit adjusts the transcoding settings accordingly based on the network bandwidth of the user terminal. If the network bandwidth decreases from 10Mbps to 5Mbps, the sampling frequency is adjusted from 48kHz to 32kHz according to the inverse relationship of the bandwidth reduction, and the bit rate is adjusted from 320kbps to 192kbps according to the logarithmic relationship of the bandwidth reduction.

4. The client audio data transmission and processing system based on multi-protocol compatibility according to claim 3, characterized in that, The control unit uses the synchronization delay time generated when sending decoded data as a benchmark to set the transmission priority. If the synchronization delay time increases from 50ms to 100ms, the segment transmission interval is adjusted from 2ms to 4ms accordingly based on the exponential ratio of the increase in synchronization delay time. The segments are then reordered based on metadata to maintain synchronization. The control unit analyzes the metadata of the audio segments within the MP4 file to identify inconsistencies in the time and sampling information contained in the segments and generates inconsistency information. The inconsistent information is sorted and reconstructed based on a queue structure stored in the storage unit to generate a new MP4 file that maintains the continuity of the timeline. The queue structure is a first-in, first-out (FIFO) structure. The new MP4 file is transcoded and sent to the user terminal. If network delays or data omissions that may occur during the transmission are detected, the control unit will re-request and reprocess the omissions based on the corresponding metadata.

5. The client audio data transmission and processing system based on multi-protocol compatibility according to claim 4, characterized in that, The control unit adjusts the audio output volume based on the pulse code modulation data proportionally according to the user terminal's usage environment. An automatic gain control algorithm is used to maintain the audio output volume between 70dB and 90dB. The control unit uses cyclic redundancy check to identify whether data in the audio segment file sent via the HTTP-based protocol is lost. If the identification result indicates that data in the audio segment file sent via the HTTP-based protocol is lost, it requests the lost audio segment file from the second service provider. Until the audio segment file obtained through a re-request is received, existing segments are repeatedly output or temporary replacement data is inserted in a manner that maintains the continuity of the MP4 file. After the loss repair is completed, the MP4 file is updated and sent to the user terminal in real time.

6. The client audio data transmission and processing system based on multi-protocol compatibility according to claim 5, characterized in that, The control unit monitors the integrity of the decoded data sent. If the data loss rate increases from 1% to 5%, the data repair request frequency is increased from 5 times to 10 times in accordance with the inverse relationship of the increase in the loss rate. The repaired data is integrated in real time and sent to the user terminal. The control unit identifies the sound quality of the audio output at a preset second time interval based on the pulse code modulation data of the decoded MP4 file. If the identification result is audio distortion or noise is detected, the audio data corresponding to the corresponding time interval is reprocessed by comparing it with existing segments in the storage unit. If the cause of the audio distortion is network latency, then monitor the network status and send audio data using transcoding settings that include a new sampling frequency and bit rate corresponding to the network conditions.

Images