Method, apparatus, device, medium and product for audio processing

Abstract

The present disclosure provides an audio processing method, device, equipment, medium and product, relates to the field of data processing, and particularly relates to the fields of voice technology and information flow. The specific implementation scheme is as follows: an audio processing method comprises the following steps: periodically acquiring audio data according to a first time period; assembling the acquired audio data according to a second time period to obtain assembled audio data, wherein the second time period is an integer multiple of the first time period; performing audio mixing processing on the assembled audio data; and sending the audio mixing processed audio data to an audio playing device. The present disclosure can assemble the periodically acquired audio data, perform audio mixing processing on the assembled audio data, can reduce the number of times of audio mixing processing, save audio processing time, and thus reduce the delay of audio playing.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to the fields of voice technology and information flow technology. Background Technology

[0002] With the advancement of technology, electronic devices are widely used in users' lives, enabling them to communicate via video or voice. For example, users can use electronic devices for online voice calls, multi-person conferences, live interactive sessions, and other scenarios. Microphones and other sound acquisition devices capture the speaker's voice and the surrounding sound, transmitting the captured sound signals to a playback device for playback.

[0003] The transmission of audio between system devices causes a delay in the sound signal. In addition, the acquired audio signal needs to undergo encoding, transmission, mixing and synthesis, which is a complicated process and causes a delay in audio playback. Summary of the Invention

[0004] This disclosure provides a method, apparatus, device, medium, and product for audio processing.

[0005] According to one aspect of this disclosure, an audio processing method is provided, comprising: periodically acquiring audio data according to a first time period; assembling the acquired audio data according to a second time period to obtain assembled audio data, wherein the second time period is an integer multiple of the first time period; performing mixing processing on the assembled audio data; and sending the mixed audio data to an audio playback device.

[0006] According to another aspect of this disclosure, an audio processing apparatus is provided, comprising: an acquisition module for periodically acquiring audio data according to a first time period; an assembly module for assembling the acquired audio data according to a second time period to obtain assembled audio data, wherein the second time period is an integer multiple of the first time period; a processing module for mixing the assembled audio data; and a sending module for sending the mixed audio data to an audio playback device.

[0007] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to said at least one processor; wherein the memory stores instructions executable by said at least one processor, said instructions being executed by said at least one processor to enable said at least one processor to perform the audio processing method described in any one of this disclosure.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the audio processing method described in any one of this disclosure.

[0009] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the audio processing method described in any one of this disclosure.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0012] Figure 1 This is a flowchart illustrating the audio processing method according to this disclosure;

[0013] Figure 2 This is a flowchart illustrating the method for assembling acquired audio frames to obtain assembled audio data according to this disclosure.

[0014] Figure 3 This is a flowchart illustrating the method for sending mixed audio data to an audio playback device according to this disclosure;

[0015] Figure 4 This is a flowchart illustrating the audio processing method according to this disclosure;

[0016] Figure 5 This is a flowchart illustrating the method for acquiring audio data according to this disclosure;

[0017] Figure 6 This is a block diagram of a device for acquiring temperature according to this disclosure;

[0018] Figure 7 This is a block diagram of a device for acquiring temperature according to this disclosure;

[0019] Figure 8 This is a block diagram of an electronic device used to implement the temperature acquisition method of the embodiments of this disclosure. Detailed Implementation

[0020] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0021] With the advancement of technology, Virtual Reality (VR) is gradually entering people's lives. VR technology combines reality and virtuality, using computer technology to simulate the real world and create a computer simulation system that allows users to immerse themselves in the virtual environment. VR technology utilizes real-life data and electronic signals generated by computer technology, combining these signals with various output devices to transform them into phenomena that people can perceive. These phenomena can be real objects or invisible substances represented through three-dimensional models.

[0022] Through VR technology, users can experience the most realistic sensations in the virtual reality world, gaining a sense of immersion and truly realizing human-computer interaction. During operation, users can freely manipulate the environment and receive the most realistic feedback, which has been recognized by more and more users.

[0023] In VR technology, in order to bring users a realistic and immersive auditory experience with a sense of space, it is necessary to reproduce and simulate a large amount of real-time user sound, so that users can experience the effect of being there through headphones and other means.

[0024] Therefore, this disclosure provides an audio processing method that can save time in sound mixing and reduce latency caused by mixing.

[0025] Figure 1 This is a flowchart illustrating the audio processing method according to this disclosure. For example... Figure 1 As shown, the audio processing method includes the following steps.

[0026] In step S101, audio data is acquired periodically according to the first time period.

[0027] In this embodiment, the acquired audio is mixed and then played back, with audio data acquired at a first time period. The acquired audio can come from different sound sources, such as human voices or ambient sounds, and the locations of the sound sources can also be different. Audio data is acquired once every first time period, meaning the acquired audio data is in frames. The first time period can be determined according to requirements. For example, if the first time period is 20ms, then audio data is acquired at 0ms, 20ms, 40ms, and so on.

[0028] In step S102, the acquired audio data is assembled according to the second time period to obtain assembled audio data. The second time period is an integer multiple of the first time period.

[0029] The audio data obtained in step S101 is assembled to obtain assembled audio data. The audio data is acquired with a first time period as the period and assembled according to a second time period, which is a period longer than the first time period and an integer multiple of the first time period. Taking the first time period as 20ms as an example, the second period can be 40ms, 60ms, 80ms, etc. It can be understood that after assembling the audio data, the length of the unit audio data is greater than the length of the received audio data.

[0030] In this embodiment of the disclosure, audio data integrity verification is also included. For example, the integrity of the audio data is verified by information such as the duration of the audio data to ensure that the audio data used for assembly is complete audio data.

[0031] In step S103, the assembled audio data is mixed.

[0032] In step S104, the mixed audio data is sent to the audio playback device.

[0033] The assembled audio data undergoes mixing, overlaying, and other processing, and the resulting audio data is sent to an audio playback device for playback.

[0034] In summary, according to the audio processing method disclosed herein, the audio data obtained periodically over a first time period is assembled to obtain assembled audio data that is an integer multiple of the first time period, and the assembled audio data is mixed. This reduces the number of mixing operations, thereby saving audio processing time, improving processing efficiency, and ultimately reducing audio playback latency.

[0035] In an exemplary embodiment of this disclosure, the periodically acquired audio data includes at least one audio data stream. Different audio data streams are generated by different sound sources. Each audio data stream has an audio stream identifier (id) to distinguish between different audio streams, and the audio stream ID is unique. The audio data stream also includes a timestamp, which is used to characterize the sound emission time corresponding to the audio stream. The time sequence of the sounds emitted by the corresponding audio stream can be determined by the timestamp.

[0036] Figure 2 This is a flowchart illustrating a method for assembling acquired audio frames to obtain assembled audio data, according to this disclosure. Figure 2 As shown, the method includes the following steps.

[0037] In step S201, the number of audio data to be assembled is determined based on the second time period.

[0038] In step S202, based on the amount of audio data, the timestamp corresponding to the current time and the audio data stream corresponding to the timestamp are determined sequentially from the currently acquired audio data to obtain the first timestamp and the first audio data stream corresponding to the first timestamp.

[0039] In step S203, the first audio data stream is assembled sequentially according to the order of the first timestamps to obtain assembled audio data.

[0040] In this embodiment of the disclosure, the acquired audio data is assembled according to a second time period, and the number of audio data to be assembled is determined by the second time period. The number of audio data to be assembled is determined by the ratio of the second time period to the first time period, which is a multiple of the first time period. For example, if periodic audio data is acquired at a period of 20ms, three 20ms audio data points are acquired and assembled according to a time period of 60ms.

[0041] In an exemplary embodiment of this disclosure, audio data is acquired at a period of 20ms, and the 20ms dual-channel 48k sampling rate audio data is assembled into 60ms dual-channel 48k sampling rate audio data.

[0042] In the currently acquired audio data, the timestamp corresponding to the current time and the audio data stream corresponding to the timestamp are determined. Specifically, this process can be as follows: In the audio data acquired within the first time period, the first timestamp corresponding to the first time period is determined, and the first audio data stream corresponding to the first timestamp is determined; in the audio data acquired within the second time period, the first timestamp corresponding to the second time period is obtained, and the first audio data stream corresponding to the first timestamp is determined, and so on, obtaining the first audio data stream corresponding to the number of audio data points. For example, in the audio data acquired at time 0, the first audio data stream corresponding to the current time with a first timestamp of 0 is determined. The first audio data stream may include at least one audio stream ID. Similarly, in the audio data acquired at time 20ms, the first audio data stream corresponding to the timestamp of 20ms is determined, and so on. Through these steps, a number of first timestamps equal to the number of audio data points are obtained, along with multiple first audio data streams corresponding to each first timestamp.

[0043] The multiple first audio data streams are sequentially assembled according to their timestamps to obtain assembled audio data. For example, if periodic audio data is acquired at a 20ms interval, three 20ms audio data streams are acquired at a 60ms interval and then assembled. From the audio data acquired at times 0, 20ms, and 40ms, the timestamp corresponding to time 0 is identified as the first audio data stream at time 0, the timestamp at time 20ms, and the timestamp at time 40ms, respectively. Within each timestamp-corresponding first audio data stream, different data stream IDs are distinguished. Assembling the data streams in the order of timestamps 0, 20ms, and 40ms yields audio data of length 60ms.

[0044] In summary, according to the audio processing method disclosed herein, the audio data obtained periodically in the first time period is assembled according to the timestamp sequence of the audio data streams in the audio data and distinguishing different audio data stream identifiers to obtain the assembled audio data in the second time period. While assembling the audio data, the order of sounding of different audio data streams is ensured, thus providing a guarantee for the mixing processing of the audio data.

[0045] Figure 3 This is a flowchart illustrating the method for sending mixed audio data to an audio playback device according to this disclosure. Figure 3 As shown, the method includes the following steps.

[0046] In step S301, the audio data after mixing is split according to the first time period.

[0047] In step S302, the split audio data is sent to the audio playback device.

[0048] In this embodiment, audio data is acquired periodically according to a first time period, and then assembled according to a second time period to obtain assembled audio data. The assembled audio data is then mixed, and the mixed audio data is sent to an audio playback device for playback. Understandably, the audio data obtained from the mixing process has the same length as the assembled audio data. To facilitate transmission, the mixed audio data is split according to the first time period, and the split audio data is sent to the audio playback device. For example, audio data is acquired periodically at a 20ms cycle, and three 20ms audio data segments are acquired at a 60ms time period and then assembled. The assembled 60ms audio data is mixed, resulting in another 60ms audio data segment. After transmission, the processed audio data is split into 20ms audio data segments for transmission, i.e., audio data is sent every 20ms with a 20ms transmission cycle. Understandably, the audio data can be split sequentially based on the order of timestamps in the audio data.

[0049] In summary, the audio processing method disclosed herein assembles audio data to obtain assembled audio data, mixes the assembled audio data, splits the mixed audio data, and sends the split audio data. This can further improve the transmission speed of the mixed audio data, ensure transmission quality, and thus further reduce latency.

[0050] Figure 4 This is a flowchart illustrating the audio processing method according to this disclosure. For example... Figure 4 As shown, the audio processing method includes the following steps.

[0051] In step S401, audio data is acquired periodically according to the first time period.

[0052] In this embodiment of the disclosure, audio data is acquired periodically at a first time period.

[0053] In step S402, the number of audio data to be received included in the audio data queue is determined.

[0054] In this embodiment of the disclosure, the process of acquiring and assembling audio data, the process of mixing the assembled audio data, and the process of sending the mixed audio data correspond to independent threads, which can be implemented through a pipeline structure, further improving processing efficiency.

[0055] In an exemplary embodiment of this disclosure, the processing time cycle of the process of acquiring and assembling audio data, the process of mixing the assembled audio data, and the process of sending the mixed audio data can be controlled to a preset time. When the processing time of the current pipeline stage exceeds the preset time, scheduling is performed to compress the processing time of adjacent subsequent processing stages, thereby further realizing the control of processing time.

[0056] In step S403, in response to the number of audio data to be received in the audio data queue exceeding a preset number, a preset number of audio data to be received is deleted from the audio data queue.

[0057] In an exemplary embodiment of this disclosure, audio data is transmitted in the form of a data queue. That is, audio data is obtained from the queue, the audio data is assembled, the assembled audio data is pushed to the mixing queue, the assembled audio data received from the mixing queue is mixed, the processed audio data is sent to the sending queue for transmission, the audio data in the sending queue is split, and the split audio data is sent.

[0058] When acquiring audio data, if the processes of acquiring and assembling audio data, mixing the assembled audio data, or sending the mixed audio data are not completed within the preset time, the number of audio data to be received in the data queue will increase. When the number of audio data to be received in the audio data queue exceeds the preset number, it will cause the audio data queue to be blocked. The preset number of audio data to be received will then be deleted from the audio data queue, i.e., the preset number of audio data to be received will be cleared. For example, if audio data is acquired periodically at a 20ms interval, and the preset number is set to 5, then when the number of audio data to be received in the audio data queue is greater than 5 (i.e., more than 5 20ms data points), 100ms of audio data to be received will be deleted from the audio data queue.

[0059] In step S404, the acquired audio data is assembled according to the second time period to obtain assembled audio data. The second time period is an integer multiple of the first time period.

[0060] In step S405, the assembled audio data is mixed.

[0061] In step S406, the mixed audio data is sent to the audio playback device.

[0062] The acquired audio data is assembled according to the second time period to obtain assembled audio data. The length of the assembled audio data is greater than that of the acquired audio data. The assembled audio data is then mixed and superimposed, and the mixed audio data is sent to the audio playback device for playback.

[0063] In summary, according to the audio processing method disclosed herein, the audio data acquired periodically in a first time period is assembled, and the assembled audio data is mixed. If the number of audio data to be received in the audio data queue exceeds a preset number, the preset number of audio data to be received is deleted from the audio data queue. This can save audio processing time and improve processing efficiency while ensuring audio processing quality, thereby reducing audio playback delay.

[0064] Figure 5 This is a flowchart illustrating the method for acquiring audio data according to this disclosure. Figure 5 As shown, the method includes the following steps.

[0065] In step S501, the storage location of the audio data in the remote dictionary service is determined based on the Kafka messaging system.

[0066] In step S502, audio data is obtained from a remote dictionary service based on the storage location.

[0067] In this embodiment, audio data acquisition can be achieved through the Kafka messaging system in conjunction with a Remote Dictionary Server (Redis) storage database. Kafka is an open-source stream processing platform, a high-throughput distributed publish-subscribe messaging system that unifies online and offline message processing through a parallel loading mechanism and provides real-time messaging through a cluster. Kafka features high performance, persistence, multi-replica backup, and horizontal scaling capabilities. The Redis storage database is used for caching, event publishing and subscription, high-speed queues, and other scenarios, periodically writing updated data to disk or appending modification operations to an append-only log file. Audio data acquisition is based on push messages from the Kafka messaging system, which include the storage information of the audio data in the Redis storage database. The storage message includes the storage location of the audio data in the database, the timestamp information of the corresponding audio data, etc. Based on the storage location of the audio data included in the push message, the audio data is retrieved from the Redis storage database for subsequent mixing processing, etc.

[0068] In summary, based on the audio processing method disclosed herein, using Kafka and Redis to acquire audio data simplifies the system architecture, ensures the coordinated synchronization of audio data caching, further improves data acquisition speed, and enhances execution efficiency.

[0069] In an exemplary embodiment of this disclosure, the audio processing method is applied to audio processing in a VR scene. This scene involves multi-person interaction within the same space, allowing users to experience real-time sound effects from various sources and distances. The acquired audio data originates from online multi-person audio data, undergoes mixing processing, and is then sent to the user. The user can experience an immersive auditory effect of multiple voices emanating from the virtual space via a personal computer, mobile phone, wearable device, etc. Since the distance between the multiple voices emanating from the virtual space and the current user's virtual avatar varies, when the sound source is far from the current user, the voices of multiple people need to be mixed before being sent to the user to achieve a realistic auditory effect in the scene. The audio processing method of this disclosure can effectively reduce the latency of audio reception for the user.

[0070] The processes of acquiring and assembling audio data, mixing the assembled audio data, and sending the mixed audio data are each handled by independent threads, implemented through a pipelined structure, which further improves processing efficiency. Audio data is transmitted in a queue during these processes. If the number of audio data to be received in the queue of an adjacent process exceeds a preset limit, the preset number of audio data to be received is removed from the audio data queue, reducing queue congestion and thus minimizing sound latency.

[0071] Audio data is acquired periodically according to a first time period. This audio data is obtained online in real time from multiple people. The integrity of the acquired audio data can be verified. Complete audio data is used for assembly. Based on a second time period longer than the first time period, the number of audio data to be assembled is determined. Based on the number of audio data, the timestamp corresponding to the current time and the corresponding audio data stream are determined sequentially from the currently acquired audio data to obtain the first timestamp and the first audio data stream corresponding to the first timestamp. The first audio data stream is then assembled sequentially according to the order of the first timestamps to obtain the assembled audio data corresponding to the second time period. By assembling the audio, the number of calls to the mixing algorithm in the mixing process is reduced, thereby reducing audio processing time and latency.

[0072] In an exemplary embodiment of this disclosure, the process of mixing assembled audio data involves mixing the assembled audio data received from the mixing queue and sending the processed audio data to the sending queue. The process of sending the mixed audio data involves splitting the audio data in the sending queue. This can be done by sending the split audio data based on Real-Time Communication (RTC), which provides stable, high-quality real-time audio and video services, enabling real-time audio and video applications and further reducing audio communication latency.

[0073] In summary, according to the audio processing method disclosed herein, in a VR scenario, the audio data acquired periodically in the first time period is assembled to obtain the assembled audio data in the second time period, and the assembled audio data is mixed. This reduces the number of mixing operations, thereby saving audio processing time, improving processing efficiency, and reducing audio playback latency.

[0074] Based on the same concept, embodiments of this disclosure also provide an audio processing apparatus.

[0075] It is understood that the apparatus provided in this disclosure includes hardware structures and / or software modules corresponding to each function in order to achieve the above-mentioned functions. In conjunction with the units and algorithm steps of the various examples disclosed in this disclosure, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of this disclosure.

[0076] Figure 6 This is a block diagram of an audio processing apparatus according to the present disclosure.

[0077] like Figure 6 As shown, the audio processing apparatus 600 of this embodiment includes: an acquisition module 601, an assembly module 602, a processing module 603, and a transmission module 604.

[0078] The acquisition module 601 is used to periodically acquire audio data according to a first time period;

[0079] Assembly module 602 is used to assemble the acquired audio data according to the second time period to obtain assembled audio data. The second time period is an integer multiple of the first time period.

[0080] Processing module 603 is used to perform mixing processing on the assembled audio data;

[0081] The sending module 604 is used to send the mixed audio data to the audio playback device.

[0082] In an exemplary embodiment of this disclosure, the acquired audio data includes at least one audio data stream, and the at least one audio data stream has a stream identifier and a timestamp.

[0083] The assembly module 602 is further configured to: determine the number of audio data to be assembled based on the second time period; based on the number of audio data, sequentially determine the timestamp corresponding to the current time and the audio data stream corresponding to the timestamp from the currently acquired audio data to obtain a first timestamp and a first audio data stream corresponding to the first timestamp, wherein the number of first timestamps is the number of audio data; and sequentially assemble the first audio data stream according to the order of the first timestamps to obtain assembled audio data.

[0084] In an exemplary embodiment of this disclosure, the sending module 604 is further configured to: split the mixed audio data according to a first time period; and send the split audio data to an audio playback device.

[0085] Figure 7 This is a block diagram of an audio processing apparatus according to the present disclosure. Figure 7 As shown, the audio processing apparatus 600 of this embodiment further includes a determination module 605.

[0086] The determining module 605 is used to determine the number of audio data to be received included in the audio data queue; in response to the number of audio data to be received in the audio data queue exceeding a preset number, the preset number of audio data to be received is deleted from the audio data queue.

[0087] In an exemplary embodiment of this disclosure, the acquisition module 601 is further configured to: determine the storage location of the audio data in the remote dictionary service based on the Kafka messaging system; and acquire the audio data from the remote dictionary service based on the storage location.

[0088] In summary, according to the audio processing method disclosed herein, the audio data obtained periodically over a first time period is assembled to obtain assembled audio data that is an integer multiple of the first time period, and the assembled audio data is mixed. This reduces the number of mixing operations, thereby saving audio processing time, improving processing efficiency, and ultimately reducing audio playback latency.

[0089] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0090] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0091] Figure 8 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0092] like Figure 8 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded into random access memory (RAM) 703 from storage unit 708. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0093] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0094] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as audio processing methods. For example, in some embodiments, the audio processing methods may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the audio processing methods described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform audio processing methods by any other suitable means (e.g., by means of firmware).

[0095] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0096] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0097] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0098] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0099] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0100] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0101] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0102] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An audio processing method, comprising: According to the first time period, audio data is acquired periodically. The acquired audio data includes at least one audio data stream, and the at least one audio data stream has a stream identifier and a timestamp. Different audio data streams are generated by different sound sources. The stream identifier is used to distinguish different audio data streams. The audio data stream also includes a timestamp, which is used to characterize the sound time corresponding to the audio data stream. Based on the second time period, the number of audio data to be assembled is determined, wherein the second time period is greater than the first time period and is an integer multiple of the first time period; Based on the amount of audio data, the timestamp corresponding to the current time and the audio data stream corresponding to the timestamp are determined sequentially from the currently acquired audio data to obtain a first timestamp and a first audio data stream corresponding to the first timestamp, wherein the number of the first timestamps is the amount of audio data; According to the order of the first timestamps, the first audio data stream is assembled sequentially to obtain assembled audio data; The assembled audio data is then mixed. The mixed audio data is split according to the first time period and the split audio data is sent to the audio playback device. The method further includes: Determine the number of audio data to be received included in the audio data queue; In response to the number of audio data to be received in the audio data queue exceeding a preset number, the preset number of audio data to be received is deleted from the audio data queue.

2. The method according to claim 1, wherein, Acquire audio data, including: Based on the Kafka messaging system, the storage location of the audio data in the remote dictionary service is determined; The audio data is obtained from the remote dictionary service based on the storage location.

3. An audio processing apparatus, comprising: The acquisition module is used to periodically acquire audio data according to a first time period. The acquired audio data includes at least one audio data stream, and the at least one audio data stream has a stream identifier and a timestamp. Different audio data streams are generated by different sound sources. The stream identifier is used to distinguish different audio data streams. The audio data stream also includes a timestamp, which is used to characterize the sound generation time corresponding to the audio data stream. An assembly module is used to determine the number of audio data to be assembled based on a second time period. Based on the number of audio data, it sequentially determines the timestamp corresponding to the current time and the audio data stream corresponding to the timestamp from the currently acquired audio data to obtain a first timestamp and a first audio data stream corresponding to the first timestamp. The number of first timestamps is the number of audio data. The first audio data stream is assembled sequentially according to the order of the first timestamps to obtain assembled audio data. The second time period is an integer multiple of the first time period. The processing module is used to perform mixing processing on the assembled audio data; The sending module is used to split the mixed audio data according to the first time period and send the split audio data to the audio playback device. A determining module is used to determine the number of audio data to be received included in the audio data queue; in response to the number of audio data to be received in the audio data queue exceeding a preset number, the preset number of audio data to be received is deleted from the audio data queue.

4. The apparatus according to claim 3, wherein, The acquisition module is also used for: Based on the Kafka messaging system, the storage location of the audio data in the remote dictionary service is determined; The audio data is obtained from the remote dictionary service based on the storage location.

5. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the audio processing method according to any one of claims 1-2.

6. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the audio processing method according to any one of claims 1-2.

7. A computer program product comprising a computer program that, when executed by a processor, implements the audio processing method according to any one of claims 1-2.

Images