Audio transmission method and apparatus

Abstract

The application relates to the technical field of electronic products, in particular to an audio transmission method and device. The method comprises the following steps: a main device inputs an original audio stream into an audio encoder for encoding through a first sub-thread, and obtains a target audio stream coded from the audio encoder through a second sub-thread; and the main device sends the target audio stream to an auxiliary device. Since the first sub-thread does not need to obtain the target audio stream from the audio encoder, when the audio encoder codes slowly, the first sub-thread will not be blocked, and the original audio stream can be input into the audio encoder for coding in time. The original audio stream will not be delayed to be input into the audio encoder for coding, and the situation that audio frames are discarded will not occur; even if the audio encoder codes slowly, continuous target audio streams can be obtained, so that the auxiliary device can play fluent audio.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electronic products, and in particular to an audio transmission method and device. BACKGROUND

[0002] With the development of communication technology, different electronic devices can establish a communication connection, share data in real time, and cooperatively process the same transaction, such as cooperatively playing audio. When playing audio cooperatively, one electronic device (a primary device) can send an audio stream to be played to another interconnected electronic device (a secondary device), and the secondary device plays the audio stream. At this time, the user can indirectly receive and listen to the audio played by the primary device using the secondary device.

[0003] Among them, the audio stream to be played can be referred to as an original audio stream, which is generally not suitable for transmission, so the primary device needs to encode the original audio stream into a target audio stream suitable for transmission and send the target audio stream to the secondary device.

[0004] In related technologies, when the primary device sends the original audio stream to the secondary device, it first encodes the original audio stream into the target audio stream through a sub-thread, and then sends the target audio stream to the secondary device. When the sub-thread is blocked, the original audio stream cannot be input into the audio encoder in time for encoding, which will cause the target audio stream received by the secondary device to be not smooth, thereby causing the audio played by the secondary device to be not smooth. SUMMARY

[0005] The embodiments of the present application provide an audio transmission method and device, which can solve the problem of not smooth audio played by the secondary device when playing the audio of the primary device.

[0006] In a first aspect, an audio transmission method is provided, comprising:

[0007] The primary device encodes an original audio stream to be played into a target audio stream through a first sub-thread;

[0008] The primary device acquires the target audio stream from the audio encoder through a second sub-thread, and the second sub-thread is different from the first sub-thread;

[0009] The primary device sends the target audio stream to the secondary device.

[0010] For example, after obtaining the original audio stream, the main thread can create and start the first and second sub-threads. The first sub-thread inputs the original audio stream into the encoder for encoding. At the same time, the second sub-thread periodically accesses the audio encoder to obtain the encoded target audio stream from the audio encoder. After the second sub-thread obtains the target audio stream, it sends the target audio stream to the main thread, which can then package the target audio stream and send it to the auxiliary device.

[0011] In the embodiment of the present application, during the process of encoding and sending the original audio stream, since the first sub-thread does not need to obtain the target audio stream from the audio encoder, when the audio encoder encodes slowly, the first sub-thread will not be blocked and the original audio stream can be input into the audio encoder for encoding in a timely manner. The original audio stream will not be delayed in inputting the audio encoder for encoding, and no audio frames will be discarded. Even if the audio encoder encodes slowly, a continuous target audio stream can be obtained, thereby allowing the auxiliary device to play smooth audio.

[0012] In one embodiment, the master device sends the target audio stream to the slave device, comprising: the master device sends the target audio stream to the slave device through a main thread, and the first sub-thread obtains the original audio stream from the main thread.

[0013] In one embodiment, the second sub-thread calls the first notification object to send the target audio stream to the main thread.

[0014] In one embodiment, when the second sub-thread does not detect the first notification object, a second notification object is created, and the second notification object is called to send the target audio stream to the main thread.

[0015] For example, the main thread can create a first notification object when it starts. After obtaining the original audio stream, the first child thread inputs the original audio stream into the audio encoder for encoding, and the second child thread obtains the target audio stream from the audio encoder. After obtaining the target audio stream, if the second child thread detects the existence of the first notification object, it calls the first notification object and sends the target audio stream to the main thread. Conversely, if the second child thread does not detect the first notification object, it creates and calls the second notification object to send the target audio stream to the main thread.

[0016] In an embodiment of the present application, when the second sub-thread sends the target audio stream to the main thread, it first determines whether the pre-created first notification object exists. When the first notification object exists, it calls the first notification object to send the target audio stream to the main thread. When the first notification object does not exist, it creates and calls the second notification object to send the target audio stream to the main thread, which can avoid the occurrence of a null pointer situation.

[0017] In an embodiment, the second sub-thread creates and invokes the notification object to send the target audio stream to the main thread.

[0018] For example, the second sub-thread can directly create the notification object when starting. The second sub-thread can directly invoke the notification object to send the target audio stream to the main thread after obtaining the target audio stream from the audio encoder.

[0019] In the embodiments of the present application, the second sub-thread directly creates the notification object and invokes the notification object to send the target audio stream to the main thread, which can avoid the situation that the notification object cannot be invoked when the notification object is destroyed by the main thread, thereby avoiding the problem of null pointer.

[0020] In an embodiment, the first sub-thread and / or the second sub-thread are started when the main thread obtains the original audio stream.

[0021] For example, after starting, the main thread only creates the audio encoder, and when obtaining the original audio stream from the audio subsystem, the first sub-thread and the second sub-thread are created and started.

[0022] In the embodiments of the present application, the first sub-thread and the second sub-thread are started when the original audio stream is obtained, which can avoid the situation that the thread is idle before the original audio stream is obtained, thereby avoiding the processor resource occupied by the idle thread, and reducing the power consumption of the processor.

[0023] In an embodiment, the main device stops the started target thread if the original audio stream is not obtained within a preset time length, the target thread including the first sub-thread and / or the second sub-thread.

[0024] For example, after starting, the main thread can set a global thread stop flag corresponding to an audio encoder of a certain audio type. When obtaining the corresponding original audio stream from the audio subsystem, the main thread sets the thread stop flag to a first state value. Conversely, if the corresponding original audio stream is not obtained from the audio subsystem within a preset time length, the thread stop flag is set to a second state value, which is different from the first state value. The sub-thread continues to run when the thread stop flag is in the first state value, and ends automatically when the thread stop flag is in the second state value. Alternatively, the sub-thread can invoke a waiting method and enter a waiting state when the thread stop flag is in the second state value. The method of stopping the sub-thread can include but is not limited to the above examples, which are not limited in the embodiments.

[0025] In the embodiments of the present application, the main device stops the started sub-thread if the original audio stream is not obtained within a preset time length, which can avoid the idle thread, thereby avoiding the processor resource occupied by the idle thread, and reducing the power consumption of the processor.

[0026] In a second aspect, an audio transmission device is provided, comprising:

[0027] The encoding module is used for the main device to input the original audio stream to be played into the audio encoder through the first sub-thread for encoding to obtain the target audio stream;

[0028] An acquisition module, configured for the master device to acquire the target audio stream from the audio encoder through a second sub-thread, where the second sub-thread is different from the first sub-thread;

[0029] A sending module is used for the master device to send the target audio stream to the slave device.

[0030] Optionally, the sending module is specifically configured to enable the master device to send the target audio stream to the slave device through a main thread, and the first sub-thread obtains the original audio stream from the main thread.

[0031] Optionally, the acquisition module is specifically configured to cause the second sub-thread to call a first notification object to send the target audio stream to the main thread.

[0032] Optionally, the acquisition module is further configured to create a second notification object when the second sub-thread fails to detect the first notification object, and call the second notification object to send the target audio stream to the main thread.

[0033] Optionally, the acquisition module is specifically configured for the second sub-thread to create and call a notification object to send the target audio stream to the main thread.

[0034] Optionally, the first sub-thread and / or the second sub-thread is started when the main thread obtains the original audio stream.

[0035] Optionally, the apparatus further includes: a stopping module, configured to stop a started target thread if the master device fails to obtain the original audio stream within a preset time period, the target thread including the first sub-thread and / or the second sub-thread.

[0036] In a third aspect, an electronic device is provided, comprising: one or more processors and one or more memories; the one or more processors are coupled to the one or more memories, the one or more memories are used to store computer program code, the computer program code comprising computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the method described in the first aspect.

[0037] In a fourth aspect, a readable storage medium is provided, in which a computer program product is stored. The computer program product includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method described in the first aspect.

[0038] In a fifth aspect, a chip system is provided, which is applied to an electronic device. The chip system includes one or more processors, and the processor is used to call computer instructions to enable the electronic device to execute the method described in the first aspect.

[0039] In a sixth aspect, a computer program product is provided, comprising computer instructions, which, when executed on an electronic device, causes the electronic device to execute the method as described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present application is shown.

[0041] Figure 2 A schematic diagram of audio stream transmission provided in an embodiment of the present application is shown.

[0042] Figure 3 A schematic diagram of the transmission principle of an audio stream in related technology is shown.

[0043] Figure 4 A schematic diagram of the hardware structure of an electronic device suitable for the above method is shown.

[0044] Figure 5 A flowchart of the steps of an audio transmission method provided in an embodiment of the present application is shown.

[0045] Figure 6 A flow chart of an audio transmission method provided in an embodiment of the present application is shown.

[0046] Figure 7 A schematic diagram of the transmission principle of an audio stream provided in an embodiment of the present application is shown.

[0047] Figure 8 A structural diagram of an audio transmission device provided in an embodiment of the present application is shown.

[0048] Figure 9 A schematic structural diagram of another electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0049] The technical solutions of this application will be described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them.

[0050] In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular architectures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known methods, devices, circuits, and

[0051] The term "include" as used herein means the presence of the described feature, integer, step, operation, element, and / or component but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. The terms "include", "comprise", "have" and any variations thereof are meant to cover non-exclusive inclusions, unless otherwise specifically indicated.

[0052] Hereinafter, the terms "first" and "second" are used only for the purpose of description and can not be understood as indicating or implying relative importance or implying the number of the technical features indicated. Therefore, the features defined with "first" and "second" can explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise specified.

[0053] The term "and / or" as used herein merely describes an association relationship of associated objects, and means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects are a "or" relationship.

[0054] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present application is shown. The electronic device is, for example Figure 1 The mobile phone 1 and the tablet computer 2 shown in the figure can establish a communication connection between the mobile phone 1 and the tablet computer 2. The communication connection can be established through short-range wireless communication protocols such as near field communication (NFC), Bluetooth (BT), and wireless fidelity (WIFI), but is not limited thereto.

[0055] After the mobile phone 1 establishes a communication connection with the tablet computer 2, the mobile phone 1 and the tablet computer 2 can share audio, video and text data and the like to cooperatively process the same transaction. In the process of cooperatively playing audio, the mobile phone 1 as a master device can directly play an audio stream or send the audio stream to the tablet computer 2, and the tablet computer 2 as a secondary device plays the audio stream. Based on this, the user can directly listen to the audio played by the mobile phone 1 or indirectly listen to the audio played by the mobile phone 1 through the tablet computer 2, so as to flexibly switch between the mobile phone 1 and the tablet computer 2.

[0056] Taking voice call as an example, after the mobile phone 1 establishes a communication connection with another mobile phone, the mobile phone 1 and the other mobile phone can send audio streams to each other, and the audio stream includes one or more continuous audio frames. At this time, the other mobile phone sends the audio stream to the mobile phone 1 after collecting the audio stream of the voice signal sent by the user. The mobile phone 1 enters an audio playing process after receiving the audio stream. In the playing process, the mobile phone 1 as a master device can send the audio stream to be played to the tablet computer 2, and the tablet computer 2 as a secondary device plays the audio stream.

[0057] The audio stream includes not only the audio stream in the voice call process, but also the audio stream in the playing process of other audio types such as music, ringtone and background sound, but is not limited thereto.

[0058] The original audio stream played by the electronic device is generally not suitable for transmission, such as a pulse code modulation (PCM) format audio stream which is suitable for program processing but not suitable for transmission. Therefore, the master device needs to first encode the original audio stream into a target audio stream suitable for transmission, and send the target audio stream to the secondary device, such as an advanced audio coding (AAC) format audio stream. The secondary device can decode the original audio stream from the target audio stream and play it.

[0059] Taking voice call as an example, the audio stream sent by the other mobile phone to the mobile phone 1 is generally an AAC format audio stream. When the mobile phone 1 plays the audio stream, it first decodes the audio stream into a PCM format original audio stream, and then plays the original audio stream. Since the PCM format original audio stream is not suitable for transmission, the mobile phone 1 needs to first encode the PCM format original audio stream into an AAC format target audio stream, and then send the AAC format target audio stream to the tablet computer 2. The formats of the original audio stream and the target audio stream include but are not limited to the above examples.

[0060] Figure 2A transmission diagram of an audio stream is shown. When an application (APP) in a master device plays an audio stream, the original audio stream to be played is first input into an access unit of the master device, and then input into an input buffer area of an audio encoder. The audio encoder encodes the original audio stream in the input buffer area, and inputs the target audio stream obtained through encoding into an output buffer area of the audio encoder.

[0061] The master device caches the target audio stream in the output buffer area into a queue buffer, and then packs the target audio stream in the queue buffer into a data packet in a real-time transport protocol (RTP) format, and sends the RTP data packet to a secondary device by a communication module.

[0062] After the communication module of the secondary device receives the RTP data packet, the target audio stream is parsed from the RTP data packet, and then input into an access unit of the secondary device. The target audio stream in the access unit is input into an input buffer area of an audio decoder, the audio decoder decodes the target audio stream in the input buffer area, and inputs the decoded audio stream into an output buffer area. The audio stream in the output buffer area is input into a queue buffer of the secondary device, and the audio module converts the target audio stream in the queue buffer into a sound signal and outputs the sound signal.

[0063] The audio encoder and the audio decoder are, for example, media codec components in an operating system (OS) of an electronic device, or other software modules that can encode or decode an audio stream.

[0064] Figure 3 A sending principle diagram of an audio stream in the related art is shown. A coordination module is arranged in a master device, and the coordination module is, for example, a distribute mobile sensing development platform (DMSDP), which is used to provide functional support for multiple electronic devices to collaboratively process the same transaction. When the master device plays an original audio stream, the coordination module can encode the original audio stream, and send the target audio stream obtained through encoding to a secondary device.

[0065] Figure 3The audio subsystem shown is composed of various software modules in the operating system for playing audio streams. The audio subsystem can also be called an audio framework. For example, it can be composed of software modules such as the audio track, audio flinger, and audio driver in the operating system. When an application plays a raw audio stream, it first creates an audio track and then inputs the raw audio stream into the audio track's buffer (i.e. Figure 2 The audio subsystem is configured as a buffered access unit (see access unit). When the buffer is full, the audio guard mixes the audio streams from all audio tracks created in the operating system and sends the mixed audio stream to the audio module via the audio driver. The audio module converts the audio stream into a sound signal for output. The specific composition of the audio subsystem can be configured as required and is not limited in this embodiment.

[0066] like Figure 3 As shown in the figure, when the application (abbreviated as application) plays the original audio stream, the main thread of the collaborative module can create and start a child thread and create a notification object. The main thread can obtain the original audio stream from the audio track buffer and send the original audio stream to the child thread.

[0067] When the child thread receives the original audio stream, it inputs the original audio stream into the audio encoder for encoding. After inputting the original audio stream into the audio encoder, the child thread obtains the target audio stream from the audio encoder. If the target audio stream is obtained from the audio encoder, the notification object is called to send the target audio stream to the main thread, and then the steps of obtaining the original audio stream from the main thread and inputting the original audio stream into the audio encoder are continued. If the target audio stream is not obtained from the audio encoder, the process waits until the target audio stream is obtained, sends the target audio stream to the main thread, and then the steps of obtaining the original audio stream from the main thread and inputting the original audio stream into the audio encoder are continued. The main thread enters the target audio stream sent by the child thread into the cache queue, and packages the target audio stream in the cache queue and sends it to the auxiliary device.

[0068] It can be seen that during the execution of the child thread, it can only continue to obtain the original audio stream from the main thread after obtaining the target audio stream from the audio encoder. If the audio encoder encodes slowly, the child thread cannot obtain the target audio stream from the audio encoder in time, resulting in the child thread being unable to continue to obtain the original audio stream from the main thread, and the child thread will be blocked. When the child thread is blocked, the subsequent input original audio stream will be delayed in being sent to the audio encoder for encoding, resulting in a delay in the target audio stream received by the primary device, which in turn causes a delay in the target audio stream received by the secondary device.

[0069] Further, since the main thread continuously obtains the original audio stream from the audio subsystem, when the sub-thread is blocked for a long time, part of the audio frames in the original audio stream will be lost, which will cause part of the audio frames in the target audio stream to be lost. When the target audio stream received by the secondary device is delayed and / or part of the audio frames is lost, the audio played by the secondary device will not be smooth.

[0070] The embodiment of the present application provides an audio transmission method and device. When the primary device sends the original audio stream to the secondary device, the primary device inputs the original audio stream into an audio encoder through a first sub-thread for encoding, and obtains the target audio stream encoded by the audio encoder through a second sub-thread.

[0071] Since the first sub-thread does not need to obtain the target audio stream from the audio encoder, when the audio encoder encodes slowly, the first sub-thread will not be blocked, and the original audio stream can be input into the audio encoder for encoding in time. The original audio stream will not be delayed to be input into the audio encoder for encoding, and the audio frame will not be discarded. Even if the audio encoder encodes slowly, the main thread can obtain continuous target audio stream, so that the secondary device can play smooth audio.

[0072] It should be noted that the cooperative module in the above example obtains the original audio stream from the audio subsystem after being started and run, encodes the original audio stream, and sends the target audio stream encoded to the secondary device for playing. When the cooperative module stops running, the original audio stream can be played by the primary device according to the normal playing process. The cooperative module can be started or stopped by the user, and the control method of the cooperative module can be set according to the requirement, which is not limited in the embodiment.

[0073] The electronic device can also be a smart screen, a wearable electronic device, a vehicle-mounted terminal, an augmented reality (AR) device, a virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a projector, etc., but is not limited thereto.

[0074] Figure 4A hardware structure diagram of an electronic device suitable for the above method is shown. The electronic device 400 can include a processor 410, a wireless communication module 420, an audio module 430, a mobile communication module 440, a display screen 450, a storage module 460, and a power module 470. The electronic device can also include a microphone 431, a receiver 432, a speaker 433, an antenna 1 and an antenna 2, as well as a sensor module, a universal serial bus (USB) interface, an external memory interface, a key, a motor, an indicator, a subscriber identification module (SIM) card interface, etc., but is not limited thereto.

[0075] The processor 410 can include one or more processing units. For example, the processor 410 can include at least one of an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, a neural-network processing unit (NPU). Different processing units can be independent devices or integrated devices.

[0076] The wireless communication module 420 can provide wireless communication solutions such as wireless local area networks (WLAN), Bluetooth, near field communication, etc. on the electronic device 400, but is not limited thereto. The wireless communication module 420 can be one or more devices integrated with at least one communication processing module. The wireless communication module 420 receives electromagnetic waves via the antenna 1, frequency-modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 410. The wireless communication module 420 can also receive signals to be sent from the processor 410, frequency-modulate and amplify them, and convert the signals into electromagnetic wave radiation via the antenna 1.

[0077] The electronic device 400 can implement audio functions through the audio module 430, the microphone 431, the receiver 432, the speaker 433, and the application processor, etc. For example, audio playback, recording, etc.

[0078] The audio module 430, for example Figure 2The audio module shown is used to convert the audio stream into an analog sound signal output, and can also be used to convert the analog sound signal into an audio stream. In some embodiments, the audio module 430 or some functional modules of the audio module 430 can be set in the processor 410.

[0079] The microphone 431 , also called a “microphone” or a “speaker”, is used to convert a sound signal into an audio stream.

[0080] The receiver 432, also called a "handset", is used to convert the audio stream into a sound signal. When the electronic device 400 receives a call or a voice message, the user can place the receiver 433 close to the ear to listen to the voice.

[0081] Speaker 433 , also known as a "speaker," converts audio streams into sound signals for output. For example, when an application on an electronic device plays an audio file in PCM format, the audio file is first decoded to generate a PCM audio stream. This PCM audio stream is then transmitted to speaker 433 , where it is converted into a sound signal for output.

[0082] The mobile communication module 440 can provide second-generation (2G) mobile communication solutions, third-generation (3G) mobile communication solutions, fourth-generation (5G) mobile communication solutions, and fifth-generation (5G) mobile communication solutions, etc., applied to the electronic device 400. The mobile communication module 440 may include at least one filter, a switch, a power amplifier, a low-noise amplifier (LNA), etc. The mobile communication module 440 can receive electromagnetic waves from the antenna 2, filter and amplify the received electromagnetic waves, and then transmit them to the modem processor for demodulation. The mobile communication module 440 can also amplify the signal modulated by the modem processor, and the amplified signal is converted into electromagnetic waves and radiated through the antenna 2.

[0083] Electronic device 400 can implement display functions through a GPU, display screen 450, and an application processor. The GPU is a microprocessor for image processing that connects display screen 450 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 410 may include one or more GPUs that execute program instructions to generate or modify display information.

[0084] Storage module 460 is used to store instructions and data. Storage module 460, such as a cache memory, can store instructions or data that have just been used or are being recycled by processor 410. If processor 410 needs to use the same instruction or data again, it can directly access it from storage module 460. This avoids duplicate accesses, reduces processor 410 latency, and thus improves system efficiency.

[0085] The processor 410 and the storage module 460 may be combined into a processing device, or more commonly, they are independent components. In specific implementations, the storage module 460 may also be integrated into the processor 410 or independent of the processor 410.

[0086] The power module 470 is used to provide power to various devices or circuits in the electronic device 400, and may include a charging management unit, a power management unit, a battery, and the like.

[0087] It should be noted that Figure 4 The connection relationship between the modules shown is only for illustrative purposes and does not limit the connection relationship between the modules of the electronic device 400. Optionally, the modules of the electronic device 400 may also adopt a combination of the multiple connection modes in the above embodiments. Figure 4 The structure shown does not constitute a specific limitation on the electronic device 400. The electronic device 400 may include Figure 4 More or fewer components than those shown, or the electronic device 400 may include Figure 4 Combinations of some of the components shown, or alternatively, the electronic device 400 may include Figure 4 Subassemblies of some of the components shown. Figure 4 The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0088] like Figure 5 As shown, Figure 5 A flowchart of an audio transmission method provided by an embodiment of the present application is shown. The method may include:

[0089] Step 501: The master device inputs the original audio stream to be played into the audio encoder through the first sub-thread for encoding to obtain a target audio stream.

[0090] In one embodiment, after the collaborative module is started, the main thread of the collaborative module is started. After the main thread is started, for different audio types, an audio encoder for each audio type can be created respectively, and a first sub-thread and a second sub-thread for each audio encoder can be created. When the application plays an audio stream of a certain audio type, the original audio stream is input into the corresponding audio encoder for encoding, and the encoded target audio data is sent to the auxiliary device.

[0091] In conjunction with the above example, for audio types such as call voice, music, ringtones, and background sound, the main thread can create an audio encoder for each audio type after startup, and create and start the first sub-thread and second sub-thread for each audio encoder. After creating the audio encoder, the main thread initializes the audio encoder and sets parameters such as the encoding format, sampling rate, and frame rate of the audio encoder. The specific method of creating and initializing the audio encoder can be set according to needs, and this embodiment does not limit this.

[0092] In this embodiment, when the collaborative module is running, if it obtains the original audio stream from the audio subsystem, it can input the original audio stream into the audio encoder for encoding through the first sub-thread. Figure 1 As shown, after the call application in mobile phone 1 is started, it receives the audio stream sent by another mobile phone and enters the playback process. The received audio stream is input into the audio subsystem, and the audio subsystem decodes the audio stream into the original audio stream. At the same time, if the call application detects that the collaborative module is started, it can send a control instruction to the collaborative module. In response to the control instruction, the main thread can call the native interface (Java Native Interface, JNI) to obtain the original audio stream played by the call application from the audio subsystem, and then send the original audio stream to the first child thread.

[0093] Correspondingly, after receiving the original audio stream, the first subthread requests an empty input buffer from the audio encoder's input buffer corresponding to the call voice, inputs the original audio stream into this input buffer, and then submits the input buffer to the audio encoder. The audio encoder encodes the original audio stream in the input buffer and inputs the resulting target audio stream into an empty output buffer in the output buffer, completing the encoding of the original audio stream.

[0094] During playback of the original audio stream, the main thread continuously obtains the original audio stream from the audio subsystem and sends the obtained original audio stream to the first child thread until the application stops playing the original audio stream. Similarly, the first child thread continuously receives the original audio stream sent by the main thread and inputs the received original audio stream into the input buffer of the audio encoder until the main thread stops sending the original audio stream.

[0095] Step 502: The master device obtains a target audio stream from an audio encoder through a second sub-thread.

[0096] In this embodiment, after being started, the second sub-thread may periodically access the audio encoder to obtain the target audio stream from the audio encoder when the audio encoder encodes the target audio stream.

[0097] In conjunction with the above example, after starting, the second sub-thread can call the audio encoder's interface at regular intervals to send a data request to the audio encoder to detect whether the target audio stream is stored in the audio encoder's output buffer. After receiving the data request, if the target audio stream is stored in the output buffer, the audio encoder submits the output buffer containing the target audio stream to the second sub-thread. The second sub-thread obtains the target audio stream from the output buffer and releases the empty output buffer, completing the acquisition of the target audio stream. The method for the second sub-thread to obtain the target audio stream from the audio encoder may include but is not limited to the above example.

[0098] Optionally, the main thread may create a first notification object after being started, and after the second sub-thread obtains the target audio stream, it may call the first notification object to send the target audio stream to the main thread.

[0099] In one embodiment, after obtaining the target audio stream, the second sub-thread first checks whether the pre-created first notification object exists, that is, whether it has been destroyed. If the first notification object exists, the second sub-thread can call the first notification object to send the target audio stream to the main thread.

[0100] On the contrary, when the first notification object is not detected, the second sub-thread can create a second notification object and call the second notification object to send the target audio stream to the main thread.

[0101] The main thread and the second sub-thread are different threads. When the notification object is created by the main thread, it will destroy the notification object after the original audio stream stops inputting. However, when the original audio stream stops inputting and the main thread destroys the notification object, the second sub-thread may be running. At this time, the second sub-thread cannot call the notification object, and the operating system will report a null pointer, causing the collaborative module to stop running.

[0102] In an embodiment of the present application, when the second sub-thread sends the target audio stream to the main thread, it first determines whether the pre-created first notification object exists. When the first notification object exists, it calls the first notification object to send the target audio stream to the main thread. When the first notification object does not exist, it creates and calls the second notification object to send the target audio stream to the main thread. This can avoid the occurrence of null pointer conditions and enable the collaborative module to run stably.

[0103] The first notification object and the second notification object can be the same notification object or different notification objects. The first notification object can be created by the main thread or the first child thread, and this embodiment does not limit this.

[0104] Optionally, when sending the target audio stream to the auxiliary device, the second sub-thread may create a notification object and call the notification object to send the target audio stream to the main thread.

[0105] In an embodiment, after being started, the second sub-thread can create a notification object and obtain the target audio stream from the audio encoder. After obtaining the target audio stream, the second sub-thread can call the created notification object to send the target audio stream to the main thread. The second sub-thread can create the notification object when being started or after obtaining the target audio stream from the audio encoder.

[0106] In the embodiment of the application, the second sub-thread directly creates a notification object and calls the notification object to send the target audio stream to the main thread, which can avoid the situation that the notification object cannot be called when the notification object is destroyed by the main thread, thereby avoiding the problem of null pointer.

[0107] Optionally, after receiving the original audio stream, the first sub-thread can determine whether the original audio stream is a suitable transmission audio stream, and if the original audio stream is a suitable transmission audio stream, the original audio stream is sent to the main thread.

[0108] The suitable transmission audio stream is the target audio stream. In combination with the above example, if the original audio stream obtained by the main thread from the audio subsystem is an AAC format audio stream, the original audio stream is a suitable transmission target audio stream. After receiving the original audio stream sent by the main thread, if the first sub-thread determines that the original audio stream is an ACC format, the first sub-thread directly calls the notification object to send the original audio stream as the target audio stream to the main thread. The main thread can input the target audio stream into the buffer queue and send the target audio stream in the buffer queue to the auxiliary device. The method of the first sub-thread for calling the notification object to send the audio stream to the main thread is the same as that of the second sub-thread, which is not described herein.

[0109] In actual application, some application programs can directly encode the original audio stream into a suitable transmission target audio stream when playing the audio stream. When the first sub-thread receives the suitable transmission audio stream, the first sub-thread can directly send the suitable transmission audio stream to the main thread, so as to avoid re-encoding the audio stream by the encoder.

[0110] In step 503, the main device sends the target audio stream to the auxiliary device.

[0111] In the embodiment, after obtaining the target audio stream by encoding, the main device can send the target audio stream to the auxiliary device, so that the auxiliary device decodes the target audio stream to obtain and play the original audio stream. In combination with the above example, after receiving the target audio stream sent by the second sub-thread, the main thread can input the target audio stream into the buffer queue, then pack the target audio stream stored in the buffer queue into an RTP format data packet, and send the data packet to the auxiliary device.

[0112] Correspondingly, the auxiliary device parses the target audio stream from the data packet after receiving the data packet, and inputs the target audio stream into the audio decoder for decryption to obtain the original audio stream, and plays the original audio stream.

[0113] In combination with the above example, when the mobile phone 1 and the tablet computer 2 perform voice communication, the tablet computer 2 can input the target audio stream into the audio decoder for decoding to obtain the original audio stream after receiving the target audio stream sent by the mobile phone 1, and input the original audio stream into the audio module, and control the loudspeaker to output the original audio stream as a sound signal by the audio module.

[0114] Optionally, the first sub-thread and / or the second sub-thread are started when the main thread obtains the original audio stream.

[0115] In an embodiment, the main thread can start the first sub-thread and the second sub-thread when the original audio stream is obtained, so as to avoid the first sub-thread and the second sub-thread being started before the original audio stream is obtained, and to avoid thread idling. In combination with the above example, the main thread can only create the audio encoder of each audio type after being started. The main thread creates and starts the corresponding first sub-thread and second sub-thread after obtaining the original audio stream of a certain audio type from the audio subsystem.

[0116] Before the original audio stream is obtained, the first sub-thread and the second sub-thread do not need to act. If the first sub-thread and the second sub-thread are started before the original audio stream is obtained, the first sub-thread and the second sub-thread will idle, and the idling sub-thread will not only occupy the processor resources of the main device, but also increase the power consumption of the processor.

[0117] In the embodiment of the application, the first sub-thread and the second sub-thread are started when the original audio stream is obtained, which can avoid thread idling, avoid thread occupying processor resources, and reduce the load and power consumption of the processor.

[0118] Optionally, the method can further include: if the main device does not obtain the original audio stream within a preset time length, stopping the started target thread, and the target thread includes the first sub-thread and / or the second sub-thread.

[0119] Exemplarily, after being started, the main thread can set one global thread stop flag for each audio type of audio encoder. The main thread can set the thread stop flag to a first state value when obtaining the corresponding original audio stream from the audio subsystem. Conversely, if the main thread does not obtain the corresponding original audio stream from the audio subsystem within a preset time length, the thread stop flag is set to a second state value, and the second state value is different from the first state value. The specific value of the preset time length can be set according to requirements, and the embodiment does not limit this.

[0120] Taking the second child thread as an example, the second child thread continues to run when the thread stop flag is in the first state value. When the thread stop flag is in the second state value, the second child thread can exit the loop and automatically terminate the second child thread. Alternatively, when the thread stop flag is in the second state value, the second child thread can call the wait method to enter the wait state, causing the second child thread to stop running. Specific methods for stopping a child thread may include but are not limited to the above examples, and this embodiment does not limit this.

[0121] In an embodiment of the present application, when the main device does not receive the original audio stream within a preset time period, it stops the started sub-thread, which can avoid thread idling, thereby preventing the idle thread from occupying processor resources and reducing the processor load and power consumption.

[0122] In the above example, if the main thread creates a first and second sub-thread corresponding to each audio encoder at startup, a larger number of first and second sub-threads will be created when there are a large number of audio encoders. However, the main thread generally encodes and sends an audio stream of only one audio type, and only uses the first and second sub-threads of one audio encoder, while the other sub-threads will idle. By stopping the sub-threads when the main thread does not receive the corresponding original audio stream, it can prevent unused sub-threads from idling, thereby freeing up processor resources and reducing processor load and power consumption.

[0123] Figure 6 The flowchart of an audio transmission method provided by an embodiment of the present application is shown. The method may include steps 60 to 610.

[0124] Combine Figure 1 As shown, when the call application in mobile phone 1 receives an audio stream sent by another mobile phone, it executes step 60 to input the audio stream into the audio subsystem. The audio subsystem decodes the input audio stream to obtain the original audio stream.

[0125] After determining the input of the original audio stream, the main thread executes step 61 to obtain the original audio stream from the audio subsystem.

[0126] After the main thread obtains the original audio stream, it executes step 62 to start the second child thread and executes step 63 to start the first child thread. At the same time, the main thread can execute step 64 to create the first notification object. The execution order of steps 62, 63 and 64 can be as follows: Figure 6 As shown, it can also be adjusted according to needs.

[0127] After starting to create the first notification object, the main thread may execute step 65 to send the original audio stream to the first child thread.

[0128] After receiving the original audio stream, the first sub-thread executes step 66 to request an empty input buffer from the input buffer of the audio encoder, input the original audio stream into the input buffer, and submit the input buffer to the audio encoder to input the original audio stream into the audio encoder.

[0129] After receiving the original audio stream, the audio encoder executes step 67 to obtain the original audio stream from the input buffer for encoding, and input the encoded target audio stream into the output buffer.

[0130] After being started, the second sub-thread may periodically send data requests to the audio encoder. After receiving the data requests, if the output buffer of the audio encoder contains the target audio stream, the audio encoder executes step 68 to submit the output buffer filled with the target audio stream to the second sub-thread. The second sub-thread may obtain the target audio stream from the output buffer and release the empty output buffer.

[0131] After obtaining the target audio stream, the second child thread executes step 69 to check whether the first notification object exists. If the first notification object is detected to exist, step 610 is executed to call the first notification object to send the target audio stream to the main thread. Conversely, if the first notification object is detected to not exist, step 610 is executed to create a second notification object and call the second notification object to send the target audio stream to the main thread.

[0132] After receiving the target audio stream sent by the second sub-thread, the main thread stores the target audio stream in the cache queue, and then executes step 611 to package the target audio stream in the cache queue and send it to the auxiliary device.

[0133] Correspondingly, after receiving the target audio stream, the auxiliary device can decode the target audio stream to obtain the original audio stream and play the original audio stream.

[0134] Figure 7 A schematic diagram of the transmission principle of an audio stream provided in an embodiment of the present application is shown.

[0135] Combine Figure 5 and Figure 6 After obtaining the original audio stream, the main thread can create and start the first sub-thread and the second sub-thread. When sending the audio stream to the auxiliary device, the main thread obtains the original audio stream from the audio subsystem and sends the original audio stream to the first sub-thread. The first sub-thread inputs the original audio stream into the audio encoder for encoding.

[0136] Meanwhile, the second sub-thread polls the audio encoder to detect whether the target audio stream is stored in an output buffer of the audio encoder. When the target audio stream is stored in the output buffer, the target audio stream is obtained from the output buffer and sent to the main thread.

[0137] The main thread stores the target audio stream in a cache queue and then sends the target audio stream in the cache queue to the auxiliary device.

[0138] Since the first sub-thread is only responsible for inputting the original audio stream into the audio encoder and does not need to obtain the target audio stream from the audio encoder, when the audio encoder encodes slowly, the first sub-thread will not be blocked and can input the original audio stream into the audio encoder in time for encoding. The original audio stream will not be delayed to be input into the audio encoder for encoding, and audio frames in the original audio stream will not be lost. Therefore, the target audio stream obtained by encoding will not be delayed to be sent to the main thread. Even if the audio encoder encodes slowly, the main thread can still obtain continuous target audio streams, so that the auxiliary device can play audio smoothly.

[0139] In summary, in the embodiment, the main device inputs the original audio stream into the audio encoder for encoding through the first sub-thread and obtains the target audio stream encoded by the audio encoder through the second sub-thread and sends the target audio stream to the auxiliary device, so that the auxiliary device decodes the original audio stream according to the target audio stream and plays the original audio stream. Since the first sub-thread does not need to obtain the target audio stream from the audio encoder, when the audio encoder encodes slowly, the first sub-thread will not be blocked and can input the original audio stream into the audio encoder in time for encoding. The original audio stream will not be delayed to be input into the audio encoder for encoding, and audio frames in the original audio stream will not be lost. Even if the audio encoder encodes slowly, the main thread can still obtain continuous target audio streams, so that the auxiliary device can play audio smoothly.

[0140] The input buffer of the audio encoder can store a certain number of audio frames, and when the audio encoder encodes slowly, the original audio stream that is not encoded in time by the first sub-thread can be stored in the input buffer. Therefore, even if the audio encoder encodes slowly, the audio encoder can encode all the input audio frames, so that continuous target audio streams can be obtained, and the auxiliary device can play audio smoothly when the target audio streams are smooth.

[0141] Figure 8 A structure schematic diagram of an audio transmission device provided by an embodiment of the present application is shown, and the audio transmission device 8 includes:

[0142] The encoding module 81 is configured to input, by the main device through the first sub-thread, an original audio stream to be played into an audio encoder for encoding to obtain a target audio stream.

[0143] An acquisition module 82 is configured to enable the master device to acquire the target audio stream from the audio encoder through a second sub-thread, where the second sub-thread is different from the first sub-thread;

[0144] The sending module 83 is configured for the master device to send the target audio stream to the slave device.

[0145] Optionally, the sending module 83 is specifically configured for the master device to send the target audio stream to the slave device through a main thread, and the first sub-thread obtains the original audio stream from the main thread.

[0146] Optionally, the acquisition module 82 is specifically configured to cause the second sub-thread to call a first notification object to send the target audio stream to the main thread.

[0147] Optionally, the acquisition module 82 is further configured to create a second notification object when the second sub-thread does not detect the first notification object, and call the second notification object to send the target audio stream to the main thread.

[0148] Optionally, the acquisition module 82 is specifically configured for the second sub-thread to create and call a notification object to send the target audio stream to the main thread.

[0149] Optionally, the first sub-thread and / or the second sub-thread is started when the main thread obtains the original audio stream.

[0150] Optionally, the apparatus 8 further comprises: a stopping module, configured to stop a started target thread if the master device fails to obtain the original audio stream within a preset time period, the target thread including the first sub-thread and / or the second sub-thread.

[0151] Figure 9 A schematic diagram of the structure of another electronic device provided in an embodiment of the present application is shown. The electronic device 9 is for example Figure 4 The electronic device 400 shown includes a processor 91, a memory 92, a communication interface 93, and a bus 94. The memory 92 is used to store instructions, and the processor 91 is used to execute the instructions stored in the memory 92. The processor 91, the memory 92, and the communication interface 93 are connected to each other via the bus 94.

[0152] The present application also provides a chip system, which is applied to an electronic device. The chip system includes one or more processors, and the processor is used to call computer instructions to enable the electronic device to execute the method described above.

[0153] The present application also provides a computer program product, which includes: computer program code, which, when executed by a copy device of a smart card, implements the method described in any method embodiment of the present application.

[0154] The computer program product may also be a code solidified in a chip. This application does not limit the specific form of the computer program product.

[0155] The present application also provides a readable storage medium on which a computer program is stored, which, when executed by a smart card copying device, implements the method described in any method embodiment of the present application. The computer program can be a high-level language program or an executable target program.

[0156] The readable storage medium may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0157] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and equipment and the technical effects produced can refer to the corresponding processes and technical effects in the aforementioned method embodiments, and will not be repeated here.

[0158] In several embodiments provided in the present application, the disclosed system, device and method can be implemented in other manners. For example, some features of the above-described method embodiments can be ignored or not performed. The above-described device embodiments are only illustrative, and the division of units is only a logical function division, and there can be another division manner in actual implementation. Multiple units or components can be combined or integrated into another system, or some features can be ignored or not performed. In addition, the coupling or connection between the units, or the coupling or connection between the components can be direct or indirect, and can be electrical, mechanical or in other forms.

[0159] It should be understood that, in various embodiments of the present application, the size of the sequence of each process does not mean the execution order, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0160] In summary, the above description is only a preferred embodiment of the technical scheme of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. An audio transmission method, characterized in that: include: The main device inputs the original audio stream to be played into the audio encoder through the first sub-thread; The master device encodes the original audio stream through the audio encoder to obtain a target audio stream; The master device obtains the target audio stream from the audio encoder through a second sub-thread, where the second sub-thread is different from the first sub-thread; The master device sends the target audio stream to the slave device through a master thread, wherein the first sub-thread obtains the original audio stream from the master thread.

2. The method according to claim 1, wherein The second sub-thread calls the first notification object to send the target audio stream to the main thread.

3. The method according to claim 2, wherein The method further comprises: When the second sub-thread does not detect the first notification object, a second notification object is created, and the second notification object is called to send the target audio stream to the main thread.

4. The method according to claim 1, wherein The second sub-thread creates and calls a notification object to send the target audio stream to the main thread.

5. The method according to any one of claims 1 to 4, wherein The first sub-thread and / or the second sub-thread are started when the main thread obtains the original audio stream.

6. The method according to any one of claims 1 to 5, wherein The method further comprises: If the master device fails to obtain the original audio stream within a preset time period, the master device stops the started target thread, where the target thread includes the first sub-thread and / or the second sub-thread.

7. An audio transmission device, characterized in that: include: The encoding module is used for the main device to input the original audio stream to be played into the audio encoder through the first sub-thread for encoding to obtain the target audio stream; An acquisition module, configured for the master device to acquire the target audio stream from the audio encoder through a second sub-thread, where the second sub-thread is different from the first sub-thread; A sending module is configured to enable the master device to send the target audio stream to the slave device through a main thread, wherein the first sub-thread obtains the original audio stream from the main thread.

8. The device according to claim 7, wherein The acquisition module is specifically configured for the second sub-thread to call the first notification object to send the target audio stream to the main thread.

9. The device according to claim 8, wherein The acquisition module is further configured to create a second notification object when the second sub-thread fails to detect the first notification object, and call the second notification object to send the target audio stream to the main thread.

10. The device according to claim 7, wherein The acquisition module is specifically used for the second sub-thread to create and call a notification object to send the target audio stream to the main thread.

11. The device according to any one of claims 7 to 10, characterized in that The first sub-thread and / or the second sub-thread are started when the main thread obtains the original audio stream.

12. The device according to any one of claims 7 to 11, characterized in that The device further comprises: A stopping module is configured to stop a started target thread if the master device fails to obtain the original audio stream within a preset time period, wherein the target thread includes the first sub-thread and / or the second sub-thread.

13. An electronic device, characterized in that: include: One or more processors and one or more memories; the one or more processors are coupled to the one or more memories, the one or more memories are used to store computer program code, the computer program code includes computer instructions, when the one or more processors execute the computer instructions, the electronic device performs the method according to any one of claims 1 to 6.

14. A readable storage medium, characterized in that The readable storage medium stores a computer program product, wherein the computer program product includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 6.

15. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions to enable the electronic device to execute the method as described in any one of claims 1-6.

16. A computer program product, characterized in that The method comprises computer instructions, which, when executed on an electronic device, enable the electronic device to execute the method according to any one of claims 1 to 6.

Images