An audio synchronous acquisition method, device and terminal equipment

Abstract

The embodiment of the present application provides a kind of audio synchronous acquisition method, device and terminal equipment, it is related to speech processing technical field, to realize synchronous acquisition to the audio played between different devices.The method comprises: collecting the first audio data played by near-end device and the second audio data played by remote device;First audio data includes first reference signal and the first audio data to be played of near-end device, and second audio data includes second reference signal and the second audio data to be played of remote device;The frequency of first reference signal is different from second reference signal;Determine the first time difference of the collection time of first reference signal and the collection time of second reference signal;According to the first time difference, adjust the time interval between first reference signal and first audio data to be played, so that the collection time of first audio data to be played and the collection time of second audio data to be played are synchronized.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, and in particular to an audio synchronous acquisition method, apparatus and terminal equipment. Background Technology

[0002] Remote conferencing and voice calls have become an important part of collaborative work, and many devices now support remote calling functionality. However, implementing remote calling requires devices with corresponding microphones. Generally, the voice captured by the microphone is not sent directly to the recipient's device. Instead, it undergoes a series of processing steps (generally called preprocessing), such as noise reduction, dereverberation, and echo cancellation, before being sent to the recipient's device. This ensures that the recipient hears the voice without interference from other noise sources. In other words, the quality of the preprocessing directly affects the quality of the voice heard; poor preprocessing will directly impact the user experience. Therefore, necessary subjective and objective testing must be conducted during the development of preprocessing algorithms.

[0003] The subjective and objective tests may include dual-talk performance testing. During this test, it is crucial to strictly control the timing of the far-end signal playback and the near-end speaker's speech; otherwise, the test results will be biased. However, in actual testing, the far-end signal is usually played through the device under test (DUT), while the near-end signal is played by another device (usually a human mouth) in the same room as the DUT. The DUT is typically an Android device. The system is either a Windows operating system or a user-controlled audio system, while the artificial mouth can be controlled via a personal computer (PC). In practice, the time from when the user presses the play button to when the actual audio begins playing is not fixed, but rather affected by the current system load of the device under test. Therefore, the timing of the actual audio playback cannot be directly controlled.

[0004] In other words, existing technologies have the problem that audio played between different devices cannot be synchronously captured, which leads to deviations in the results of dual-talk performance tests. Summary of the Invention

[0005] This application provides an audio synchronization acquisition method, apparatus, and terminal device for synchronous acquisition of audio played between different devices.

[0006] In a first aspect, embodiments of this application provide an audio synchronization acquisition method, comprising: acquiring first audio data played by a near-end device and second audio data played by a far-end device; the first audio data includes a first reference signal and first audio data to be played by the near-end device, and the second audio data includes a second reference signal and second audio data to be played by the far-end device; the first reference signal and the second reference signal have different frequencies; when playback is started, the time interval between the first reference signal and the first audio data to be played, and the time interval between the second reference signal and the second audio data to be played are the same; determining a first time difference between the acquisition time of the first reference signal and the acquisition time of the second reference signal; adjusting the time interval between the first reference signal and the first audio data to be played according to the first time difference, so that the acquisition time of the first audio data to be played is synchronized with the acquisition time of the second audio data to be played.

[0007] Based on the above scheme, by acquiring a first reference signal and a second reference signal, a first time difference can be determined between the first audio data to be played on the near-end device and the second audio data to be played on the far-end device. Based on this first time difference, the time interval between the first reference signal and the first audio data to be played can be adjusted, thereby achieving synchronous acquisition of audio played on different devices. This method can automatically calculate the delay between audio data played on different devices without manual intervention.

[0008] In one possible implementation, adjusting the time interval between the first reference signal and the first audio data to be played according to the first time difference includes: if the acquisition time of the first reference signal is earlier than the acquisition time of the second reference signal, then inserting blank audio data with a duration equal to the first time difference between the first reference signal and the first audio data to be played.

[0009] In one possible implementation, adjusting the time interval between the first reference signal and the first audio data to be played according to the first time difference includes: if the acquisition time of the second reference signal is earlier than the acquisition time of the first reference signal, then deleting data whose duration is the first time difference between the first reference signal and the first audio data to be played.

[0010] In one possible implementation, the first audio data further includes N third reference signals, and the second audio data further includes N fourth reference signals, where N is a positive integer; the N third reference signals are located between the first reference signal and the first audio data to be played, and the N fourth reference signals are located between the second reference signal and the second audio data to be played; the N third reference signals correspond one-to-one with the N fourth reference signals, the difference between the first time interval and the second time interval is less than or equal to the duration of the third reference signal, the duration of the third reference signal is the same as the duration of the fourth reference signal, the first time interval is the time interval between the third reference signal and the first reference signal, and the second time interval is the time interval between the fourth reference signal corresponding to the third reference signal and the second reference signal; the method further includes: determining a second time difference between the acquisition time of each of the N third reference signals and the acquisition time of the corresponding fourth reference signal; adjusting the time interval between the first reference signal and the first audio data to be played according to the first time difference, including: adjusting the time interval between the first reference signal and the first audio data to be played according to the first time difference and the N second time differences.

[0011] In one possible implementation, N is determined based on the duration of the third reference signal, or N is determined based on the duration of the fourth reference signal.

[0012] In one possible implementation, adjusting the time interval between the first reference signal and the first audio data to be played based on the first time difference and N second time differences includes: determining M third time differences from the N second time differences; the third time difference being greater than the sum of the durations of the first time difference and the third reference signal, where M is an integer greater than or equal to 0 and less than or equal to N; determining the time delay deviation between the near-end device and the far-end device based on the number of third time differences; and adjusting the time interval between the Nth third reference signal and the first audio data to be played based on the time delay deviation and the first time difference.

[0013] In one possible implementation, the time interval between any two adjacent third reference signals among the N third reference signals is a reference interval value; the time interval between any two adjacent fourth reference signals among the N fourth reference signals is different, and the time interval between any two adjacent fourth reference signals is determined based on the duration of the fourth reference signal and the reference interval value; or, the time interval between any two adjacent fourth reference signals among the N fourth reference signals is a reference interval value; the time interval between any two adjacent third reference signals among the N third reference signals is different, and the time interval between any two adjacent third reference signals is determined based on the duration of the third reference signal and the reference interval value.

[0014] Secondly, embodiments of this application provide an audio synchronization acquisition device, comprising: an acquisition unit, configured to acquire first audio data played by a near-end device and second audio data played by a far-end device, wherein the first audio data includes a first reference signal and first audio data to be played by the near-end device; the second audio data includes a second reference signal and second audio data to be played by the far-end device; the first reference signal and the second reference signal have different frequencies; when playback is started, the time interval between the first reference signal and the first audio data to be played, and the time interval between the second reference signal and the second audio data to be played are the same;

[0015] The processing unit is configured to determine a first time difference between the acquisition time of the first reference signal and the acquisition time of the second reference signal; and adjust the time interval between the first reference signal and the first audio data to be played according to the first time difference, so that the acquisition time of the first audio data to be played is synchronized with the acquisition time of the second audio data to be played.

[0016] In one possible implementation, when the processing unit adjusts the time interval between the first reference signal and the first audio data to be played according to the first time difference, it is further configured to: if the acquisition time of the first reference signal is earlier than the acquisition time of the second reference signal, then insert blank audio data with a duration equal to the first time difference between the first reference signal and the first audio data to be played.

[0017] In one possible implementation, when the processing unit adjusts the time interval between the first reference signal and the first audio data to be played according to the first time difference, it is further configured to: if the acquisition time of the second reference signal is earlier than the acquisition time of the first reference signal, then delete the data between the first reference signal and the first audio data to be played with a duration equal to the first time difference.

[0018] In one possible implementation, the first audio data further includes N third reference signals, and the second audio data further includes N fourth reference signals, where N is a positive integer; the N third reference signals are located between the first reference signal and the first audio data to be played, and the N fourth reference signals are located between the second reference signal and the second audio data to be played; the N third reference signals correspond one-to-one with the N fourth reference signals, the difference between the first time interval and the second time interval is less than or equal to the duration of the third reference signal, the duration of the third reference signal is the same as the duration of the fourth reference signal, the first time interval is the time interval between the third reference signal and the first reference signal, and the second time interval is the time interval between the fourth reference signal corresponding to the third reference signal and the second reference signal. The processing unit is further configured to: determine a second time difference between the acquisition time of each of the N third reference signals and the acquisition time of the corresponding fourth reference signal; adjust the time interval between the first reference signal and the first audio data to be played according to the first time difference, including: adjusting the time interval between the first reference signal and the first audio data to be played according to the first time difference and the N second time differences.

[0019] In one possible implementation, N is determined based on the duration of the third reference signal, or N is determined based on the duration of the fourth reference signal.

[0020] In one possible implementation, when the processing unit adjusts the time interval between the first reference signal and the first audio data to be played based on the first time difference and N second time differences, it is further configured to: determine M third time differences from the N second time differences; the third time differences are greater than the sum of the durations of the first time difference and the third reference signal, and M is an integer greater than or equal to 0 and less than or equal to N; determine the time delay deviation between the near-end device and the far-end device based on the number of third time differences; and adjust the time interval between the Nth third reference signal and the first audio data to be played based on the time delay deviation and the first time difference.

[0021] In one possible implementation, the time interval between any two adjacent third reference signals among the N third reference signals is a reference interval value; the time interval between any two adjacent fourth reference signals among the N fourth reference signals is different, and the time interval between any two adjacent fourth reference signals is determined based on the duration of the fourth reference signal and the reference interval value; or, the time interval between any two adjacent fourth reference signals among the N fourth reference signals is a reference interval value; the time interval between any two adjacent third reference signals among the N third reference signals is different, and the time interval between any two adjacent third reference signals is determined based on the duration of the third reference signal and the reference interval value.

[0022] Thirdly, embodiments of this application provide a terminal device, including:

[0023] Memory, used to store computer instructions;

[0024] A processor, connected to the memory, is configured to execute computer instructions in the memory, and, in executing the computer instructions, implement the method as described in any one of the first aspects.

[0025] Fourthly, embodiments of this application provide a computer-readable storage medium, comprising:

[0026] The computer-readable storage medium stores computer instructions that, when executed on a computer, cause the computer to perform the method as described in any one of the first aspects.

[0027] Fifthly, this application provides a computer program product, including a computer program;

[0028] When the computer program is executed by a processor, it implements the method as described in any one of the first aspects above.

[0029] For the various aspects of the second to fifth aspects mentioned above, and the technical effects that each aspect may achieve, please refer to the above description of the technical effects that can be achieved for the first aspect or the various possible solutions in the first aspect, which will not be repeated here. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application.

[0031] Figure 1 This is a schematic diagram of the audio alignment effect provided in an embodiment of this application;

[0032] Figure 2 A schematic diagram of the system structure for the audio synchronization acquisition method provided in the embodiments of this application;

[0033] Figure 3 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application;

[0034] Figure 4 A software structure block diagram of a terminal device provided in an embodiment of this application;

[0035] Figure 5 One of the exemplary flowcharts of the audio synchronization acquisition method provided in the embodiments of this application;

[0036] Figure 6A One of the schematic diagrams showing the acquisition time of the first reference signal and the acquisition time of the second reference signal provided in the embodiments of this application;

[0037] Figure 6B One of the schematic diagrams showing the acquisition time of the first reference signal and the acquisition time of the second reference signal provided in the embodiments of this application;

[0038] Figure 7A This is one of the schematic diagrams of the first reference signal and the second reference signal before synchronization provided in the embodiments of this application;

[0039] Figure 7B This is one of the schematic diagrams of the synchronized first reference signal and second reference signal provided in the embodiments of this application;

[0040] Figure 8A This is one of the schematic diagrams of the first reference signal and the second reference signal before synchronization provided in the embodiments of this application;

[0041] Figure 8B This is one of the schematic diagrams of the synchronized first reference signal and second reference signal provided in the embodiments of this application;

[0042] Figure 9 This is one of the schematic diagrams of the third and fourth reference signals provided in the embodiments of this application;

[0043] Figure 10 This is one of the schematic diagrams of the third and fourth reference signals provided in the embodiments of this application;

[0044] Figure 11 This is one of the schematic diagrams of the third and fourth reference signals provided in the embodiments of this application;

[0045] Figure 12 This is a schematic diagram of the near-end reference signal and the far-end reference signal provided in the embodiments of this application;

[0046] Figure 13AOne of the exemplary flowcharts of the audio synchronization acquisition method provided in the embodiments of this application;

[0047] Figure 13B One of the exemplary flowcharts of the audio synchronization acquisition method provided in the embodiments of this application;

[0048] Figure 13C One of the exemplary flowcharts of the audio synchronization acquisition method provided in the embodiments of this application;

[0049] Figure 13D One of the exemplary flowcharts of the audio synchronization acquisition method provided in the embodiments of this application;

[0050] Figure 13E One of the exemplary flowcharts of the audio synchronization acquisition method provided in the embodiments of this application;

[0051] Figure 14 This is a schematic diagram of an audio synchronization acquisition device provided in an embodiment of this application;

[0052] Figure 15 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0054] The terms "first" and "second" in the embodiments of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising" and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. The term "multiple" in this application can mean at least two, for example, two, three, or more; the embodiments of this application do not impose any limitations.

[0055] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0056] Remote conferencing and voice calls have become an important part of collaborative work, and many devices now support remote calling. However, if the audio captured by the device is directly sent to the other party's device, it often includes echoes, noise, and other interference, making it difficult to distinguish the speaker's true voice. Echoes occur when sound from the device's speaker is received by the device's microphone and then sent to the other party, causing them to hear their own voice. Therefore, captured audio is typically not sent directly to the other party's device. Instead, it undergoes a series of preprocessing steps, such as noise reduction, dereverberation, and echo cancellation, before being sent. Echo cancellation is further divided into one-way and two-way. One-way means only the distant speaker is speaking. In this case, the distant speaker must not hear their own echo in the receiver. Two-way means both the distant speaker and the near speaker using the device are speaking simultaneously. In this case, the distant speaker must hear the near speaker's voice while not hearing their own echo.

[0057] When determining the echo cancellation effect of a device under test (DUT) through dual-talk performance testing, it is crucial to strictly control the speaking timing of the far-end speaker and the near-end speaker; otherwise, the test results may be biased. (See also...) Figure 1 This is a schematic diagram illustrating the audio alignment effect provided in an embodiment of this application. Figure 1 As shown in Figures A and B, these represent two scenarios where the audio of the near-end signal is aligned with the audio of the far-end signal. Figure A indicates that the far-end signal begins playing after the near-end signal finishes playing, while Figure B indicates that the near-end signal begins playing after the far-end signal finishes playing. In actual testing, the far-end signal is usually played by the device under test (DUT), while the near-end signal is usually played by another device (usually an artificial mouth) in the same room as the DUT.

[0058] To achieve alignment between near-end and far-end signals, it is first necessary to synchronously acquire the audio played by the two different devices used for testing. The device under test is typically... In Windows systems, the time between when the user presses the play button and when the actual sound begins to play is affected by the current system load and is not fixed. Therefore, it is impossible to directly control when the device under test plays the actual audio. In other words, there is currently a technical problem that audio data played from different devices cannot be collected synchronously.

[0059] In view of this, this application provides an audio synchronization acquisition method. This method involves adding reference signals of different frequencies before the audio data to be played by the near-end device and the audio data to be played by the far-end device, respectively, thus forming the audio data played by the near-end device and the audio data played by the far-end device. When acquiring the audio data played by the near-end device and the audio data played by the far-end device, the terminal device can determine the device to which the audio data belongs by using the frequency of the acquired audio data, thereby determining the time difference between the time when the audio data to be played by the near-end device is acquired and the time when the audio data to be played by the far-end device is acquired. Then, the terminal device can adjust the time interval between the reference signal and the audio data to be played in the audio data played by the near-end device according to this time difference, so that the acquisition time of the audio data to be played by the near-end device is synchronized with the acquisition time of the audio data to be played by the far-end device.

[0060] See Figure 2 This application describes a system for the audio synchronization acquisition method provided in this embodiment. System 200 may include a remote device 201, a recording device 202, a terminal device 203, a playback control device 204, and a near-end device 205. The remote device 201 is used to play remote audio data and may be a terminal device including a multi-microphone array and dual-channel speakers, such as a smartphone, computer, or tablet computer; this application does not limit this. The recording device 202 may be connected to the terminal device 203 and is used to acquire the remote audio data played by the remote device 201 and the audio data played by the near-end device 205. The recording device 202 may be an external microphone. The terminal device 203 may include an audio acquisition and calculation module 2031 and a command control module 2032. The audio acquisition and calculation module 2031 can analyze and calculate the audio data acquired by the recording device 202. The command control module 2032 can respond to a user's operation to play near-end audio data by sending an instruction to the playback control device to instruct the near-end device 205 to play the near-end audio data. The playback control device 204 receives instructions from the terminal device 203 to control the near-end device 205 to play near-end audio data. The near-end device 205 is used to play near-end audio data, and may be a device such as an artificial mouth.

[0061] Figure 3 A schematic diagram of the structure of a terminal device 300 is shown. Figure 2 The terminal device 203 shown can be Figure 3 The terminal device 300. It should be understood that... Figure 3 The terminal device 300 shown is merely an example, and the terminal device 300 may have more than Figure 3 The more or fewer components shown can be combined into two or more components, or they can have different component configurations. Figure 3The various components shown can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0062] Figure 3 The diagram illustrates a hardware configuration block diagram of a terminal device 300 according to an exemplary embodiment. Figure 3 As shown, the terminal device 300 includes components such as a radio frequency (RF) circuit 310, a memory 320, a display unit 330, a camera 340, a sensor 350, an audio circuit 360, a wireless Fidelity (Wi-Fi) module 370, a processor 380, a Bluetooth module 381, and a power supply 390.

[0063] The RF circuit 310 can be used to receive and transmit signals during information transmission or calls. It can receive downlink data from the base station and hand it over to the processor 380 for processing; it can also send uplink data to the base station. Typically, the RF circuit includes, but is not limited to, devices such as antennas, at least one amplifier, transceivers, couplers, low-noise amplifiers, and duplexers.

[0064] The memory 320 can be used to store software programs and data. The processor 380 executes various functions of the terminal device 300 and performs data processing by running the software programs or data stored in the memory 320. The memory 320 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. The memory 320 stores an operating system that enables the terminal device 300 to run. In this application, the memory 320 may store the operating system and various application programs, and may also store program code that executes the audio synchronization acquisition method of the embodiments of this application.

[0065] The display unit 330 can be used to receive input digital or character information and generate signal inputs related to user settings and function control of the terminal device 300. Specifically, the display unit 330 may include a touch screen 331 disposed on the front of the terminal device 300, which can collect touch operations on or near the user, such as clicking a button.

[0066] The display unit 330 can also be used to display information input by the user or information provided to the user, as well as various menus of the terminal device 300, forming a graphical user interface (GUI). Specifically, the display unit 330 may include a display screen 332 disposed on the front of the terminal device 300. The display screen 332 may be configured as a liquid crystal display, a light-emitting diode, or the like.

[0067] The touchscreen 331 can be placed over the display screen 332, or the touchscreen 331 and the display screen 332 can be integrated to realize the input and output functions of the terminal device 300. After integration, it can be referred to as a touch display screen. In this application, the display unit 330 can display the application program and the corresponding operation steps.

[0068] Camera 340 can be used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to processor 380 to be converted into a digital image signal.

[0069] The terminal device 300 may also include at least one sensor 350, such as an accelerometer 351, a proximity sensor 352, a fingerprint sensor 353, and a temperature sensor 354. The terminal device 300 may also be equipped with other sensors such as a gyroscope, barometer, hygrometer, thermometer, infrared sensor, light sensor, and motion sensor.

[0070] Audio circuitry 360, speaker 361, and microphone 362 provide an audio interface between the user and terminal device 300. Audio circuitry 360 converts received audio data into electrical signals, which are then transmitted to speaker 361, where they are converted into sound signals for output. Terminal device 300 may also be equipped with volume buttons for adjusting the volume of the sound signal, and these buttons can be combined with other buttons to adjust the enclosed area. On the other hand, microphone 362 converts collected sound signals into electrical signals, which are received by audio circuitry 360, converted into audio data, and then output to RF circuitry 310 for transmission to, for example, another terminal device, or to memory 320 for further processing.

[0071] Wi-Fi is a short-range wireless transmission technology. Terminal device 300 can use Wi-Fi module 370 to help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access.

[0072] The processor 380 is the control center of the terminal device 300. It connects various parts of the terminal device via various interfaces and lines, and performs various functions and processes data by running or executing software programs stored in the memory 320 and calling data stored in the memory 320. In some embodiments, the processor 380 may include one or more processing units; the processor 380 may also integrate an application processor and a baseband processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the baseband processor mainly handles wireless communication. It is understood that the baseband processor may not be integrated into the processor 380. In this application, the processor 380 can run the operating system, applications, user interface display and touch response, as well as the audio synchronization acquisition method of this embodiment. Furthermore, the processor 380 is coupled to the display unit 330.

[0073] Bluetooth module 381 is used to exchange information with other Bluetooth devices that also have Bluetooth modules via the Bluetooth protocol. For example, terminal device 300 can establish a Bluetooth connection with wearable terminal devices (such as smartwatches) that also have Bluetooth modules through Bluetooth module 381, thereby exchanging data.

[0074] The terminal device 300 also includes a power supply 390 (such as a battery) that supplies power to various components. The power supply can be logically connected to the processor 380 through a power management system, thereby enabling the management of charging, discharging, and power consumption. The terminal device 300 may also be equipped with a power button for powering on and off the terminal device, as well as screen locking.

[0075] See Figure 4 This is a software structure block diagram of a terminal device 300 provided in an embodiment of this application. The layered architecture divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the software can be... The system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0076] The application layer can include a series of application packages. For example... Figure 4 As shown, the application package can include applications such as phone, MMS, WiFi, WeChat, messaging, alarm clock, gallery, calendar, and WLAN.

[0077] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example... Figure 4As shown, the application framework layer can include a window manager, content providers, a view system, a phone manager, a resource manager, a notification manager, etc. The window manager manages window programs. It can obtain the screen size, determine if a status bar is present, lock the screen, and capture the screen. The content provider stores and retrieves data, making this data accessible to the application. Data can include video, images, audio, made and received calls, browsing history and bookmarks, phone books, SMS messages, etc. The view system includes visual controls, such as controls for displaying text and controls for displaying images. The view system can be used to build the application. The display interface can consist of one or more views. For example, a display interface including an SMS notification icon can include a view for displaying text and a view for displaying images. The phone manager provides communication functions for the terminal device 300, such as managing call status (including connection, hang-up, etc.). The resource manager provides various resources for the application, such as localized strings, icons, images, layout files, video files, etc. The notification manager allows applications to display notification information (such as SMS message content) in the status bar. It can be used to convey informational messages and can disappear automatically after a short pause without user interaction. For example, the notification manager can be used to notify users of download completion or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating the device, and flashing indicator lights.

[0078] The Android Runtime comprises the core libraries and the virtual machine. The Android runtime is responsible for scheduling and managing the Android system. The core libraries consist of two parts: one part contains the functionalities that Java calls, and the other part contains the core Android libraries. The application layer and application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used for managing object lifecycles, stack management, thread management, security and exception management, and garbage collection. System libraries can include multiple functional modules. Examples include: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), and 2D graphics engines (e.g., SGL). The surface manager manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The media libraries support playback and recording of various common audio and video formats, as well as still image files. The media libraries support various audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG. 3D graphics processing libraries are used to implement 3D graphics drawing, image rendering, compositing, and layer processing. 2D (an animation method) graphics engines are drawing engines for 2D graphics.

[0079] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0080] It should be noted that the terminal device 300 in the embodiments of this application can be a terminal device including but not limited to smartphones, tablets, laptops, etc., and this application does not limit it.

[0081] See Figure 5 This is one of the exemplary flowcharts of an audio synchronization acquisition method provided in the embodiments of this application, which can be applied to, for example... Figure 2 The terminal device 203 shown or as Figure 3 In the terminal device 300 shown, the method may include the following steps:

[0082] S501 collects the first audio data played by the near-end device and the second audio data played by the far-end device.

[0083] The terminal device can acquire first audio data played by a near-end device and second audio data played by a far-end device through a connected audio receiver. The first audio data can be the aforementioned near-end audio data, and may include a first reference signal and first audio data to be played from the near-end device. The second audio data can be the aforementioned far-end audio data, and may include a second reference signal and second audio data to be played from the far-end device. To improve the accuracy of audio data acquisition, the audio receiver is set to a relatively high sampling rate, for example, a sampling rate of 48kHz.

[0084] In one possible implementation, the start-up playback time of the first audio data is approximately the same as the start-up playback time of the second audio data. Furthermore, during playback startup, the time interval between the first reference signal and the first audio data to be played, and the time interval between the second reference signal and the second audio data to be played, are the same. The start-up playback time of the first audio data can be the time when the terminal device sends a command to the playback control device, instructing the near-end device to play the first audio data. The start-up playback time of the second audio data can be the time when the remote device is controlled to play the second audio data. It should be noted that the start-up playback time of the first audio data and the start-up playback time of the second audio data can also be exactly the same.

[0085] To distinguish between the first and second audio tracks to be played, the first and second reference signals have different frequencies. For example, the first reference signal may have a frequency of 2kHz, and the second reference signal may have a frequency of 3kHz.

[0086] S502, determine the first time difference between the acquisition time of the first reference signal and the acquisition time of the second reference signal.

[0087] The audio acquisition and calculation module in the terminal device can determine whether the acquired audio data is a first reference signal or a second reference signal based on its frequency, thereby determining a first time difference between the acquisition time of the first reference signal and the acquisition time of the second reference signal. For example, suppose the frequency of the first reference signal is 2kHz and the frequency of the second reference signal is 3kHz. If the audio acquisition and calculation module determines that the frequency of the acquired audio data is 2kHz, then the terminal device can determine that the acquired audio data is the first reference signal. The time at which the audio acquisition and calculation module determines that the frequency of the acquired audio data is 2kHz is the acquisition time of the first reference signal. If the audio acquisition and calculation module determines that the frequency of the acquired audio data is 3kHz, then the terminal device can determine that the acquired audio data is the second reference signal. The time at which the audio acquisition and calculation module determines that the frequency of the acquired audio data is 3kHz is the acquisition time of the second reference signal.

[0088] In one example, when the audio acquisition and calculation module in the terminal device determines the frequency of the acquired audio data, it can use a Fast Fourier Transform (FFT) based on the duration of the first reference signal and the duration of the second reference signal to calculate the frequency of the acquired audio data. The durations of the first and second reference signals are the same, and these durations can be set according to actual conditions, such as 4ms or 5ms; this application does not limit this setting.

[0089] In one possible implementation, when determining the first time difference between the acquisition time of the first reference signal and the acquisition time of the second reference signal, timing can begin when either the first or second reference signal is first acquired, and end when the second or first reference signal, which is different from the first acquired reference signal, is acquired. The first time difference can be determined based on the timing result. For example, assuming the reference signal initially acquired by the terminal device is the first reference signal, timing begins when the first reference signal is acquired and ends when the terminal device acquires the second reference signal. If the timing result is 100ms, then the first time difference is 100ms.

[0090] In another possible implementation, when determining the first time difference between the acquisition time of the first reference signal and the acquisition time of the second reference signal, the audio frame at which the first or second reference signal is acquired can be set to 1. The number of audio frames between the acquisition of the second or first reference signal (different from the first acquired reference signal) and the time interval between setting the audio frame to 1 is recorded. Based on the number of audio frames, the first time difference can be determined by multiplying the duration of the audio frame by the number of audio frames. For example, assuming the reference signal initially acquired by the terminal device is the first reference signal, the audio frame at which the first reference signal is acquired is set to 1. If the second reference signal is acquired after an interval of 100 audio frames, and the duration of the audio frame is 4ms, then the first time difference is 400ms. It should be noted that the duration of the audio frame can be the same as or different from the duration of the first or second reference signal; this application does not limit this.

[0091] S503, adjust the time interval between the first reference signal and the first audio data to be played according to the first time difference.

[0092] Because remote devices and playback control devices play audio data through an audio decoding module, the audio data from a specific memory area is played according to a specific format. Furthermore, the audio data being played is not sent to the audio decoding module all at once, but rather frame by frame, over a set period of time. For example, if one frame is 10ms, the next 10ms of data needs to be sent to the audio decoding module within that 10ms period, resulting in 20ms of continuous audio playback. Therefore, the terminal device can adjust the time interval between the first reference signal and the first audio data to be played based on the first time difference determined in S502, synchronizing the acquisition time of the first audio data to be played with the acquisition time of the second audio data to be played.

[0093] Based on the above scheme, by acquiring a first reference signal and a second reference signal, a first time difference can be determined between the first audio data to be played on the near-end device and the second audio data to be played on the far-end device. Based on this first time difference, the time interval between the first reference signal and the first audio data to be played can be adjusted, thereby achieving time synchronization of audio data acquisition from different devices. This method can automatically calculate the delay between audio data played on different devices without manual intervention.

[0094] In some embodiments, the acquisition time of the first reference signal and the acquisition time of the second reference signal may include the following three cases:

[0095] Scenario 1: The acquisition time of the first reference signal precedes the acquisition time of the second reference signal. See also... Figure 6A and Figure 6B This diagram illustrates the acquisition time of the first reference signal and the acquisition time of the second reference signal, as provided in an embodiment of this application. Figure 6A The terminal device shown acquires the first reference signal and then acquires the second reference signal after a certain time interval. Since the time interval between the first reference signal and the first audio data to be played is the same as the time interval between the second reference signal and the second audio data to be played, this situation indicates that the first audio data to be played by the near-end device precedes the second audio data to be played. The terminal device can delay the playback time of the first audio data to be played to synchronize the acquisition time of the first audio data to be played with the acquisition time of the second audio data to be played.

[0096] In one example, when the terminal device delays the playback time of the first audio data to be played, it can insert blank audio data with a duration of a first time difference between the first reference signal and the first audio data to be played.

[0097] For example, see Figure 7AThis is one of the schematic diagrams of the first reference signal and the second reference signal before synchronization provided in the embodiments of this application. Figure 7A The first reference signal has a frequency of 2kHz, and the second reference signal has a frequency of 3kHz. Assuming the terminal device acquires the first reference signal 400ms later, and then acquires the second reference signal, a 400ms blank audio data segment can be inserted between the first reference signal and the first audio data to be played. See [link / reference]. Figure 7B This is one of the schematic diagrams of the synchronized first reference signal and second reference signal provided in an embodiment of this application. Assuming the duration of an audio frame is 4ms, 100 frames of blank audio data can be inserted between the first reference signal and the first audio data to be played. Figure 7B As shown, inserting blank data from frames 0 to 99 effectively delays the playback time of the first audio data to be played, thereby synchronizing the acquisition time of the first audio data to be played with the acquisition time of the second audio data to be played.

[0098] Scenario 2: The acquisition time of the first reference signal is the same as the acquisition time of the second reference signal. This indicates that the acquisition time of the first audio data to be played is synchronized with the acquisition time of the second audio data to be played, and no operation is required.

[0099] Scenario 3: The acquisition time of the second reference signal is earlier than the acquisition time of the first reference signal. For example... Figure 6B The terminal device, after acquiring the second reference signal, acquires the first reference signal after a certain time interval. Since the time interval between the first reference signal and the first audio data to be played is the same as the time interval between the second reference signal and the second audio data to be played, this situation indicates that the second audio data to be played by the near-end device precedes the first audio data to be played. The terminal device can advance the playback time of the first audio data to be played to synchronize the acquisition time of the first audio data to be played with the acquisition time of the second audio data to be played.

[0100] In one example, when the terminal device advances the playback time of the first audio data to be played, it can delete data with a duration of a first time difference between the first reference signal and the first audio data to be played.

[0101] In another example, when the terminal device advances the playback time of the first audio data to be played, it can also control the playback control device to skip the data with a duration of a first time difference between the first reference signal and the first audio data to be played.

[0102] For example, see Figure 8A This is one of the schematic diagrams of the first reference signal and the second reference signal before synchronization provided in the embodiments of this application. Figure 8AThe first reference signal has a frequency of 2kHz, and the second reference signal has a frequency of 3kHz. Assume the terminal device acquires the second reference signal 100ms later. The time interval between the first reference signal and the first audio data to be played is x, in milliseconds. See [link to relevant documentation]. Figure 8B This is one of the schematic diagrams of the synchronized first reference signal and second reference signal provided in an embodiment of this application. For example... Figure 8B As shown, the terminal device can skip 100ms of data by controlling the playback control device. That is, the terminal device adjusts the time interval between the first reference signal and the first audio data to be played to (x-100)ms, which is equivalent to advancing the playback time of the first audio data to be played, thereby synchronizing the acquisition time of the first audio data to be played with the acquisition time of the second audio data to be played.

[0103] In one possible implementation, since the receiving device, the remote device, and the near-end device are three different devices, it is impossible to control frame synchronization during audio processing within each device. To improve the synchronization accuracy between the acquisition time of the first audio data to be played and the acquisition time of the second audio data to be played, the deviation during audio processing within each device can be determined, and the time interval between the first reference signal and the first audio data to be played can be adjusted based on this deviation and the first time difference determined by the terminal device.

[0104] Specifically, N third reference signals can be added between the first reference signal in the first audio data and the first audio data to be played. N fourth reference signals are also included between the second reference signal in the second audio data and the second audio data to be played. After acquiring the first audio data played by the near-end device and the second audio data played by the far-end device, the terminal device can also determine a second time difference between the acquisition time of each of the N third reference signals and the acquisition time of the corresponding fourth reference signal. The terminal device can adjust the time interval between the first reference signal and the first audio data to be played based on the first time difference and the N second time differences.

[0105] Where N is a positive integer. There is a one-to-one correspondence between the N third reference signals and the N fourth reference signals. The duration of the third reference signal is the same as the duration of the fourth reference signal, and the duration of the third reference signal is the same as the duration of the first reference signal. The frequency of the third reference signal is also the same as the frequency of the first reference signal, and the frequency of the fourth reference signal is the same as the frequency of the second reference signal. The difference between the first time interval and the second time interval is less than or equal to the duration of the third reference signal. The first time interval is the time interval between the third reference signal and the first reference signal, and the second time interval is the time interval between the fourth reference signal corresponding to the third reference signal and the second reference signal.

[0106] For example, see Figure 9 This is one of the schematic diagrams of the third and fourth reference signals provided in the embodiments of this application. Figure 9 Assuming N is 3, and the durations of the third and fourth reference signals are both 4ms, the first audio data includes three third reference signals, ordered by their time intervals with the first reference signal from smallest to largest: signal 0, signal 1, and signal 2. The second audio data includes three fourth reference signals, ordered by their time intervals with the second reference signal from smallest to largest: signal 3, signal 4, and signal 5. Therefore, the fourth reference signal corresponding to signal 0 is signal 3, the fourth reference signal corresponding to signal 1 is signal 4, and the fourth reference signal corresponding to signal 2 is signal 5. Thus, when the time interval between signal 0 and the first reference signal is the first time interval, the time interval between signal 3 and the second reference signal is the second time interval. Therefore, the difference between the first and second time intervals is less than or equal to 4ms.

[0107] In one possible implementation, N can be determined based on the duration of the third reference signal, or N can be determined based on the duration of the fourth reference signal.

[0108] In one example, N can satisfy formula (1).

[0109] N = 2*(n-1) Formula (1)

[0110] In the formula, n is the duration of the third reference signal or the duration of the fourth reference signal.

[0111] For example, when the duration of the third reference signal is 4ms, the first audio data may include 6 third reference signals. When the duration of the third reference signal is 5ms, the first audio data may include 8 third reference signals.

[0112] In one possible implementation, when the terminal device adjusts the time interval between the first reference signal and the first audio data to be played based on the first time difference and N second time differences, it can determine M third time differences from the N second time differences. The time delay deviation between the near-end device and the far-end device is determined based on the number of third time differences. Then, the time interval between the Nth third reference signal and the first audio data to be played is adjusted based on the time delay deviation and the first time difference. Here, the absolute value of the third time difference is greater than the sum of the durations of the first time difference and the third reference signal, and M is an integer greater than or equal to 0 and less than or equal to N. It should be understood that the method for adjusting the time interval between the Nth third reference signal and the first audio data to be played can refer to the method for adjusting the time interval between the first reference signal and the first audio data to be played described above, and will not be repeated here.

[0113] For example, suppose the first audio data includes six third reference signals, the second audio data includes six fourth reference signals, the first time difference is 100ms, and the duration of the third reference signal is 4ms. The absolute values ​​of the six second time differences are 101ms, 103ms, 106ms, 102ms, 101ms, and 101ms, respectively. Since 106ms is greater than 104ms, we can determine that 106ms is the third time difference; that is, one third time difference can be determined from the six second time differences. Based on the number of third time differences, the time delay deviation between the near-end and far-end devices can be determined to be 1ms. Then, the time interval between the sixth third reference signal and the first audio data to be played is adjusted based on the sum of the time delay deviation and the first time difference.

[0114] In one possible implementation, the time interval between any two adjacent third reference signals among the N third reference signals is a reference interval value. The time interval between any two adjacent fourth reference signals among the N fourth reference signals is different, and the time interval between any two adjacent fourth reference signals is determined based on the duration of the fourth reference signal and the reference interval value.

[0115] In one example, see Figure 10 This is one of the schematic diagrams of the third and fourth reference signals provided in an embodiment of this application. For example... Figure 10 As shown, assume the duration of the fourth reference signal is n, and the reference interval is K. Then, the time interval between any two adjacent third reference signals is K. The time interval between the first fourth reference signal and the second reference signal is K-1, and the time intervals between any two subsequent adjacent fourth reference signals are K-2, K-3, ..., K+n, K+1, K+2, ..., K+(n-1)-1. The first fourth reference signal represents the fourth reference signal among the N fourth reference signals that has the shortest time interval with the first reference signal.

[0116] Alternatively, the time interval between any two adjacent fourth reference signals among the N fourth reference signals is the reference interval value; the time interval between any two adjacent third reference signals among the N third reference signals is different, and the time interval between any two adjacent third reference signals is determined based on the duration of the third reference signal and the reference interval value.

[0117] It should be noted that the reference interval value can be set based on experience or actual conditions, but it must be a multiple of the duration of the third reference signal or the fourth reference signal. For example, when the duration of the third reference signal is 4ms, the reference interval value can be 200ms, 240ms, or 320ms, etc. This application does not impose specific limitations on the value of the reference interval.

[0118] In one example, see Figure 11 This is one of the schematic diagrams of the third and fourth reference signals provided in an embodiment of this application. For example... Figure 11 As shown, assuming the duration of the third reference signal is n and the reference interval is K, the time interval between the first third reference signal and the first reference signal is K-1, and the time intervals between any two subsequent third reference signals are K-2, K-3, ..., K+n, K+1, K+2, ..., K+(n-1)-1. The time interval between any two adjacent fourth reference signals is K. The first third reference signal represents the third reference signal with the shortest time interval to the first reference signal among the N third reference signals.

[0119] In one possible implementation, the first and third reference signals can be collectively referred to as near-end reference signals, and the second and fourth reference signals as far-end reference signals. The following explanation uses an example where the first time difference is 0, the first audio data includes 7 near-end reference signals, and the second audio data includes 7 far-end reference signals, with the time interval between any two adjacent near-end reference signals serving as the reference interval. The audio frames acquired by the receiving device are aligned with the audio frames of the near-end device. (See also...) Figure 12 This is a schematic diagram of the near-end reference signal and the far-end reference signal provided in the embodiments of this application. Figure 12 Assuming a reference interval of 200ms, the time interval between any two adjacent far-end reference signals is 200ms. The time intervals between any two adjacent far-end reference signals are 199ms, 198ms, 197ms, 204ms, 201ms, and 202ms, respectively. AL represents 1ms of data. Figure 12 In the above scenario, the EFGH frames of the near-end reference signal and the far-end reference signal are acquired by the recording device as the same audio frame. In this case, the 0th to 6th near-end reference signals and the far-end reference signal can all be acquired simultaneously. Therefore, it can be determined that the time delay deviation between the near-end device and the far-end device is 0 in this situation.

[0120] Suppose that the recording device did not capture a complete frame of the near-end reference signal, but rather the captured audio frame included both signal and no signal.

[0121] In one example, the radio receiver captured BCDE. For example... Figure 12As shown, since only 1ms of data at E is acquired in BCDE, while 3ms of data is acquired in the next frame FGHI, the terminal device discards BCDE and determines the audio frame of the acquired near-end reference signal as FGHI. Therefore, when acquiring the first far-end reference signal, data can be acquired at FGHI. When acquiring the second far-end reference signal, data can be acquired at FGHI. When acquiring the third far-end reference signal, no data can be acquired at FGHI, but data can be acquired at the preceding audio frame BCDE. When acquiring the fourth far-end signal, data can be acquired again at FGHI. Since one of the seven far-end reference signals cannot be acquired at FGHI, but can be acquired at the preceding audio frame FGHI, it can be determined that the time delay deviation between the far-end and near-end devices is 1ms, and the far-end device is 1ms ahead of the near-end device. Therefore, when the terminal device adjusts the time interval between the first reference signal and the first audio data to be played based on the first time difference and N second time differences, it can adjust the time interval between the first reference signal and the first audio data to be played based on the difference between the first time difference and 1ms.

[0122] In another example, the recording device captures DEFG. For example... Figure 12 As shown, when acquiring the first far-end reference signal, data can be acquired at DEFG. When acquiring the second far-end reference signal, data can be acquired at DEFG. When acquiring the third far-end reference signal, data can be acquired at DEFG. When acquiring the fourth far-end reference signal, data can be acquired at FGG. When acquiring the fifth far-end reference signal, data can be acquired at GGG. When acquiring the sixth far-end reference signal, no data can be acquired at DEFG, but it is expected that data can be acquired at the next audio frame HIJK of DEFG. Since one of the seven far-end reference signals cannot be acquired at DEFG, but can be acquired at the next audio frame of DEFG, it can be determined that the time delay deviation between the far-end device and the near-end device is 1ms, and the far-end device is delayed by 1ms compared to the near-end device. Therefore, when the terminal device adjusts the time interval between the first reference signal and the first audio data to be played based on the first time difference and N second time differences, it can adjust the time interval between the first reference signal and the first audio data to be played based on the sum of the first time difference and 1ms. Other cases can be deduced similarly, and will not be elaborated further in this application.

[0123] In one possible implementation, if the first time difference is not zero, the third reference signal can be adjusted based on the first time difference, and then the time delay deviation between the remote and near-end devices can be determined using the method described above. For example, suppose the acquisition time of the first reference signal is earlier than the acquisition time of the second reference signal, and the first time difference is 100ms. Then, each third reference signal can be delayed by 100ms, and the time delay deviation between the remote and near-end devices can be determined using the method described above.

[0124] Below, in order to more clearly understand the solution proposed in the embodiments of this application, an audio synchronization acquisition method provided by this application will be introduced in conjunction with specific embodiments.

[0125] See Figures 13A-13E This is one of the exemplary flowcharts of an audio synchronization acquisition method provided in this application embodiment. In this embodiment, the frequencies of the first reference signal and the third reference signal are both 2kHz, and the frequencies of the second reference signal and the fourth reference signal are both 3kHz. The process may include the following steps:

[0126] S1301, activates the remote device, the near device, and the radio device.

[0127] The terminal device starts the radio recording equipment to collect audio data, controls the near-end device to play the first audio data, and simultaneously controls the far-end device to play the second audio data.

[0128] S1302, set the data acquisition flag to 0.

[0129] S1303, the terminal device calculates the frequency of the collected audio data.

[0130] The terminal device calculates the frequency of the acquired audio data through the audio acquisition and calculation module.

[0131] S1304 determines whether the frequency is 2KHz.

[0132] If S1303 calculates the frequency of the audio data to be 2kHz, then S1305 is executed. If S1303 calculates the frequency of the audio data to be other than 2kHz, then... Figure 13B S1307 is shown.

[0133] S1305, determine if the data acquisition identifier is 0.

[0134] If the data acquisition flag is determined to be 0, then execute S1306. If the data acquisition flag is determined to be non-0, then execute... Figure 13C S1308 is shown.

[0135] S1306, set the data acquisition flag to 1 and start timing.

[0136] When the acquisition flag is set to 1, it indicates that the first audio data to be played is played before the second audio data to be played.

[0137] S1307, determine if the frequency is 3KHz.

[0138] If S1303 calculates the frequency of the audio data to be 3kHz, then execute... Figure 13D S1312 is shown. If the frequency of the audio data calculated by S1303 is not 3KHz, then S1303 is executed.

[0139] S1308, the second audio data to be played precedes the first audio data to be played, and the timer ends.

[0140] S1309, determine the first time difference and the second time difference.

[0141] The terminal device can determine the first time difference and N second time differences based on the timing results.

[0142] S1310, send a command to the playback control device.

[0143] S1311, Adjust the time interval between the first reference signal and the first audio data to be played.

[0144] The method for adjusting the time interval between the first reference signal and the first audio data to be played can be found in, for example... Figure 2 The relevant descriptions in the method embodiments shown are not repeated here.

[0145] S1312, determine whether the collection identifier is 1.

[0146] If the data acquisition flag is determined to be 1, then execute S1313. If the data acquisition flag is determined to be not 1, then execute... Figure 13E S1314 is shown.

[0147] S1313, set the data acquisition flag to 2 and start timing.

[0148] When the acquisition flag is set to 2, it indicates that the second audio data to be played is played before the first audio data to be played.

[0149] S1314, the first audio data to be played precedes the second audio data to be played, and the timer ends.

[0150] Based on the same concept as the above method, see [link to relevant documentation]. Figure 14 This application provides an audio synchronization acquisition device 1400, which is capable of executing the steps in the above-described method. To avoid repetition, these steps will not be described in detail here. The device 1400 includes an acquisition unit 1401 and a processing unit 1402. In one scenario:

[0151] Acquisition unit 1401 is used to acquire first audio data played by a near-end device and second audio data played by a far-end device. The first audio data includes a first reference signal and first audio data to be played by the near-end device; the second audio data includes a second reference signal and second audio data to be played by the far-end device; the first reference signal and the second reference signal have different frequencies; when playback is started, the time interval between the first reference signal and the first audio data to be played, and the time interval between the second reference signal and the second audio data to be played are the same.

[0152] The processing unit 1402 is configured to determine a first time difference between the acquisition time of the first reference signal and the acquisition time of the second reference signal; and adjust the time interval between the first reference signal and the first audio data to be played according to the first time difference, so that the acquisition time of the first audio data to be played is synchronized with the acquisition time of the second audio data to be played.

[0153] In one possible implementation, when the processing unit 1402 adjusts the time interval between the first reference signal and the first audio data to be played according to the first time difference, it is further configured to: if the acquisition time of the first reference signal is earlier than the acquisition time of the second reference signal, insert blank audio data with a duration equal to the first time difference between the first reference signal and the first audio data to be played.

[0154] In one possible implementation, when the processing unit 1402 adjusts the time interval between the first reference signal and the first audio data to be played according to the first time difference, it is further configured to: if the acquisition time of the second reference signal is earlier than the acquisition time of the first reference signal, then delete the data between the first reference signal and the first audio data to be played with a duration equal to the first time difference.

[0155] In one possible implementation, the first audio data further includes N third reference signals, and the second audio data further includes N fourth reference signals, where N is a positive integer; the N third reference signals are located between the first reference signal and the first audio data to be played, and the N fourth reference signals are located between the second reference signal and the second audio data to be played; the N third reference signals correspond one-to-one with the N fourth reference signals, the difference between the first time interval and the second time interval is less than or equal to the duration of the third reference signal, the duration of the third reference signal is the same as the duration of the fourth reference signal, the first time interval is the time interval between the third reference signal and the first reference signal, and the second time interval is the time interval between the fourth reference signal corresponding to the third reference signal and the second reference signal. The processing unit 1402 is further configured to: determine a second time difference between the acquisition time of each of the N third reference signals and the acquisition time of the corresponding fourth reference signal; adjust the time interval between the first reference signal and the first audio data to be played according to the first time difference, including: adjusting the time interval between the first reference signal and the first audio data to be played according to the first time difference and the N second time differences.

[0156] In one possible implementation, N is determined based on the duration of the third reference signal, or N is determined based on the duration of the fourth reference signal.

[0157] In one possible implementation, when the processing unit 1402 adjusts the time interval between the first reference signal and the first audio data to be played based on the first time difference and N second time differences, it is further configured to: determine M third time differences from the N second time differences; the third time differences are greater than the sum of the durations of the first time difference and the third reference signal, where M is an integer greater than or equal to 0 and less than or equal to N; determine the time delay deviation between the near-end device and the far-end device based on the number of third time differences; and adjust the time interval between the Nth third reference signal and the first audio data to be played based on the time delay deviation and the first time difference.

[0158] In one possible implementation, the time interval between any two adjacent third reference signals among the N third reference signals is a reference interval value; the time interval between any two adjacent fourth reference signals among the N fourth reference signals is different, and the time interval between any two adjacent fourth reference signals is determined based on the duration of the fourth reference signal and the reference interval value; or, the time interval between any two adjacent fourth reference signals among the N fourth reference signals is a reference interval value; the time interval between any two adjacent third reference signals among the N third reference signals is different, and the time interval between any two adjacent third reference signals is determined based on the duration of the third reference signal and the reference interval value.

[0159] Based on the same concept as the above method, see [link to relevant documentation]. Figure 15 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. The terminal device includes at least one processor 1502 and a memory 1501 connected or coupled to the at least one processor 1502. In addition, the terminal device may also include a communication interface 1503. The terminal device can interact with other devices through the communication interface 1503.

[0160] For example, the communication interface 1503 can be a transceiver, circuit, bus, module, pin, or other type of communication interface. When the terminal device is a chip-based device or circuit, the communication interface 1503 in the terminal device can also be an input / output circuit, capable of inputting information (or receiving information) and outputting information (or sending information). The processor can be an integrated processor, a microprocessor, an integrated circuit, or a logic circuit, and the processor can determine the output information based on the input information.

[0161] The coupling in this application embodiment is an indirect coupling or communication connection between devices, units, or modules, which can be electrical, mechanical, or other forms, used for information exchange between devices, units, or modules. The processor 1502 may operate in conjunction with the memory 1501 and the communication interface 1503. This application does not limit the specific connection medium between the processor 1502, the memory 1501, and the communication interface 1503.

[0162] Optional, see Figure 15The processor 1502, the memory 1501, and the communication interface 1503 are interconnected via a bus. This bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 15 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0163] In this embodiment, memory 1501, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 1501 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. Memory 1501 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. Memory 1501 in this embodiment may also be a circuit or any other device capable of implementing storage functions for storing instructions, computer programs, and / or data.

[0164] In this application embodiment, the processor 1502 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the various methods, steps, and logic block diagrams disclosed in this application embodiment. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the audio synchronization acquisition method disclosed in this application embodiment can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0165] By designing and programming the processor 1502, the code corresponding to the audio synchronization acquisition method described in the foregoing embodiments can be embedded into the chip, so that the chip can execute the steps of the aforementioned audio synchronization acquisition method when running. How to design and program the processor 1502 is a well-known technique to those skilled in the art, and will not be described in detail here.

[0166] In one or more embodiments, memory 1501 stores instructions that can be executed by at least one processor 1502, which can implement the steps of any of the above methods by calling the instructions or computer programs stored in memory 1501.

[0167] This application also provides a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the steps of any of the above methods.

[0168] In some embodiments, various aspects of the audio synchronization acquisition method provided in this application may also be implemented in the form of a computer program product, which includes program code. When the computer program product is run on a terminal device, the program code is used to cause the terminal device to perform the steps of any of the methods described above in this specification.

[0169] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0170] While specific embodiments of this application have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of this application is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of this application, and all such changes and modifications fall within the scope of protection of this application. Although preferred embodiments of this application have been described, those skilled in the art, once they understand the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0171] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An audio synchronization acquisition method, characterized in that, include: The system collects first audio data played by a near-end device and second audio data played by a far-end device; the first audio data includes a first reference signal and first audio data to be played by the near-end device, and the second audio data includes a second reference signal and second audio data to be played by the far-end device; the first reference signal and the second reference signal have different frequencies; when playback is started, the time interval between the first reference signal and the first audio data to be played, and the time interval between the second reference signal and the second audio data to be played are the same; Determine a first time difference between the acquisition time of the first reference signal and the acquisition time of the second reference signal; The time interval between the first reference signal and the first audio data to be played is adjusted according to the first time difference, so that the acquisition time of the first audio data to be played is synchronized with the acquisition time of the second audio data to be played.

2. The method according to claim 1, characterized in that, The step of adjusting the time interval between the first reference signal and the first audio data to be played according to the first time difference includes: If the acquisition time of the first reference signal is earlier than the acquisition time of the second reference signal, then blank audio data with a duration equal to the first time difference is inserted between the first reference signal and the first audio data to be played.

3. The method according to claim 1, characterized in that, The step of adjusting the time interval between the first reference signal and the first audio data to be played according to the first time difference includes: If the acquisition time of the second reference signal is earlier than the acquisition time of the first reference signal, then delete the data between the first reference signal and the first audio data to be played, with a duration equal to the first time difference.

4. The method according to claim 1, characterized in that, The first audio data further includes N third reference signals, and the second audio data further includes N fourth reference signals, where N is a positive integer; the N third reference signals are located between the first reference signal and the first audio data to be played, and the N fourth reference signals are located between the second reference signal and the second audio data to be played; the N third reference signals and the N fourth reference signals correspond one-to-one, the difference between the first time interval and the second time interval is less than or equal to the duration of the third reference signal, the duration of the third reference signal is the same as the duration of the fourth reference signal, the first time interval is the time interval between the third reference signal and the first reference signal, and the second time interval is the time interval between the fourth reference signal corresponding to the third reference signal and the second reference signal; The method further includes: Determine the second time difference between the acquisition time of each of the N third reference signals and the acquisition time of the corresponding fourth reference signal; Adjusting the time interval between the first reference signal and the first audio data to be played based on the first time difference includes: The time interval between the first reference signal and the first audio data to be played is adjusted based on the first time difference and N second time differences.

5. The method according to claim 4, characterized in that, N is determined based on the duration of the third reference signal, or N is determined based on the duration of the fourth reference signal.

6. The method according to claim 5, characterized in that, The step of adjusting the time interval between the first reference signal and the first audio data to be played based on the first time difference and N second time differences includes: M third time differences are determined from N second time differences; the third time differences are greater than the sum of the durations of the first time differences and the third reference signal, and M is an integer greater than or equal to 0 and less than or equal to N; The time delay deviation between the near-end device and the far-end device is determined based on the number of the third time difference; The time interval between the Nth third reference signal and the first audio data to be played is adjusted according to the time delay deviation and the first time difference.

7. The method according to any one of claims 4-6, characterized in that, The time interval between any two adjacent third reference signals among the N third reference signals is the reference interval value; The time intervals between any two adjacent fourth reference signals among the N fourth reference signals are all different, and the time intervals between any two adjacent fourth reference signals are determined based on the duration of the fourth reference signal and the reference interval value; or, The time interval between any two adjacent fourth reference signals among the N fourth reference signals is the reference interval value; The time intervals between any two adjacent third reference signals among the N third reference signals are all different, and the time intervals between any two adjacent third reference signals are determined based on the duration of the third reference signal and the reference interval value.

8. An audio synchronization acquisition device, characterized in that, include: The acquisition unit is used to acquire first audio data played by a near-end device and second audio data played by a far-end device. The first audio data includes a first reference signal and first audio data to be played by the near-end device; the second audio data includes a second reference signal and second audio data to be played by the far-end device; the first reference signal and the second reference signal have different frequencies; when playback is started, the time interval between the first reference signal and the first audio data to be played, and the time interval between the second reference signal and the second audio data to be played are the same. The processing unit is configured to determine a first time difference between the acquisition time of the first reference signal and the acquisition time of the second reference signal; and adjust the time interval between the first reference signal and the first audio data to be played according to the first time difference, so that the acquisition time of the first audio data to be played is synchronized with the acquisition time of the second audio data to be played.

9. A terminal device, characterized in that, include: Memory, used to store computer instructions; A processor, connected to the memory, is configured to execute computer instructions in the memory, and, in executing the computer instructions, implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, include: The computer-readable storage medium stores computer instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 7.

Images