A method and device for controlling audio-visual synchronization, a computer device and a storage medium

Abstract

The application discloses a kind of method, device, computer equipment and storage medium for controlling sound picture synchronization, to solve the problem that display device cannot carry out sound picture synchronization to audio data and picture data when mobile device respectively transmits the audio data and picture data of aging media resource to display device.The method comprises: associating the picture data in the media resource with the corresponding picture timestamp, obtaining the picture queue carrying the picture timestamp;Obtain the first frame data in the audio data in the media resource, obtain the first frame timestamp corresponding to the first frame data, and obtain the audio timestamp corresponding to the audio data;Subtract the first frame timestamp from the audio timestamp to obtain the residual timestamp;Unit conversion is carried out on the residual timestamp to obtain the audio nanosecond time in nanosecond unit;The audio nanosecond time is associated with the corresponding audio data to obtain the audio queue;Picture queue and audio queue are pushed to display device.

Description

Technical Field

[0001] This invention relates to the field of communications, and more particularly to a method, apparatus, computer device, and storage medium for controlling audio-visual synchronization. Background Technology

[0002] With the advancement of technology, more and more display devices and mobile devices are entering people's lives and work. In daily life and work, people sometimes need to transfer media data from mobile devices to display devices and play it in real time during the transfer. However, when display devices play media data in real time, because they usually need to play both the video and the corresponding audio simultaneously, there are occasional problems where the audio and video cannot be synchronized, causing abnormal playback of the media data on the display device. This problem becomes particularly serious when media resources have aged over a long period of time.

[0003] In existing technologies, audio and video can be encapsulated together for transmission to prevent audio-video mismatch issues in aging media resources, thus achieving audio-visual synchronization. However, in practical applications, due to device limitations, it is sometimes necessary to transmit the audio and video of media resources separately. In such cases, because there is no effective synchronization mechanism between the video and audio, display devices often experience playback abnormalities when playing aging media resources in real time due to the lack of synchronization between the video and audio. Summary of the Invention

[0004] This invention provides a method, apparatus, computer device, and storage medium for controlling audio-visual synchronization, in order to solve the problem that when a mobile device transmits audio data and video data of aging media resources to a display device, the display device cannot synchronize the audio data and video data.

[0005] In a first aspect, the present invention provides a method for controlling audio-visual synchronization, comprising:

[0006] By associating the image data in the media resources with the corresponding image timestamps, an image queue is obtained;

[0007] Obtain the first frame of audio data from the media resource, obtain the first frame timestamp corresponding to the first frame of data, and obtain the audio timestamp corresponding to the audio data;

[0008] Subtract the first frame timestamp from the audio timestamp to obtain the residual timestamp;

[0009] The residual timestamp is converted to obtain the audio nanosecond time in nanoseconds.

[0010] The audio nanosecond time is associated with the corresponding audio data to obtain the audio queue;

[0011] The video queue and the audio queue are pushed to the display device so that the display device can synchronously play the video data and the audio data in real time based on the video timestamp and the audio nanosecond time.

[0012] In one possible design, the step of converting the residual timestamp to obtain the audio nanosecond time in nanoseconds includes:

[0013] The residual timestamp is converted to a unit to obtain a nanosecond timestamp;

[0014] Divide the nanosecond timestamp by the sampling rate of the audio data to obtain the audio nanosecond time.

[0015] In one possible design, after obtaining the first frame data of the audio data in the media resource and obtaining the first frame timestamp corresponding to the first frame data, the method further includes:

[0016] Determine whether the audio data has been initialized;

[0017] If not initialized, the audio buffer data in the device buffer is read as the first frame data.

[0018] In one possible design, associating the audio nanosecond time with the corresponding audio data to derive the audio queue includes:

[0019] The audio nanosecond time is decoded into a display timestamp;

[0020] The audio queue is obtained by associating the displayed timestamp with the corresponding audio data.

[0021] In one possible design, pushing the audio queue to the display device includes:

[0022] The audio queue is sent to the display device via a GStreamer buffer.

[0023] In one possible design, after associating the audio nanosecond time with the corresponding audio data to obtain the audio queue, the method further includes:

[0024] Determine whether the audio nanosecond time matches the expected time;

[0025] If the conditions are not met, a synchronization adjustment command is sent to the display device to adjust the playback rate of the audio data.

[0026] In one possible design, before determining whether the audio nanosecond time matches the expected time, the method further includes:

[0027] Multiply the sampling rate, number of channels, and sampling bit depth of the audio data to obtain an intermediate value;

[0028] The expected time is obtained by dividing the amount of audio data by the intermediate value.

[0029] Secondly, a device for controlling audio-visual synchronization is provided, comprising:

[0030] The first association module is used to associate the image data in the media resources with the corresponding image timestamps to obtain the image queue;

[0031] The acquisition module is used to acquire the first frame data of the audio data in the media resource, obtain the first frame timestamp corresponding to the first frame data, and acquire the audio timestamp corresponding to the audio data.

[0032] The removal module is used to subtract the first frame timestamp from the audio timestamp to obtain the residual timestamp;

[0033] The conversion module is used to convert the residual timestamp to a unit to obtain the audio nanosecond time in nanoseconds.

[0034] The second association module is used to associate the audio nanosecond time with the corresponding audio data to obtain the audio queue;

[0035] The push module is used to push the screen queue and the audio queue to the display device, so that the display device can synchronously play the screen data and the audio data in real time according to the screen timestamp and the audio nanosecond time.

[0036] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method for controlling audio-visual synchronization.

[0037] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described method for controlling audio-visual synchronization.

[0038] The aforementioned method, apparatus, computer device, and storage medium for controlling audio-visual synchronization firstly obtain a frame queue carrying frame timestamps by associating the frame data of the media resource with the corresponding frame timestamps. Then, the first frame of audio data in the media resource is acquired, the first frame timestamp corresponding to the first frame data is obtained, and the audio timestamp corresponding to the audio data is also acquired. The first frame timestamp is subtracted from the audio timestamp to obtain the residual timestamp. In practical applications, the residual timestamp initially allows for stable audio data output. However, due to issues with device clock accuracy and aging during actual use, the time difference between audio and video will gradually increase. This is because the current residual timestamp is based on a multiple of the audio sampling rate, hence the existence of a certain time difference. Therefore, the residual timestamp is converted to obtain the audio nanosecond time in nanoseconds; the audio nanosecond time is associated with the corresponding audio data to obtain the audio queue; the picture queue and audio queue are pushed to the display device. When the display device obtains the picture queue and audio queue, it synchronously plays the picture data and the audio data in real time according to the picture timestamp and the audio nanosecond time, thereby ensuring that the media resources with synchronized audio and video are transmitted from the mobile device to the display device and played in real time. Attached Figure Description

[0039] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a schematic diagram of an application environment for a method of controlling audio-visual synchronization according to an embodiment of the present invention;

[0041] Figure 2 This is a flowchart illustrating a method for controlling audio-visual synchronization in one embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram of a device for controlling audio-visual synchronization in one embodiment of the present invention;

[0043] Figure 4 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] The method for controlling audio-visual synchronization provided in this invention can be applied to, for example... Figure 1 In this application environment, mobile devices communicate with display devices via a network. The mobile device processes the video and audio data from media resources, combining the audio and video timestamps to convert this data into video and audio queues, which are then sent to the display device. The display device can then read the corresponding video and audio data from these queues and play the synchronized video and audio data using the video timestamp and audio nanosecond timing. Display devices include, but are not limited to, televisions and personal computers. Mobile devices can be, but are not limited to, smartphones, tablets, and portable wearable devices.

[0046] In one embodiment, such as Figure 2 As shown, a method for controlling audio-visual synchronization is provided, which is then applied to... Figure 1 Taking mobile devices as an example, the explanation includes the following steps:

[0047] S10: Associate the video data of the media resources with the corresponding video timestamps to obtain the video queue.

[0048] The mobile device acquires media resources that need to be transmitted to the display device, reads the video data from the media resources, and associates the video timestamps with the corresponding video data to obtain a video queue. Here, the video timestamp refers to the device timestamp corresponding to the video data, and the video timestamp is determined by the playback frame rate. The video queue refers to the data stream queue containing the video frames with their respective timestamps.

[0049] Specifically, when a mobile device connects to a display device, the mobile device obtains the corresponding screen timestamp each time it reads the screen data of the media resource, and saves the screen data and screen timestamp into the screen queue through the following procedure.

[0050] S20: Obtain the first frame data of the audio data in the media resource, obtain the first frame timestamp corresponding to the first frame data, and obtain the audio timestamp corresponding to the audio data.

[0051] Acquire audio data from media resources, specifically the first frame of the audio data. Use the device timestamp corresponding to the first frame as the first frame timestamp. Use the device timestamp corresponding to the audio data as the audio timestamp.

[0052] S30: Subtract the first frame timestamp from the audio timestamp to obtain the residual timestamp.

[0053] Subtract the first frame timestamp from the audio timestamp to obtain the residual timestamp.

[0054] It is worth noting that the residual timestamp obtained in step S30 is a stable timestamp. At this point, the audio and video of the media resource are synchronized. Therefore, step S30 essentially improves the timing error between the playback of video and audio data. However, due to device timing accuracy errors and media resource aging issues, the longer the real-time playback time, the greater the playback error between video and audio data.

[0055] S40: Convert the residual timestamp to obtain the audio nanosecond time in nanoseconds.

[0056] Since the residual timestamp is in multiples of the audio sampling rate, this embodiment converts the residual timestamp to obtain the audio nanosecond time in nanoseconds.

[0057] It is worth noting that the reason for the unit conversion of the residual timestamp in step S40 is because, as discovered in step S30 above, due to device time accuracy errors and media resource aging, the longer the real-time playback time, the greater the playback error between the video and audio data. In practical applications, it was found that this error is caused by the unit of the residual timestamp being a multiple of the audio sampling rate, rather than nanoseconds. Therefore, step S40 converts the unit of the residual timestamp to reduce the time error between the video and audio data during playback.

[0058] S50: Associate the audio nanosecond time with the corresponding audio data to obtain the audio queue.

[0059] By associating audio nanosecond times with the corresponding audio data, an audio queue is obtained. Here, the audio queue refers to a data stream queue containing video and audio data with audio nanosecond times.

[0060] S60: Push the screen queue and the audio queue to the display device so that the display device can synchronously play the screen data and the audio data in real time according to the screen timestamp and the audio nanosecond time.

[0061] The mobile device pushes the screen queue and audio queue to the display device. Upon receiving the screen queue and audio queue, the display device synchronously plays the screen and audio data in real time, based on the screen timestamp in the screen queue and the audio nanosecond duration in the audio queue.

[0062] It should be noted that this embodiment associates the video data of the media resource with the corresponding video timestamp to obtain a video queue carrying video timestamps. Then, the first frame of audio data in the media resource is obtained, the first frame timestamp corresponding to the first frame data is obtained, and the audio timestamp corresponding to the audio data is obtained. The first frame timestamp is subtracted from the audio timestamp to obtain the residual timestamp. In practical applications, the audio data can be output stably at first based on the residual timestamp. However, due to the accuracy of the device clock and the aging problem in actual use, the time difference between audio and video will gradually increase. This is because the current residual timestamp is based on a multiple of the audio sampling rate, hence the existence of a certain time difference. Therefore, the residual timestamp is converted to obtain the audio nanosecond time in nanoseconds; the audio nanosecond time is associated with the corresponding audio data to obtain the audio queue; the picture queue and audio queue are pushed to the display device. When the display device obtains the picture queue and audio queue, it synchronously plays the picture data and the audio data in real time according to the picture timestamp and the audio nanosecond time, thereby ensuring that the media resources with synchronized audio and video are transmitted from the mobile device to the display device and played in real time.

[0063] In one embodiment, step S40, which involves converting the residual timestamp to obtain the audio nanosecond time in nanoseconds, specifically includes the following steps:

[0064] S41: Convert the residual timestamp to a unit to obtain a nanosecond timestamp.

[0065] S42: Divide the nanosecond timestamp by the sampling rate of the audio data to obtain the audio nanosecond time.

[0066] First, the residual timestamp is converted to nanosecond timestamps. Then, the nanosecond timestamp is divided by the sampling rate of the audio data to obtain the audio nanosecond time.

[0067] It should be noted that while the residual timestamp is measured in multiples of the audio sampling rate, in practice, it has been found that the mismatch between the residual timestamp and the image timestamp leads to increasingly larger playback errors between the image and audio data. Therefore, in this embodiment, the unit of the residual timestamp is converted to a nanosecond timestamp, which is then divided by the audio data sampling rate to obtain the audio nanosecond time in nanoseconds. This audio nanosecond time matches the image timestamp, so when the display device receives the image queue and audio queue and reads the image and audio data for playback, it can maintain the synchronization of the image and audio data through the audio nanosecond time and image timestamp, thereby ensuring that the media resources transmitted from the mobile device to the display device and played in real time are synchronized with the audio.

[0068] In one embodiment, after step S20, i.e., after obtaining the first frame data of the audio data in the media resource and obtaining the first frame timestamp corresponding to the first frame data, the method further includes the following steps:

[0069] S71: Determine whether the audio data has been initialized.

[0070] S72: If not initialized, read the audio buffer data from the device buffer as the first frame data.

[0071] It checks whether the audio data has been initialized. If not, it reads the audio buffer data from the device buffer as the first frame. The audio buffer data consists of the audio data frames currently entering the device buffer.

[0072] It should be noted that the first frame of data initialized in this embodiment is not assigned a value. Therefore, it is necessary to determine whether the audio data has been initialized. If not, it means that the first frame of data is empty. At this time, it is necessary to read the audio buffer data from the device buffer as the first frame of data. Since the audio data needs to be processed based on the first frame of data in the future, it is necessary to ensure that an accurate residual timestamp is obtained. Therefore, steps S71-S72 effectively ensure that an accurate residual timestamp is obtained in the future. In order to obtain the audio nanosecond time through the residual timestamp, when the display device receives the picture queue and the audio queue and reads the picture data and audio data for playback, it can maintain the synchronization of the picture data and audio data through the audio nanosecond time and the picture timestamp, thereby ensuring that the media resources with synchronized audio and video are transmitted from the mobile device to the display device and played in real time.

[0073] In one embodiment, step S50, which involves associating the audio nanosecond time with the corresponding audio data to obtain an audio queue, specifically includes the following steps:

[0074] S51: Decode the audio nanosecond time into a display timestamp.

[0075] S52: Associate the displayed timestamp with the corresponding audio data to obtain the audio queue.

[0076] The audio nanosecond time is decoded into a display timestamp (PST), and then the display timestamp is associated with the corresponding audio data to obtain the audio queue.

[0077] It should be noted that in this embodiment, the audio nanosecond time is essentially decoded into a display timestamp, and then the display timestamp is associated with the audio data to obtain the audio queue. Since the display timestamp is more conducive to the transmission of the audio queue than the audio nanosecond time, the display device's reception is more accurate and efficient. This ensures that when the display device receives the video queue and audio queue and reads the video and audio data for playback, it can efficiently maintain the synchronization of the video and audio data through the display timestamp and video timestamp, thereby guaranteeing the transmission and real-time playback of synchronized audio and video media resources from the mobile device to the display device.

[0078] In one embodiment, step S60, which involves pushing the audio queue to the display device, specifically includes the following steps:

[0079] S61: Send the audio queue to the display device via the GStreamer buffer.

[0080] The audio queue is sent to the display device via a GStreamer (a pipe-based multimedia framework) buffer.

[0081] It should be noted that in this embodiment, an audio queue is sent to the display device through a GStreamer buffer so that the display device can receive the message after receiving the audio queue. This ensures that the audio queue can be transmitted to the display device accurately and efficiently, thereby ensuring that media resources with synchronized audio and video are transmitted from the mobile device to the display device and played in real time.

[0082] In one embodiment, after step S50, i.e., after associating the audio nanosecond time with the corresponding audio data to obtain the audio queue, the method further includes the following steps:

[0083] S81: Determine whether the audio nanosecond time matches the expected time.

[0084] S82: If not, a synchronization adjustment command is sent to the display device to cause the display device to adjust the playback rate of the audio data.

[0085] It determines whether the audio nanosecond time matches the expected time. If not, it sends a synchronization adjustment command to the display device to adjust the playback rate of the audio data.

[0086] It should be noted that, in order to more accurately synchronize audio and video data, this embodiment essentially provides a method for verifying whether the audio nanosecond time meets the expected time. When the display device first plays the audio data, if the audio and video data differ significantly, it will not cause audio-video desynchronization. The main reason for audio-video desynchronization is that the error between the audio and video data increases with time. To prevent the audio nanosecond time error from increasing over time, steps S81-S82 perform a judgment process. If the audio nanosecond time does not meet the expected time, the mobile device sends a synchronization adjustment command to the display device. After receiving the synchronization adjustment command, the display device adjusts the audio data according to the audio sampling rate or audio playback speed, so that the audio and video data are played synchronously, thereby ensuring that the media resources transmitted from the mobile device to the display device and played in real time with synchronized audio and video are achieved.

[0087] In one embodiment, before step S81, i.e. before determining whether the audio nanosecond time meets the expected time, the method further includes the following steps:

[0088] S91: Multiply the sampling rate, number of channels, and number of sampling bits of the audio data to obtain an intermediate value.

[0089] S92: Divide the amount of audio data by the intermediate value to obtain the expected time.

[0090] Multiply the sampling rate, number of channels, and bit depth of the audio data to obtain an intermediate value. Then, divide the amount of audio data by the intermediate value to obtain the expected time.

[0091] It should be noted that this embodiment provides a method for calculating the expected time, which estimates the duration of the audio data based on the amount of data, thereby ensuring that media resources with synchronized audio and video are transmitted from the mobile device to the display device and played in real time.

[0092] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0093] In one embodiment, a device for controlling audio-visual synchronization is provided, which corresponds one-to-one with the method for controlling audio-visual synchronization described in the above embodiments. For example... Figure 3 As shown, the device for controlling audio-visual synchronization includes a first association module 10, an acquisition module 20, a removal module 30, a conversion module 40, a second association module 50, and a push module 60. Detailed descriptions of each functional module are as follows:

[0094] The first association module 10 is used to associate the picture data in the media resources with the corresponding picture timestamps to obtain the picture queue;

[0095] The acquisition module 20 is used to acquire the first frame data of the audio data in the media resource, obtain the first frame timestamp corresponding to the first frame data, and acquire the audio timestamp corresponding to the audio data.

[0096] The removal module 30 is used to subtract the first frame timestamp from the audio timestamp to obtain the residual timestamp;

[0097] The conversion module 40 is used to convert the residual timestamp to a unit to obtain an audio nanosecond time in nanoseconds.

[0098] The second association module 50 is used to associate the audio nanosecond time with the corresponding audio data to obtain an audio queue;

[0099] The push module 60 is used to push the screen queue and the audio queue to the display device, so that the display device can synchronously play the screen data and the audio data in real time according to the screen timestamp and the audio nanosecond time.

[0100] The conversion module 40 is specifically used for:

[0101] The residual timestamp is converted to a unit to obtain a nanosecond timestamp;

[0102] Divide the nanosecond timestamp by the sampling rate of the audio data to obtain the audio nanosecond time.

[0103] The conversion module 40 further includes:

[0104] The judgment module 70 is used to determine whether the audio data has been initialized;

[0105] The reading module 80 is used to read the audio buffer data of the device buffer if it is not initialized, and use it as the first frame data.

[0106] The second association module 50 is specifically used for:

[0107] The audio nanosecond time is decoded into a display timestamp;

[0108] The audio queue is obtained by associating the displayed timestamp with the corresponding audio data.

[0109] The push module 60 specifically includes:

[0110] The audio queue is sent to the display device via a GStreamer buffer.

[0111] The device for controlling audio-visual synchronization also includes:

[0112] The first calculation module 110 is used to multiply the sampling rate, number of channels, and sampling bit depth of the audio data to obtain an intermediate value;

[0113] The second calculation module 120 is used to divide the amount of audio data by the intermediate value to obtain the expected time.

[0114] Expectation module 90 is used to determine whether the audio nanosecond time meets the expected time;

[0115] The sending module 100 is used to send a synchronization adjustment command to the display device if the data does not match, so that the display device adjusts the playback rate of the audio data.

[0116] Specific limitations regarding the device for controlling audio-visual synchronization can be found in the limitations of the method for controlling audio-visual synchronization described above, and will not be repeated here. Each module in the aforementioned device for controlling audio-visual synchronization can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0117] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database is used for all data generated in the aforementioned method for controlling audio-visual synchronization. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for controlling audio-visual synchronization.

[0118] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0119] By associating the image data in the media resources with the corresponding image timestamps, an image queue is obtained;

[0120] Obtain the first frame of audio data from the media resource, obtain the first frame timestamp corresponding to the first frame of data, and obtain the audio timestamp corresponding to the audio data;

[0121] Subtract the first frame timestamp from the audio timestamp to obtain the residual timestamp;

[0122] The residual timestamp is converted to obtain the audio nanosecond time in nanoseconds.

[0123] The audio nanosecond time is associated with the corresponding audio data to obtain the audio queue;

[0124] The video queue and the audio queue are pushed to the display device so that the display device can synchronously play the video data and the audio data in real time based on the video timestamp and the audio nanosecond time.

[0125] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0126] By associating the image data in the media resources with the corresponding image timestamps, an image queue is obtained;

[0127] Obtain the first frame of audio data from the media resource, obtain the first frame timestamp corresponding to the first frame of data, and obtain the audio timestamp corresponding to the audio data;

[0128] Subtract the first frame timestamp from the audio timestamp to obtain the residual timestamp;

[0129] The residual timestamp is converted to obtain the audio nanosecond time in nanoseconds.

[0130] The audio nanosecond time is associated with the corresponding audio data to obtain the audio queue;

[0131] The video queue and the audio queue are pushed to the display device so that the display device can synchronously play the video data and the audio data in real time based on the video timestamp and the audio nanosecond time.

[0132] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0133] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0134] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for controlling audio-visual synchronization, characterized in that, include: By associating the image data in the media resources with the corresponding image timestamps, an image queue is obtained; Obtain the first frame of audio data from the media resource, obtain the first frame timestamp corresponding to the first frame of data, and obtain the audio timestamp corresponding to the audio data; Subtract the first frame timestamp from the audio timestamp to obtain the residual timestamp; The residual timestamp is converted to obtain the audio nanosecond time in nanoseconds. The audio nanosecond time is associated with the corresponding audio data to obtain the audio queue; The video queue and the audio queue are pushed to the display device so that the display device can synchronously play the video data and the audio data in real time according to the video timestamp and the audio nanosecond time. After associating the audio nanosecond time with the corresponding audio data to obtain the audio queue, the method further includes: Multiply the sampling rate, number of channels, and sampling bit depth of the audio data to obtain an intermediate value; Divide the amount of audio data by the intermediate value to obtain the expected time; Determine whether the audio nanosecond time matches the expected time; If the conditions are not met, a synchronization adjustment command is sent to the display device to adjust the playback rate of the audio data.

2. The method for controlling audio-visual synchronization as described in claim 1, characterized in that, The step of converting the residual timestamp to obtain the audio nanosecond time in nanoseconds includes: The residual timestamp is converted to a unit to obtain a nanosecond timestamp; Divide the nanosecond timestamp by the sampling rate of the audio data to obtain the audio nanosecond time.

3. The method for controlling audio-visual synchronization as described in claim 1, characterized in that, After obtaining the first frame data of the audio data in the media resource and obtaining the first frame timestamp corresponding to the first frame data, the method further includes: Determine whether the audio data has been initialized; If not initialized, the audio buffer data in the device buffer is read as the first frame data.

4. The method for controlling audio-visual synchronization as described in claim 1, characterized in that, The step of associating the audio nanosecond time with the corresponding audio data to obtain the audio queue includes: The audio nanosecond time is decoded into a display timestamp; The audio queue is obtained by associating the displayed timestamp with the corresponding audio data.

5. The method for controlling audio-visual synchronization as described in claim 1, characterized in that, The step of pushing the audio queue to the display device includes: The audio queue is sent to the display device via a GStreamer buffer.

6. A device for controlling audio-visual synchronization, characterized in that, include: The first association module is used to associate the image data in the media resources with the corresponding image timestamps to obtain the image queue; The acquisition module is used to acquire the first frame data of the audio data in the media resource, obtain the first frame timestamp corresponding to the first frame data, and acquire the audio timestamp corresponding to the audio data. The removal module is used to subtract the first frame timestamp from the audio timestamp to obtain the residual timestamp; The conversion module is used to convert the residual timestamp to a unit to obtain the audio nanosecond time in nanoseconds. The second association module is used to associate the audio nanosecond time with the corresponding audio data to obtain the audio queue; The push module is used to push the screen queue and the audio queue to the display device, so that the display device can synchronously play the screen data and the audio data in real time according to the screen timestamp and the audio nanosecond time. After associating the audio nanosecond time with the corresponding audio data to obtain the audio queue, the method further includes: Multiply the sampling rate, number of channels, and sampling bit depth of the audio data to obtain an intermediate value; Divide the amount of audio data by the intermediate value to obtain the expected time; Determine whether the audio nanosecond time matches the expected time; If the conditions are not met, a synchronization adjustment command is sent to the display device to adjust the playback rate of the audio data.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method for controlling audio-visual synchronization as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for controlling audio-visual synchronization as described in any one of claims 1 to 5.

Images