A method, device and storage medium for processing network video conference data

Abstract

The present application discloses a method, device, and storage medium for processing network video conference data. The method includes: obtaining an audio data stream and a video data stream; detecting the frame time and buffer queue length of the audio data stream and the video data stream; selecting either the audio data stream or the video data stream as a reference source according to the detection result and a preset synchronization strategy; calculating the time gap between the audio data stream and the video data stream, and determining whether the time gap exceeds a preset first threshold; if the time gap exceeds the first threshold, forcibly synchronize the clocks of the OPS computer, audio DSP, and video DSP; if the time gap does not exceed the first threshold, further determine whether the time gap exceeds a second threshold. If the time gap exceeds the second threshold, perform frame dropping or re-display processing according to the time gap; if the time gap does not exceed the second threshold, output the current audio data stream and video data stream. The aim is to ensure the accuracy of audio-visual synchronization.

Description

Technical Field

[0001] This application relates to the technical field of audio - video processing, and particularly to a method, device, and storage medium for processing network video conference data. Background Art

[0002] In a network video conference system, the synchronization of audio and video is a core element to ensure the user experience. Audio and video data are usually obtained through independent acquisition devices and transmitted to a central processing system for real - time processing at different frame rates and sampling rates.

[0003] However, due to the physical characteristics of the acquisition devices, the uncertainty of network transmission, and the timing deviation during the decoding process, there are often differences in the timestamps between audio and video frames, which may lead to the problem of out - of - sync audio and video. Summary of the Invention

[0004] The purpose of the embodiments of this application is to propose a method, device, and storage medium for processing network video conference data to solve the technical problems of out - of - sync or low synchronization accuracy of audio and video in network video conferences.

[0005] To solve the above - mentioned technical problems, the embodiments of this application provide a method for processing network video conference data, which adopts the following technical solutions:

[0006] A method for processing network video conference data includes the following steps:

[0007] Obtain an audio data stream and a video data stream;

[0008] Detect the frame time and buffer queue length of the audio data stream and the video data stream;

[0009] Select the audio data stream or the video data stream as a reference source according to the detection result and a preset synchronization strategy;

[0010] Calculate the time gap between the audio data stream and the video data stream, and determine whether the time gap exceeds a preset first threshold;

[0011] If the time gap exceeds the first threshold, forcibly synchronize the clocks of the OPS computer, audio DSP, and video DSP;

[0012] If the time gap does not exceed the first threshold, further determine whether the time gap exceeds a second threshold.

[0013] If the time gap exceeds the second threshold, perform frame dropping or re - display processing according to the time gap;

[0014] If the time gap does not exceed the second threshold, output the current audio data stream and video data stream.

[0015] In a possible implementation manner, the step of selecting an audio data stream or a video data stream as a reference source according to the detection result and a preset synchronization policy specifically includes:

[0016] Calculating the timestamp difference of the frame times of the audio data stream and the video data stream;

[0017] Comparing the queue length difference between the audio data stream and the video data stream;

[0018] Forming a stability scoring criterion according to the timestamp difference and the queue length difference;

[0019] Selecting a reference source according to the stability scoring criterion.

[0020] In a possible implementation manner, the step of forming a stability scoring criterion according to the timestamp difference and the queue length difference specifically includes:

[0021] Performing normalization processing on the timestamp difference and the queue length difference;

[0022] Assigning weights according to the influence degrees of the timestamp difference and the queue length difference;

[0023] Calculating a stability score.

[0024] In a possible implementation manner, the step of forcibly synchronizing the clocks of the OPS computer, the audio DSP, and the video DSP if the time gap exceeds a first threshold specifically includes:

[0025] Obtaining the current time of the OPS computer;

[0026] Sending the current time of the OPS computer to the audio DSP and the video DSP respectively;

[0027] Updating the clocks of the audio DSP and the video DSP;

[0028] Verifying whether the clocks of the audio DSP and the video DSP are consistent with the OPS computer.

[0029] In a possible implementation manner, in the step of performing frame dropping or re-display processing according to the time gap if the time gap exceeds a second threshold:

[0030] If the audio data stream is faster than the video data stream, performing frame dropping processing;

[0031] If the audio data stream is slower than the video data stream, performing re-display processing.

[0032] In a possible implementation manner, before the step of obtaining the audio data stream and the video data stream, further included:

[0033] Initialize the audio DSP, video DSP, and OPS computer, set the audio sampling rate and video frame rate, and establish a communication channel between the DSP and the OPS.

[0034] In a possible implementation, after the step of obtaining the audio data stream and the video data stream, the method further includes:

[0035] Perform echo cancellation, noise suppression, and mixing matrix processing on the audio data stream;

[0036] Decode the video data stream, add timestamps, enhance the video frames, and perform matrix stitching.

[0037] To solve the above technical problems, an embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0038] A computer device includes a memory and a processor. Computer-readable instructions are stored in the memory, and when the processor executes the computer-readable instructions, the steps of the network video conference data processing method described above are implemented.

[0039] To solve the above technical problems, an embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0040] A computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the network video conference data processing method described above are implemented.

[0041] Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects:

[0042] The network video conferencing data processing method disclosed in this application obtains audio data streams and video data streams; detects the frame time and buffer queue length of the audio data stream and the video data stream; selects the audio data stream or the video data stream as the reference source according to the detection results and a preset synchronization strategy; calculates the time gap between the audio data stream and the video data stream, and determines whether the time gap exceeds a preset first threshold; if the time gap exceeds the first threshold, forcibly synchronize the clocks of the OPS computer, the audio DSP, and the video DSP; if the time gap does not exceed the first threshold, further determine whether the time gap exceeds a second threshold, and if the time gap exceeds the second threshold, perform frame dropping or re-display processing according to the time gap; if the time gap does not exceed the second threshold, output the current audio data stream and video data stream. By detecting the timestamp difference and queue length difference between the audio and video data streams and combining the preset synchronization strategy to select the reference source and calculate the time gap, this application enables the system to efficiently determine whether clock synchronization or frame dropping / re-display processing is required, thereby ensuring the synchronization of the audio and video streams. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] To more clearly illustrate the solutions in this application, the following briefly introduces the drawings required for the description of the embodiments of this application. Obviously, the following described drawings are some embodiments of this application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0044] Figure 1 is an exemplary system architecture diagram to which this application can be applied;

[0045] Figure 2 is a flowchart of an embodiment of the network video conferencing data processing method according to this application;

[0046] Figure 3 is a schematic structural diagram of an embodiment of a computer device according to this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the art belonging to the technical field of this application; the terms used in the specification of this application are only for the purpose of describing specific embodiments, and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above drawings are intended to cover non-exclusive inclusion. The terms "first", "second", etc. in the specification and claims of this application or the above drawings are used to distinguish different objects, not to describe a specific order.

[0048] References to "embodiments" in this specification mean that particular features, structures, or characteristics described in connection with the embodiments can be included in at least one embodiment of the present application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0049] To enable those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0050] As Figure 1 shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0051] Users can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0052] The terminal devices 101, 102, 103 may be various electronic devices with a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, and desktop computers, etc.

[0053] The server 105 may be a server providing various services, such as a background server supporting the pages displayed on the terminal devices 101, 102, 103.

[0054] It should be noted that the network video conference data processing method provided by the embodiments of the present application is generally executed by the server, and correspondingly, Topic Two is generally set in the server.

[0055] It should be understood, Figure 1The numbers of the terminal devices, networks, and servers in [it] are merely illustrative. According to the implementation requirements, there can be any number of terminal devices, networks, and servers.

[0056] Continuing to refer to Figure 2 , a flowchart of an embodiment of a network video conference data processing method according to the present application is shown. The network video conference data processing method includes the following steps:

[0057] Step S201, obtain an audio data stream and a video data stream.

[0058] In this embodiment, the electronic device (such as Figure 1 the server shown) on which the network video conference data processing method runs can send or receive data through a wired connection method or a wireless connection method. It should be noted that the above wireless connection methods can include but are not limited to 3G / 4G / 5G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra wideband) connections, and other currently known or future-developed wireless connection methods.

[0059] In this embodiment, first, it is necessary to obtain audio and video data streams from sources such as conference devices or webcams. These data streams contain information related to audio signals and video images, usually compressed data (such as the audio stream may be in the AAC format, and the video stream is in the H.264 format, etc.).

[0060] Step S202, detect the frame time and buffer queue length of the audio data stream and the video data stream.

[0061] In this embodiment, the frame time refers to the timestamp of each frame of audio or video data, indicating the time when the frame data is collected. The buffer queue length refers to the size or data volume of the buffer storing audio or video data, which affects the real-time processing ability of the data. The goal of this step is to obtain the time information and queue status of the streaming media for making decisions during subsequent synchronization.

[0062] Step S203, select the audio data stream or the video data stream as the reference source according to the detection result and a preset synchronization strategy.

[0063] In this embodiment, based on the frame time difference and queue length difference detected in step 202, the system determines the audio stream or the video stream as the reference source according to the set synchronization strategy. The synchronization strategy can define the following rules:

[0064] Prioritize data streams with high stability: If the frame time and buffer queue length of the audio data stream fluctuate less, then prioritize audio as the reference source. If the video frame rate is stable and the buffer queue length is uniform, then prioritize video as the reference source.

[0065] Consider device characteristics: If the clock reference of the audio DSP or video DSP is more accurate, then prioritize the corresponding data stream as the reference source.

[0066] Consider actual application requirements: For conference scenarios dominated by voice, prioritize the audio data stream; for scenarios dominated by video, prioritize the video data stream.

[0067] Step S204, calculate the time gap between the audio data stream and the video data stream, and determine whether the time gap exceeds a preset first threshold;

[0068] If the time gap exceeds the first threshold, then forcibly synchronize the clocks of the OPS computer, audio DSP, and video DSP;

[0069] If the time gap does not exceed the first threshold, further determine whether the time gap exceeds a second threshold.

[0070] If the time gap exceeds the second threshold, then perform frame dropping or re-display processing according to the time gap;

[0071] If the time gap does not exceed the second threshold, then output the current audio data stream and video data stream.

[0072] In this embodiment, calculate the difference (i.e., time gap) between the timestamps of two data streams (audio and video), and then compare it with a preset first threshold. If the gap is large (e.g., exceeds a certain allowable range), it may affect the synchronization accuracy and cause out-of-sync sound and images. If the time gap exceeds the first threshold, it indicates that there is a large deviation in synchronization and clock synchronization needs to be forced. In the case where the time gap exceeds the first threshold, the system will forcibly perform clock synchronization. The OPS (Operating System Processing Unit) computer serves as the master clock and ensures audio-video synchronization by controlling the audio and video DSPs (Digital Signal Processors). Specifically, the OPS computer will send its current time to the audio and video DSPs to ensure that their clocks are updated and consistent.

[0073] If the time gap is small (not exceeding the first threshold), the system will further detect whether the gap exceeds a second, smaller threshold (the second threshold). This means that complex processing such as frame dropping or redisplay will only be performed in cases of relatively severe time differences. If the time gap exceeds the second threshold, the system will take measures such as frame dropping or redisplay to adjust the synchronization. For example, if the audio stream is faster than the video stream, the system will discard some frames in the audio stream to prevent the audio from getting ahead. If the audio stream is slower than the video stream, the system will repeat the display of video frames so that the video can be synchronized with the audio. If the time gap is small and does not exceed the second threshold, the system directly outputs the audio and video streams, leaving them unchanged without additional processing.

[0074] This application detects the timestamp difference and queue length difference between the audio and video data streams, and combines the preset synchronization strategy to select a reference source and calculate the time gap, which enables the system to efficiently determine whether clock synchronization or frame dropping / redisplay processing is required, thus ensuring the synchronization of the audio and video streams.

[0075] In some optional implementation manners of this embodiment, the step of selecting the audio data stream or the video data stream as the reference source according to the detection result and the preset synchronization strategy specifically includes:

[0076] Calculate the timestamp difference of the frame times of the audio data stream and the video data stream;

[0077] Compare the queue length differences between the audio data stream and the video data stream;

[0078] Form a stability scoring criterion based on the timestamp difference and the queue length difference;

[0079] Select a reference source according to the stability scoring criterion.

[0080] In this embodiment, calculating the difference in timestamps between the audio and video streams, that is, comparing the timestamps of each frame of audio and video to determine the degree of out-of-synchronization between them. By comparing the lengths of the audio and video queues, it is judged which side has more buffered data, which helps to determine which stream is more suitable as the reference source. Taking the timestamp difference and the queue length difference as inputs, a stability scoring criterion is formed. The scoring criterion may be based on the weighted sum of these two factors to evaluate the stability of each stream, and then decide which stream to select as the reference. According to the result of the scoring criterion, the data stream with higher stability is selected as the reference source. This can ensure the accuracy and reliability of the synchronization.

[0081] Specifically, characteristics such as the frame time, buffer queue length, and sampling rate of the audio data stream are detected and recorded; characteristics such as the frame time, frame rate, and buffer queue length of the video data stream are detected and recorded; the integrity and stability of the data stream are identified, such as whether there are dropped frames or latency phenomena. Based on the standard deviation of the frame time fluctuation and the buffer queue length, the stability scores of the audio and video are calculated respectively. If the stability score of the audio is higher than that of the video, the audio data stream is selected as the reference source; otherwise, the video data stream is selected. During the real-time synchronization process, if it is detected that the stability of the current reference source decreases (such as the fluctuation exceeds the threshold), another data stream can be switched to as the reference source.

[0082] In this application, by combining the timestamp difference and the queue length difference of the audio and video data streams to form a stability scoring criterion, the audio or video data stream can be more precisely selected as the reference source in various situations. This measure improves the intelligence of the system in selecting the reference source, further enhances the stability and processing ability of the audio and video stream synchronization, and ensures that the audio and video streams can be processed efficiently and accurately in complex environments.

[0083] In some optional implementation manners of this embodiment, the step of forming the stability scoring criterion according to the timestamp difference and the queue length difference specifically includes:

[0084] Normalize the timestamp difference and the queue length difference;

[0085] Allocate weights according to the influence degrees of the timestamp difference and the queue length difference;

[0086] Calculate the stability score.

[0087] In this embodiment, the scoring criterion is based on the following two key indicators: Timestamp difference: The timestamp deviation of the audio and video data streams, reflecting the degree of synchronization offset. Queue length difference: The difference in the amount of data in the audio and video buffer queues, reflecting the backlog or shortage of the data stream. Map these two indicators to scoring values to evaluate the stability of each data stream.

[0088] In order to make the scores of different indicators comparable, normalization processing is required.

[0089] Timestamp difference normalization calculation formula: Timestamp difference value / Timestamp difference threshold, where the timestamp difference value is the absolute difference between the current frame timestamps of the audio and video; the timestamp difference threshold is the set maximum allowable timestamp deviation value. When it indicates that the timestamp difference exceeds the acceptable range.

[0090] Queue length difference normalization calculation formula: Queue length difference value / queue length difference threshold, where the queue length difference value is the difference in the lengths of the audio buffer queue and the video buffer queue (unit: frames or time); the queue length difference threshold is the set maximum allowable difference in queue lengths. When it indicates that the queue length difference exceeds the acceptable range.

[0091] The influencing degrees of the timestamp difference and the queue length difference may be different, so weights need to be assigned to the two metrics. Set weights and , satisfying: .

[0092] Based on the scores of the combined timestamp difference and queue length difference, calculate the stability score :

[0093] ,

[0094] When is close to 1, it indicates that the data stream is more stable; when is close to 0, it indicates that the data stream is unstable. According to the stability score results, select the reference source: compare the stability scores of the audio data stream and the video data stream, and the data stream with the higher stability score is selected as the reference source.

[0095] This application calculates the stability score by normalizing the timestamp difference and the queue length difference and assigning weights according to their influencing degrees, which enables the system to more accurately evaluate the stability of the audio - video stream. The overall effect of this step is to further improve the accuracy of the synchronization strategy, ensure that the system can adjust the synchronization strategy according to the actual stability of the data stream, and thus minimize the occurrence of out - of - sync phenomena.

[0096] In some alternative implementation manners of this embodiment, the step of forcibly synchronizing the clocks of the OPS computer, the audio DSP, and the video DSP if the time difference exceeds the first threshold specifically includes:

[0097] Obtain the current time of the OPS computer;

[0098] Send the current time of the OPS computer to the audio DSP and the video DSP respectively;

[0099] Update the clocks of the audio DSP and the video DSP;

[0100] Verify whether the clocks of the audio DSP and the video DSP are consistent with the OPS computer.

[0101] In this embodiment, when clock synchronization is required, the OPS computer obtains the current time, which will be used as a reference benchmark. The OPS computer sends its current time to the audio and video DSPs to ensure that their clocks are synchronized. After receiving the OPS time, the audio and video DSPs update their local clocks to be consistent with the OPS computer. Finally, the system checks whether the clocks of the audio and video DSPs have been successfully synchronized to the clock of the OPS computer. If they are consistent, it indicates that the clock synchronization is completed.

[0102] This application ensures the consistency of the clocks of these devices by forcibly synchronizing the clocks of the OPS computer, the audio DSP, and the video DSP. This measure effectively avoids audio-video synchronization problems caused by inconsistent clocks between devices, enhances the stability of the system and the clock calibration accuracy, thereby improving the accuracy and stability of audio-video synchronization.

[0103] In some alternative implementation manners of this embodiment, in the above step of performing frame dropping or re-display processing according to the time difference if the time difference exceeds the second threshold:

[0104] If the audio data stream is faster than the video data stream, frame dropping processing is performed;

[0105] If the audio data stream is slower than the video data stream, re-display processing is performed.

[0106] In this embodiment, if the processing speed of the audio stream is faster than that of the video stream, resulting in the audio being ahead, the system will discard some audio frames to ensure audio-video synchronization. If the processing speed of the audio stream is slower than that of the video stream, the system will repeat playing the video frames until the video and audio are synchronized.

[0107] Through frame dropping or re-display processing, this application enables the system to dynamically adjust according to the relative speeds of the audio and video streams. This measure effectively addresses the time differences that occur during audio-video synchronization, further ensuring the smoothness and consistency of the audio and video streams, and preventing frame freezes or audio-video out-of-sync phenomena caused by time differences.

[0108] In some alternative implementation manners of this embodiment, before the above step of obtaining the audio data stream and the video data stream, the following steps are further included:

[0109] Initialize the audio DSP, the video DSP, and the OPS computer, set the audio sampling rate and the video frame rate, and establish a communication channel between the DSP and the OPS.

[0110] In this embodiment, before starting the data stream processing, the system first performs initialization to ensure that the sampling rates and frame rates of the audio and video DSPs are correctly set and establish a reliable communication channel to ensure the smooth transmission of the data stream.

[0111] This application initializes the audio DSP, video DSP, and OPS computer, sets the audio sampling rate, video frame rate, and establishes a communication channel. This measure ensures that the system has the basic parameters and a stable communication channel required to process audio and video data streams from the very beginning, reducing configuration errors or data stream transmission delays that may occur during the initialization process, thereby improving the overall efficiency and stability of the system.

[0112] In some alternative implementation manners of this embodiment, after the steps of obtaining the audio data stream and the video data stream, the following steps are further included:

[0113] Perform echo cancellation, noise suppression, and mixing matrix processing on the audio data stream;

[0114] Decode the video data stream, add timestamps, enhance the picture, and perform matrix splicing.

[0115] In this embodiment, the audio data is processed to remove echoes, suppress background noise, and at the same time perform mixing processing to ensure clear sound quality. The video stream is decoded, timestamps are added for synchronous processing, the picture is enhanced to improve the visual effect, and if there are multiple picture inputs, matrix splicing is performed.

[0116] This application, after obtaining the audio and video data streams, performs echo cancellation, noise suppression, and mixing matrix processing on the audio stream, and decodes the video stream, adds timestamps, enhances the picture, and performs matrix splicing. This stage optimizes the processing of the audio and video streams, ensures the quality of the audio and video data streams, reduces interference factors such as noise and echoes, and improves the clarity and details of the video picture, thereby improving the audio-visual experience of the end user.

[0117] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results.

[0118] Artificial intelligence basic technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation / interaction systems, and mechatronics. Artificial intelligence software technologies mainly include several major directions such as computer vision technology, robotics, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0119] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0120] It should be understood that although the steps in the flowchart of the accompanying drawings are shown in sequence according to the indication of the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order restriction, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. Their execution order is not necessarily sequential, but can be executed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

[0121] To solve the above technical problems, the embodiments of the present application also provide computer devices. For details, please refer to Figure 3 , Figure 3 which is the basic structural block diagram of the computer device in this embodiment.

[0122] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are communicatively connected to each other through a system bus. It should be noted that only the computer device 4 with components 41-43 is shown in the figure, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Among them, those skilled in the art of the present technology can understand that a computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0123] The computer device may be a computing device such as a desktop computer, a notebook, a palm computer, or a cloud server. The computer device may interact with a user through a keyboard, a mouse, a remote control, a touchpad, or a voice control device.

[0124] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, a hard disk, a multimedia card, a card-type memory (such as an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as a hard disk or a memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a FlashCard, etc. equipped on the computer device 4. Of course, the memory 41 may also include both the internal storage unit and the external storage device of the computer device 4. In this embodiment, the memory 41 is generally used to store an operating system installed on the computer device 4 and various application software, such as computer-readable instructions for a network video conferencing data processing method. In addition, the memory 41 may also be used to temporarily store various data that have been output or will be output.

[0125] In some embodiments, the processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the computer-readable instructions stored in the memory 41 or process data, such as running the computer-readable instructions for the network video conferencing data processing method.

[0126] The network interface 43 may include a wireless network interface or a wired network interface, and the network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0127] The computer device provided by this application can detect the timestamp difference and queue length difference of audio and video data streams, and select a reference source and calculate the time gap in combination with a preset synchronization strategy, which enables the system to efficiently determine whether clock synchronization or frame dropping / redisplay processing is required, thereby ensuring the synchronization of audio and video streams.

[0128] The present application also provides another implementation manner, that is, to provide a computer-readable storage medium storing computer-readable instructions, which can be executed by at least one processor to cause the at least one processor to execute the steps of the network video conference data processing method as described above.

[0129] The computer-readable storage medium provided by the present application detects the timestamp difference and queue length difference of audio and video data streams, and combines a preset synchronization strategy to select a reference source and calculate the time difference, which enables the system to efficiently determine whether clock synchronization or frame dropping / re-display processing is required, thereby ensuring the synchronization of audio and video streams.

[0130] Through the description of the above implementation manners, those skilled in the art can clearly understand that the above-described example methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation manner. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc), and includes several instructions to cause a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in various embodiments of the present application.

[0131] Obviously, the above-described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The accompanying drawings show the preferred embodiments of the present application, but do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure content of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, for those skilled in the art, they can still modify the technical solutions described in the foregoing specific implementation manners, or perform equivalent replacements on some of the technical features. Any equivalent structure directly or indirectly using the content of the specification and drawings of the present application in other related technical fields is equally within the scope of the patent protection of the present application.

Claims

1. A network video conference data processing method, characterized in that: The steps include: Get audio data stream and video data stream; Detecting the frame time and buffer queue length of the audio data stream and the video data stream; Selecting an audio data stream and a video data stream as reference sources according to the detection result and a preset synchronization strategy; Calculating a timestamp difference between the audio data stream and the video data stream, and determining whether the timestamp difference exceeds a preset first threshold; If the timestamp difference exceeds a first threshold, the clocks of the OPS computer, the audio DSP, and the video DSP are forced to be synchronized; If the timestamp difference does not exceed the first threshold, further determine whether the timestamp difference exceeds the second threshold. If the timestamp difference exceeds the second threshold, performing frame dropping or re-display processing according to the timestamp difference; If the timestamp difference does not exceed the second threshold, output the current audio data stream and video data stream; The step of forcibly synchronizing the clocks of the OPS computer, the audio DSP, and the video DSP if the timestamp difference exceeds the first threshold specifically includes: Get the current time of the OPS computer; Send the current time of the OPS computer to the audio DSP and the video DSP respectively; Updating the clocks of the audio DSP and the video DSP; Verify that the clocks of the audio DSP and video DSP are consistent with the OPS computer; The step of selecting the audio data stream and the video data stream as reference sources according to the detection result and the preset synchronization strategy specifically includes: Calculate the timestamp difference between the frame times of the audio data stream and the video data stream; Compare the queue length differences between audio data streams and video data streams; forming a stability score based on the timestamp difference and the queue length difference; A reference source is selected based on the stability score.

2. The network video conference data processing method according to claim 1, characterized in that: The step of forming a stability score according to the timestamp difference and the queue length difference specifically includes: Normalizing the timestamp difference and the queue length difference; According to the influence of the timestamp difference and the queue length difference on the stability score, weights are respectively assigned to the normalized timestamp difference and the normalized queue length difference; The stability score is calculated based on the normalized timestamp difference, the normalized queue length difference, the weight of the normalized timestamp difference, and the weight of the normalized queue length difference.

3. The network video conference data processing method according to claim 1, characterized in that: If the timestamp difference exceeds the second threshold, in the step of performing frame dropping or re-display processing according to the timestamp difference: If the audio data stream is faster than the video data stream, frame dropping processing is performed; If the audio data stream is slower than the video data stream, re-display processing is performed.

4. The network video conference data processing method according to claim 1, characterized in that: After the step of obtaining the audio data stream and the video data stream, the method further includes: performing echo cancellation and noise suppression on the audio data stream; The video data stream is decoded, time stamped and image enhanced.

5. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the network video conference data processing method as claimed in any one of claims 1 to 4 when executing the computer-readable instructions.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the network video conference data processing method according to any one of claims 1 to 4 are implemented.

Images