Network video conference data processing method and device, and storage medium

Abstract

The invention discloses a network video conference data processing method and device and a storage medium, and the method comprises the steps: obtaining an audio data stream and a video data stream; detecting frame time and buffer queue length of the audio data stream and the video data stream; selecting an audio data stream or a video data stream as a reference source according to a detection result and a preset synchronization strategy; calculating a time difference between the audio data stream and the video data stream, and judging whether the time difference exceeds a preset first threshold value or not; if the time difference exceeds a first threshold value, forcibly synchronizing the clocks of the OPS computer, the audio DSP and the video DSP; if the time difference does not exceed the first threshold value, further judging whether the time difference exceeds a second threshold value, and if the time difference exceeds the second threshold value, performing frame loss or re-display processing according to the time difference; and if the time difference does not exceed the second threshold value, outputting the current audio data stream and the current video data stream. The invention aims to ensure the accuracy of audio and video synchronization.

Description

Technical Field

[0001] The present application relates to the field of audio and video processing technology, and in particular to a network video conference data processing method, device and storage medium. Background Art

[0002] In a network video conferencing system, the synchronization of audio and video is the core element to ensure user experience. Audio and video data are usually acquired through independent acquisition devices and transmitted to the central processing system at different frame rates and sampling rates for real-time processing.

[0003] However, due to the physical characteristics of the acquisition equipment, the uncertainty of network transmission and the timing deviation in the decoding process, there are often differences in the timestamps between audio and video frames, which may cause the audio and video to be out of sync. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a network video conferencing data processing method, device and storage medium to solve the technical problem of audio and video being out of sync or having low synchronization accuracy in network video conferencing.

[0005] In order to solve the above technical problems, the present application embodiment provides a network video conference data processing method, which adopts the following technical solution: A network video conference data processing method comprises the following steps: Get audio data stream and video data stream; Detecting the frame time and buffer queue length of the audio data stream and the video data stream; Selecting an audio data stream or a video data stream as a reference source according to the detection result and a preset synchronization strategy; Calculating a time difference between the audio data stream and the video data stream, and determining whether the time difference exceeds a preset first threshold; If the time difference exceeds the first threshold, the clocks of the OPS computer, the audio DSP, and the video DSP are forced to be synchronized; If the time difference does not exceed the first threshold, further determine whether the time difference exceeds the second threshold. If the time difference exceeds the second threshold, performing frame dropping or re-display processing according to the time difference; If the time difference does not exceed the second threshold, the current audio data stream and video data stream are output.

[0006] In a possible implementation manner, the step of selecting an audio data stream or a video data stream as a reference source according to the detection result and a preset synchronization strategy specifically includes: Calculate the timestamp difference between the frame times of the audio data stream and the video data stream; Compare the queue length differences between audio data streams and video data streams; forming a stability scoring criterion according to the timestamp difference and the queue length difference; The reference source is selected according to the stability scoring criteria.

[0007] In a possible implementation manner, the step of forming a stability scoring standard according to the timestamp difference and the queue length difference specifically includes: Normalizing the timestamp difference and the queue length difference; Allocating weights according to the influence of the timestamp difference and the queue length difference; Calculate the stability score.

[0008] In a possible implementation, if the time difference exceeds the first threshold, the step of forcibly synchronizing the clocks of the OPS computer, the audio DSP, and the video DSP specifically includes: Get the current time of the OPS computer; Send the current time of the OPS computer to the audio DSP and the video DSP respectively; Updating the clocks of the audio DSP and the video DSP; Verify that the clocks of the audio DSP and video DSP are consistent with the OPS computer.

[0009] In a possible implementation manner, in the step of performing frame dropping or re-display processing according to the time difference if the time difference exceeds the second threshold: If the audio data stream is faster than the video data stream, frame dropping processing is performed; If the audio data stream is slower than the video data stream, re-display processing is performed.

[0010] In a possible implementation manner, before the step of obtaining the audio data stream and the video data stream, the method further includes: Initialize the audio DSP, video DSP and OPS computer, set the audio sampling rate, video frame rate, and establish a communication channel between DSP and OPS.

[0011] In a possible implementation manner, after the step of obtaining the audio data stream and the video data stream, the method further includes: Performing echo cancellation, noise suppression and mixing matrix processing on the audio data stream; The video data stream is decoded, time stamped, image enhanced and matrix spliced.

[0012] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution: A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the network video conference data processing method as described above when executing the computer-readable instructions.

[0013] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution: A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the network video conference data processing method as described above.

[0014] Compared with the prior art, the embodiments of the present application have the following beneficial effects: The network video conference data processing method disclosed in the present application obtains an audio data stream and a video data stream; detects the frame time and buffer queue length of the audio data stream and the video data stream; selects an audio data stream or a video data stream as a reference source according to the detection result and a preset synchronization strategy; calculates the time difference between the audio data stream and the video data stream, and determines whether the time difference exceeds a preset first threshold; if the time difference exceeds the first threshold, the clocks of the OPS computer, the audio DSP, and the video DSP are forced to be synchronized; if the time difference does not exceed the first threshold, further determine whether the time difference exceeds a second threshold, and if the time difference exceeds the second threshold, perform frame drop or re-display processing according to the time difference; if the time difference does not exceed the second threshold, output the current audio data stream and the video data stream. The present application selects a reference source and calculates the time difference by detecting the timestamp difference and the queue length difference of the audio and video data streams, combined with a preset synchronization strategy, which enables the system to efficiently determine whether clock synchronization or frame drop / re-display processing is required, thereby ensuring the synchronization of the audio and video streams. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 is an exemplary system architecture diagram to which the present application may be applied; Figure 2 is a flow chart of an embodiment of a method for processing network video conference data according to the present application; Figure 3It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0017] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0018] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0019] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0020] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0021] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0022] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0023] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0024] It should be noted that the network video conference data processing method provided in the embodiment of the present application is generally executed by a server, and accordingly, Topic 2 is generally set in the server.

[0025] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0026] Continue to refer Figure 2 , shows a flow chart of an embodiment of a network video conference data processing method according to the present application. The network video conference data processing method comprises the following steps: Step S201, obtaining audio data stream and video data stream.

[0027] In this embodiment, the network video conference data processing method is executed on the electronic device (eg Figure 1 The server shown in the figure) can send or receive data through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0028] In this embodiment, it is first necessary to obtain audio and video data streams from sources such as conference equipment or network cameras. These data streams contain relevant information of audio signals and video images, usually compressed data (such as audio streams may be in AAC format, and video streams may be in H.264 format, etc.).

[0029] Step S202: Detect the frame time and buffer queue length of the audio data stream and the video data stream.

[0030] In this embodiment, the frame time refers to the timestamp of each frame of audio or video data, indicating the time when the frame data was collected. The buffer queue length refers to the size or amount of data in the buffer storing audio or video data, which affects the real-time processing capability of data. The goal of this step is to obtain the time information and queue status of the streaming media so as to make decisions during subsequent synchronization.

[0031] Step S203: Select an audio data stream or a video data stream as a reference source according to the detection result and a preset synchronization strategy.

[0032] In this embodiment, according to the frame time difference and queue length difference detected in step 202, the system determines the audio stream or the video stream as the reference source according to the set synchronization strategy. The synchronization strategy can define the following rules: Prioritize data streams with high stability: If the frame time and buffer queue length of the audio data stream fluctuate little, audio is preferred as the reference source. If the video frame rate is stable and the buffer queue length is uniform, video is preferred as the reference source.

[0033] Consider device characteristics: If the clock reference of the audio DSP or video DSP is more accurate, the corresponding data stream is selected as the reference source.

[0034] Consider actual application needs: For conference scenarios dominated by voice, audio data streams are preferred; for scenarios dominated by video, video data streams are preferred.

[0035] Step S204, calculating the time difference between the audio data stream and the video data stream, and determining whether the time difference exceeds a preset first threshold; If the time difference exceeds the first threshold, the clocks of the OPS computer, the audio DSP, and the video DSP are forced to be synchronized; If the time difference does not exceed the first threshold, further determine whether the time difference exceeds the second threshold. If the time difference exceeds the second threshold, performing frame dropping or re-display processing according to the time difference; If the time difference does not exceed the second threshold, the current audio data stream and video data stream are output.

[0036] In this embodiment, the difference (i.e., time gap) between the timestamps of the two data streams (audio and video) is calculated and then compared with a preset first threshold. If the gap is large (for example, exceeding a certain allowable range), it may affect the synchronization accuracy and cause the sound and image to be out of sync. If the time gap exceeds the first threshold, it means that there is a large deviation in the synchronization and the clock needs to be forced to synchronize. In the case where the time gap exceeds the first threshold, the system will force clock synchronization. The OPS (operating system processing unit) computer acts as the master clock and ensures audio and video synchronization by controlling the audio and video DSPs (digital signal processors). Specifically, the OPS computer sends its current time to the audio and video DSPs to ensure that their clocks are updated and consistent.

[0037] If the time difference is small (does not exceed the first threshold), the system will further detect whether the difference exceeds the second smaller threshold (the second threshold). This means that complex processing such as frame dropping or re-display will only be performed in the case of more serious time differences. If the time difference exceeds the second threshold, the system will take measures such as frame dropping or re-display to adjust the synchronization. For example: if the audio stream is faster than the video stream, the system will discard some frames in the audio stream to avoid the audio being ahead. If the audio stream is slower than the video stream, the system will repeat the video frames so that the video can be synchronized with the audio. If the time difference is small and does not exceed the second threshold, the system directly outputs the audio and video streams, keeping the output as it is, without additional processing.

[0038] This application detects the timestamp differences and queue length differences between the audio and video data streams, and combines them with a preset synchronization strategy to select a reference source and calculate the time difference. This enables the system to efficiently determine whether clock synchronization or frame loss / re-display processing is required, thereby ensuring the synchronization of the audio and video streams.

[0039] In some optional implementations of this embodiment, the step of selecting the audio data stream or the video data stream as the reference source according to the detection result and the preset synchronization strategy specifically includes: Calculate the timestamp difference between the frame times of the audio data stream and the video data stream; Compare the queue length differences between audio data streams and video data streams; forming a stability scoring criterion according to the timestamp difference and the queue length difference; The reference source is selected according to the stability scoring criteria.

[0040] In this embodiment, the difference in timestamps between the audio stream and the video stream is calculated, that is, the timestamps of each frame of the audio and video are compared to determine the degree of asynchrony between them. By comparing the lengths of the audio and video queues, it is determined which side has more buffered data, which helps to determine which stream is more suitable as a reference source. The timestamp difference and the queue length difference are used as input to form a stability scoring standard. The scoring standard may be based on the weighted sum of these two factors to evaluate the stability of each stream and then decide which stream to select as the benchmark. Based on the results of the scoring standard, a data stream with higher stability is selected as the reference source. This ensures the accuracy and reliability of synchronization.

[0041] Specifically, detect and record the frame time, buffer queue length, sampling rate and other characteristics of the audio data stream; detect and record the frame time, frame rate, buffer queue length and other characteristics of the video data stream; identify the integrity and stability of the data stream, such as whether there is frame loss or delay. Based on the standard deviation of frame time fluctuations and buffer queue lengths, calculate the stability scores of audio and video respectively. If the stability score of audio is higher than that of video, select the audio data stream as the reference source; otherwise, select the video data stream. During real-time synchronization, if it is detected that the stability of the current reference source has decreased (such as fluctuations exceeding the threshold), you can switch to another data stream as the reference source.

[0042] This application forms a stability scoring standard by combining the timestamp difference and queue length difference of audio and video data streams, which can more accurately select audio or video data streams as reference sources in many situations. This measure improves the intelligence of the system in selecting reference sources, further enhances the stability and processing capabilities of audio and video stream synchronization, and ensures that audio and video streams can still be processed efficiently and accurately in complex environments.

[0043] In some optional implementations of this embodiment, the step of forming a stability scoring standard according to the timestamp difference and the queue length difference specifically includes: Normalizing the timestamp difference and the queue length difference; Allocating weights according to the influence of the timestamp difference and the queue length difference; Calculate the stability score.

[0044] In this embodiment, the scoring criteria are based on the following two key indicators: Timestamp difference: the timestamp deviation of the audio and video data streams, reflecting the degree of synchronization offset. Queue length difference: the difference in the amount of data in the audio and video buffer queues, reflecting the backlog or shortage of data streams. These two indicators are mapped to scoring values ​​to evaluate the stability of each data stream.

[0045] In order to make the scores of different indicators comparable, normalization is required.

[0046] The normalized calculation formula for timestamp differences is: Timestamp difference value / timestamp difference threshold, where timestamp difference value: the absolute difference between the timestamps of the current audio and video frames; timestamp difference threshold: the maximum deviation of the timestamp allowed. , it means that the timestamp difference exceeds the acceptable range.

[0047] Normalized calculation formula for queue length difference: Queue length difference value / queue length difference threshold, where queue length difference value: the difference between the lengths of the audio buffer queue and the video buffer queue (unit: frame or time); queue length difference threshold: the maximum difference allowed for the queue lengths. , it indicates that the queue length difference exceeds the acceptable range.

[0048] The impact of timestamp difference and queue length difference may be different, so it is necessary to assign weights to the two indicators. and ,satisfy: .

[0049] The stability score is calculated by combining the scores of timestamp difference and queue length difference. : , when When it is close to 1, it means the data flow is more stable; when When it is close to 0, it means that the data stream is unstable. Select the reference source based on the stability score result: Compare the stability scores of the audio data stream and the video data stream, and the data stream with a higher stability score is selected as the reference source.

[0050] This application calculates the stability score by normalizing the timestamp difference and queue length difference, and assigning weights according to their impact, which allows the system to more accurately evaluate the stability of the audio and video streams. The overall effect of this step is to further improve the accuracy of the synchronization strategy, ensuring that the system can adjust the synchronization strategy according to the actual stability of the data stream, thereby minimizing the occurrence of asynchrony.

[0051] In some optional implementations of this embodiment, if the time difference exceeds the first threshold, the step of forcibly synchronizing the clocks of the OPS computer, the audio DSP, and the video DSP specifically includes: Get the current time of the OPS computer; Send the current time of the OPS computer to the audio DSP and the video DSP respectively; Updating the clocks of the audio DSP and the video DSP; Verify that the clocks of the audio DSP and video DSP are consistent with the OPS computer.

[0052] In this embodiment, when clock synchronization is required, the OPS computer obtains the current time, which will be used as a reference. The OPS computer sends its current time to the audio and video DSPs to ensure that their clocks are synchronized. After receiving the OPS time, the audio and video DSPs update their local clocks to keep them consistent with the OPS computer. Finally, the system checks whether the clocks of the audio and video DSPs have been successfully synchronized to the clock of the OPS computer. If they are consistent, it means that the clock synchronization is complete.

[0053] This application ensures that the clocks of the OPS computer, audio DSP, and video DSP are consistent by forcibly synchronizing the clocks of these devices. This measure effectively avoids audio and video synchronization problems caused by inconsistent clocks between devices, enhances system stability and clock calibration accuracy, and thus improves the accuracy and stability of audio and video synchronization.

[0054] In some optional implementations of this embodiment, in the step of performing frame dropping or re-display processing according to the time difference if the time difference exceeds the second threshold: If the audio data stream is faster than the video data stream, frame dropping processing is performed; If the audio data stream is slower than the video data stream, re-display processing is performed.

[0055] In this embodiment, if the processing speed of the audio stream is faster than the video stream, resulting in the audio being ahead, the system will discard some audio frames to ensure audio and video synchronization. If the processing speed of the audio stream is slower than the video stream, the system will repeatedly play the video frames until the video and audio are synchronized.

[0056] This application uses frame dropping or re-display processing, and the system can dynamically adjust according to the relative speed of audio and video streams. This measure effectively copes with the time difference that occurs when audio and video are synchronized, further ensuring the smoothness and consistency of audio and video streams, and preventing screen freezes or audio and video asynchrony caused by time differences.

[0057] In some optional implementations of this embodiment, before the above step of obtaining the audio data stream and the video data stream, the following step is further included: Initialize the audio DSP, video DSP and OPS computer, set the audio sampling rate, video frame rate, and establish a communication channel between DSP and OPS.

[0058] In this embodiment, before starting data stream processing, the system is first initialized to ensure that the sampling rate and frame rate of the audio and video DSP are correctly set, and to establish a reliable communication channel to ensure that the data stream can be transmitted smoothly.

[0059] This application initializes the audio DSP, video DSP and OPS computer, sets the audio sampling rate, video frame rate and establishes a communication channel. This measure ensures that the system has the basic parameters and stable communication channels required to process audio and video data streams from the beginning, reduces configuration errors or data stream transmission delays that may occur during the initialization process, and thus improves the overall efficiency and stability of the system.

[0060] In some optional implementations of this embodiment, after the above step of obtaining the audio data stream and the video data stream, the following step is further included: Performing echo cancellation, noise suppression and mixing matrix processing on the audio data stream; The video data stream is decoded, time stamped, image enhanced and matrix spliced.

[0061] In this embodiment, the audio data is processed to remove echoes, suppress background noise, and perform mixing to ensure clear sound quality. The video stream is decoded, a timestamp is added for synchronization, and image enhancement is performed to improve visual effects. If there are multiple image inputs, matrix splicing is performed.

[0062] After acquiring the audio and video data stream, this application performs echo cancellation, noise suppression and mixing matrix processing on the audio stream, and decodes, adds timestamps, enhances the picture and performs matrix splicing on the video stream. In this stage, by optimizing the audio and video streams, the quality of the audio and video data streams is ensured, interference factors such as noise and echo are reduced, and the clarity and details of the video picture are improved, thereby improving the audio-visual experience of the end user.

[0063] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0064] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0065] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0066] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0067] To solve the above technical problems, the present application also provides a computer device. Figure 3 , Figure 3 This is a basic structural block diagram of the computer device in this embodiment.

[0068] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0069] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0070] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the network video conference data processing method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0071] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the computer-readable instructions stored in the memory 41 or process data, such as running the computer-readable instructions of the network video conference data processing method.

[0072] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0073] The computer device provided in the present application detects the timestamp differences and queue length differences between the audio and video data streams, and combines a preset synchronization strategy to select a reference source and calculate the time difference. This enables the system to efficiently determine whether clock synchronization or frame loss / re-display processing is required, thereby ensuring the synchronization of the audio and video streams.

[0074] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned network video conferencing data processing method.

[0075] The computer-readable storage medium provided by the present application detects the timestamp differences and queue length differences between the audio and video data streams, and combines a preset synchronization strategy to select a reference source and calculate the time difference. This enables the system to efficiently determine whether clock synchronization or frame loss / re-display processing is required, thereby ensuring the synchronization of the audio and video streams.

[0076] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0077] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A network video conference data processing method, characterized in that: The steps include: Get audio data stream and video data stream; Detecting the frame time and buffer queue length of the audio data stream and the video data stream; Selecting an audio data stream or a video data stream as a reference source according to the detection result and a preset synchronization strategy; Calculating a time difference between the audio data stream and the video data stream, and determining whether the time difference exceeds a preset first threshold; If the time difference exceeds the first threshold, the clocks of the OPS computer, the audio DSP, and the video DSP are forced to be synchronized; If the time difference does not exceed the first threshold, further determine whether the time difference exceeds the second threshold. If the time difference exceeds the second threshold, performing frame dropping or re-display processing according to the time difference; If the time difference does not exceed the second threshold, the current audio data stream and video data stream are output.

2. The network video conference data processing method according to claim 1, characterized in that: The step of selecting the audio data stream or the video data stream as the reference source according to the detection result and the preset synchronization strategy specifically includes: Calculate the timestamp difference between the frame times of the audio data stream and the video data stream; Compare the queue length differences between audio data streams and video data streams; forming a stability scoring criterion according to the timestamp difference and the queue length difference; The reference source is selected according to the stability scoring criteria.

3. The network video conference data processing method according to claim 2, characterized in that: The step of forming a stability scoring standard according to the timestamp difference and the queue length difference specifically includes: Normalizing the timestamp difference and the queue length difference; Allocating weights according to the influence of the timestamp difference and the queue length difference; Calculate the stability score.

4. The network video conference data processing method according to claim 1, characterized in that: If the time difference exceeds the first threshold, the step of forcibly synchronizing the clocks of the OPS computer, the audio DSP, and the video DSP specifically includes: Get the current time of the OPS computer; Send the current time of the OPS computer to the audio DSP and the video DSP respectively; Updating the clocks of the audio DSP and the video DSP; Verify that the clocks of the audio DSP and video DSP are consistent with the OPS computer.

5. The network video conference data processing method according to claim 1, characterized in that: If the time difference exceeds the second threshold, in the step of performing frame dropping or re-display processing according to the time difference: If the audio data stream is faster than the video data stream, frame dropping processing is performed; If the audio data stream is slower than the video data stream, re-display processing is performed.

6. The network video conference data processing method according to claim 1, characterized in that: Before the step of obtaining the audio data stream and the video data stream, the method further includes: Initialize the audio DSP, video DSP and OPS computer, set the audio sampling rate, video frame rate, and establish a communication channel between DSP and OPS.

7. The network video conference data processing method according to claim 6, characterized in that: After the step of obtaining the audio data stream and the video data stream, the method further includes: Performing echo cancellation, noise suppression and mixing matrix processing on the audio data stream; The video data stream is decoded, time stamped, image enhanced and matrix spliced.

8. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the network video conference data processing method as described in any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the network video conference data processing method according to any one of claims 1 to 7 are implemented.

Images