Audio and video transmission method, device, electronic equipment and storage medium

By receiving video data and processing audio data synchronously in the image transmission device, the problem of asynchronous audio and video data transmission in traditional devices is solved, thus improving the user experience.

CN111447396BActive Publication Date: 2024-11-08VISIONVERA INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010153193.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-03-06
Publication Date
2024-11-08
Estimated Expiration
2040-03-06

AI Technical Summary

Technical Problem

Traditional image transmission devices cannot effectively synchronize audio and video data transmission, resulting in a poor user experience.

Method used

By receiving video data from an external video capture device and using audio data collected by its own audio capture component, the system performs synchronization processing based on timing relationships to generate synchronized audio and video data, which is then sent to the receiving end.

Benefits of technology

It enables the synchronous transmission of audio and video data, improving the user's viewing and interactive experience on the receiving end.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111447396B_ABST
    Figure CN111447396B_ABST
Patent Text Reader

Abstract

The present application provides an audio and video transmission method, device, electronic device and storage medium, the method is applied to data transmission equipment, including: receiving video data sent by an external video acquisition device; obtaining audio data collected by the audio acquisition component of the data transmission device itself, the audio data and the video data have a time sequence relationship; according to the time sequence relationship, the video data and the audio data are synchronously processed to obtain synchronous audio and video data; and the synchronous audio and video data is sent to the receiving end. In this process, the synchronous processing of the externally collected video data and the audio data of the user at the data transmission device end is realized, ensuring that the receiving end can receive the corresponding audio data in real time when receiving the video data, so that the user at the receiving end can better understand the video content and improve the user's experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an audio and video transmission method, device, electronic equipment and storage medium. Background Art

[0002] The visual network is a real-time, high-bandwidth transmission network based on Ethernet hardware, which uses a dedicated protocol to achieve high-speed transmission of high-definition video. An image transmission device is a device that connects to a video stream or encodes a video stream and transmits it. Traditional image transmission devices mainly connect to video resources of drones, surveillance cameras, selfie sticks, and VR devices, and connect video resources to corresponding business systems, such as conference systems, live broadcast systems, or other business systems. In actual business, image transmission devices use a one-way transmission method for audio data and video data. When there is a certain correlation between audio data and video data, it cannot be processed well, so that the receiving end needs to associate the audio data and video data again, resulting in a poor user experience. Summary of the invention

[0003] The embodiments of the present application provide an audio and video transmission method, device, electronic device and storage medium, aiming to achieve synchronous processing of audio data and video data.

[0004] A first aspect of an embodiment of the present application provides an audio and video transmission method, which is applied to a data transmission device, and the method includes:

[0005] Receive video data sent by an external video acquisition device;

[0006] Acquire audio data collected by the audio collection component of the data transmission device itself, wherein the audio data has a time-series correlation relationship with the video data;

[0007] According to the timing association relationship, the video data and the audio data are synchronously processed to obtain synchronous audio and video data;

[0008] The synchronous audio and video data is sent to a receiving end.

[0009] Optionally, according to the timing association relationship, synchronously processing the video data and the audio data to obtain synchronized audio and video data includes:

[0010] Processing the audio data through the first thread to obtain an audio processing result;

[0011] Processing the video data through the second thread to obtain a video processing result;

[0012] Through a third thread, according to the audio processing result, the video processing result and the timing association relationship, synchronously processing the video data and the audio data to obtain the synchronized audio and video data;

[0013] Sending the synchronous audio and video data to a receiving end includes:

[0014] Sending the synchronous audio and video data to a receiving end through the third thread;

[0015] The third thread is any one of the first thread and the second thread, or the third thread is different from the first thread and the second thread.

[0016] Optionally, processing the audio data to obtain an audio processing result includes:

[0017] Encapsulating the audio data to obtain audio data packets, each of which carries a timestamp;

[0018] Processing the video data to obtain a video processing result includes:

[0019] Encapsulating the video data to obtain video data packets, each of which carries a timestamp;

[0020] According to the audio processing result, the video processing result and the timing association relationship, synchronously processing the video data and the audio data includes:

[0021] Determine a target audio data packet and a target video data packet according to the timestamp of the audio data packet, the timestamp of the video data packet, the time interval value of adjacent video data packets and a preset threshold;

[0022] The target audio data packet and the target video data packet are synchronously processed according to the sequence of timestamps.

[0023] Optionally, determining a target audio data packet and a target video data packet according to a timestamp of the audio data packet, a timestamp of the video data packet, a time interval value between adjacent video data packets, and a preset threshold value includes:

[0024] Determine the latest video data packet as the target video data packet;

[0025] Determine the audio data packets whose timestamps are greater than a preset difference value in all audio data packets as the first type of audio data packets, wherein the preset difference value is the difference between the timestamp of the target video data packet and the preset threshold value;

[0026] The target audio data packet is determined from the audio data packets of the first type according to a magnitude relationship between the time interval value and the preset threshold.

[0027] Optionally, determining the target audio data packet from the audio data packets of the first type according to a magnitude relationship between the time interval value and the preset threshold value includes:

[0028] When the time interval value is less than the preset threshold, determining an audio data packet of the first type with a timestamp less than a first sum value as the target audio data packet, wherein the first sum value is a sum of the timestamp of the target video data packet and the time interval value;

[0029] When the time interval value is not less than the preset threshold, an audio data packet of the first type whose timestamp is less than a second sum value is determined as the target audio data packet, and the second sum value is the sum of the timestamp of the target video data packet and the preset threshold.

[0030] Optionally, the target audio data packet and the target video data packet are synchronously processed in a time stamp order, including:

[0031] Sorting the target audio data packets and the target video data packets in the order of timestamps;

[0032] Sending the synchronous audio and video data to a receiving end includes:

[0033] The target audio data packet whose timestamp is less than the timestamp of the target video data packet, the target video data packet, and the target audio data packet whose timestamp is not less than the timestamp of the target video data packet are sequentially sent to the receiving end.

[0034] Optionally, the method further comprises:

[0035] Determine the audio data packets whose timestamps are not greater than the preset difference value in all the audio data packets as the second type of audio data packets;

[0036] The audio data packets of the second type are deleted.

[0037] Optionally, after sending the synchronous audio and video data to the receiving end, the method further includes:

[0038] Receiving audio response data returned by the receiving end;

[0039] The audio response data is played.

[0040] Optionally, after encapsulating the audio data to obtain an audio data packet, the method further includes:

[0041] Buffering the audio data packet into an audio sending queue;

[0042] Periodically detecting the number of audio packets contained in the audio sending queue;

[0043] When the number of the audio packets is greater than a preset number, determining that the current network quality is poor;

[0044] Displays a prompt message indicating that the current network quality is poor.

[0045] A second aspect of the present application provides an audio and video transmission device, which is applied to a data transmission device, including:

[0046] A first receiving module, used for receiving video data sent by an external video acquisition device;

[0047] An acquisition module, used to acquire audio data collected by the audio acquisition component of the data transmission device itself, wherein the audio data has a time-series correlation relationship with the video data;

[0048] A processing module, configured to synchronously process the video data and the audio data according to the timing association relationship to obtain synchronous audio and video data;

[0049] The sending module is used to send the synchronous audio and video data to the receiving end.

[0050] Optionally, the processing module includes:

[0051] A first processing module, used to process the audio data through a first thread to obtain an audio processing result;

[0052] A second processing module, used for processing the video data through a second thread to obtain a video processing result;

[0053] A third processing module, configured to synchronously process the video data and the audio data through a third thread according to the audio processing result, the video processing result and the timing association relationship, to obtain the synchronized audio and video data;

[0054] The sending module comprises:

[0055] A first sending submodule, used for sending the synchronous audio and video data to a receiving end through the third thread;

[0056] The third thread is any one of the first thread and the second thread, or the third thread is different from the first thread and the second thread.

[0057] Optionally, the first processing module includes:

[0058] A first encapsulation module, used to encapsulate the audio data to obtain audio data packets, each of which carries a timestamp;

[0059] The second processing module comprises:

[0060] A second encapsulation module is used to encapsulate the video data to obtain video data packets, each of which carries a timestamp;

[0061] The third processing module comprises:

[0062] A first determination module, configured to determine a target audio data packet and a target video data packet according to a timestamp of the audio data packet, a timestamp of the video data packet, a time interval between adjacent video data packets, and a preset threshold;

[0063] The first processing submodule is used to synchronously process the target audio data packet and the target video data packet according to the sequence of timestamps.

[0064] Optionally, the first determining module includes:

[0065] A first determination submodule, configured to determine the latest video data packet as the target video data packet;

[0066] A second determination submodule, configured to determine, in all audio data packets, audio data packets with timestamps greater than a preset difference as first-type audio data packets, wherein the preset difference is a difference between the timestamp of the target video data packet and the preset threshold;

[0067] The third determining submodule is configured to determine the target audio data packet from the audio data packets of the first type according to a magnitude relationship between the time interval value and the preset threshold.

[0068] Optionally, the third determining submodule includes:

[0069] A fourth determination submodule, configured to, when the time interval value is less than the preset threshold, determine an audio data packet of the first type of audio data packet with a timestamp less than a first sum value as the target audio data packet, wherein the first sum value is a sum of the timestamp of the target video data packet and the time interval value;

[0070] The fifth determination submodule is used to determine, when the time interval value is not less than the preset threshold, an audio data packet whose timestamp in the audio data packet of the first type is less than a second sum value as the target audio data packet, and the second sum value is the sum of the timestamp of the target video data packet and the preset threshold value.

[0071] Optionally, the first processing submodule includes:

[0072] A second processing submodule, used for sorting the target audio data packet and the target video data packet according to the order of timestamps;

[0073] The sending module comprises:

[0074] The second sending submodule is used to sequentially send a target audio data packet whose timestamp is smaller than the timestamp of the target video data packet, the target video data packet, and a target audio data packet whose timestamp is not smaller than the timestamp of the target video data packet to the receiving end.

[0075] Optionally, the device further comprises:

[0076] A second determining module, configured to determine, in all audio data packets, audio data packets whose timestamps are not greater than the preset difference as audio data packets of the second type;

[0077] The deleting module is used to delete the second type of audio data packet.

[0078] Optionally, the device further comprises:

[0079] A second receiving module, used for receiving the audio response data returned by the receiving end;

[0080] A playing module is used to play the audio response data.

[0081] Optionally, the device further comprises:

[0082] A cache module, used for caching the audio data packet into an audio sending queue;

[0083] A detection module, used for periodically detecting the number of audio packets contained in the audio sending queue;

[0084] A third determination module, configured to determine that the current network quality is poor when the number of the audio packets is greater than a preset number;

[0085] The display module is used to display prompt information indicating that the current network quality is poor.

[0086] A third aspect of an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the steps of the method described in the first aspect of the present application.

[0087] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the method described in the first aspect of the present application.

[0088] Through the audio and video transmission method of the embodiment of the present application, the data transmission device receives the video data sent by the external video acquisition device, and collects the audio data through its own audio acquisition component, the audio data and the video data have a time sequence association relationship, and then the video data and the audio data are synchronously processed according to the time sequence association relationship to obtain the synchronous audio and video data, and finally the synchronous audio and video data is sent to the receiving end. In this process, the synchronous processing of the externally collected video data and the audio data of the user at the data transmission device end is realized, ensuring that the receiving end can receive the corresponding audio data in real time when receiving the video data, so that the user at the receiving end can better understand the video content, and improve the user's experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0090] Figure 1 It is a schematic diagram of an implementation environment shown in an embodiment of the present application;

[0091] Figure 2 is a flow chart of an audio and video transmission method shown in an embodiment of the present application;

[0092] Figure 3 It is a schematic diagram of the overall process of an audio and video transmission method shown in an embodiment of the present application;

[0093] Figure 4 is a structural block diagram of an audio and video transmission device shown in an embodiment of the present application;

[0094] Figure 5 This is a schematic diagram of a visual network shown in an embodiment of the present application;

[0095] Figure 6 It is a schematic diagram of the hardware structure of a node server shown in an embodiment of the present application;

[0096] Figure 7 It is a schematic diagram of the hardware structure of an access switch shown in an embodiment of the present application;

[0097] Figure 8 It is a schematic diagram of the hardware structure of an Ethernet protocol conversion gateway shown in an embodiment of the present application. DETAILED DESCRIPTION

[0098] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0099] Figure 1 This is a schematic diagram of an implementation environment shown in an embodiment of the present application. Figure 1 In the video transmission device, the operator can operate the image transmission device and drones, surveillance cameras, selfie sticks, VR devices, etc. The operator can record the voice data of the operator into the image transmission device through the audio acquisition component of the image transmission device, and input the video data collected by the drone, surveillance camera, selfie stick and VR device into the image transmission device through the designated port of the image transmission device, so that the image transmission device can synchronously process the audio data and video data, and then send the processed synchronous audio and video data to the receiving device in the related business, such as the live broadcast system, conference system, etc. At the same time, the image transmission device can also receive the audio data returned by the receiving device and play the audio data, so that the operator can perform the next step of processing.

[0100] This application provides an audio and video transmission method, which is applied to Figure 1 The image transmission device in the . Figure 2 1 is a flow chart of an audio and video transmission method shown in an embodiment of the present application. Figure 2 The audio and video transmission method of the present application may include the following steps:

[0101] Step S11: receiving video data sent by an external video acquisition device.

[0102] In this embodiment, the external video acquisition device can be any device with a video acquisition component, such as a drone, a surveillance camera, a selfie stick, and a VR device, etc., and this application does not impose specific restrictions on this. The data transmission device is a device with audio and video input and output functions, such as Figure 1 The image transmission device in can be used to receive video data sent by an external video acquisition device.

[0103] Step S12: obtaining audio data collected by the audio collection component of the data transmission device itself, wherein the audio data has a time-series correlation relationship with the video data.

[0104] In this embodiment, the data transmission device itself also has an audio acquisition component, which can be used to collect audio data, such as the voice data of the operator. The data transmission device collects audio data and receives video data synchronously. The audio data with a time-series correlation relationship with the video data can be: the voice data generated when the operator explains the video data received in real time. The voice data can enable the receiving end to better understand the content of the video based on the audio data when playing the video data.

[0105] Step S13: Synchronously process the video data and the audio data according to the timing association relationship to obtain synchronized audio and video data.

[0106] In this embodiment, if the data transmission device directly sends the received video data to the receiving end, and directly sends the collected audio data to the receiving end, usually, due to network conditions and other reasons, the audio data and video data received by the receiving end are not synchronized. Therefore, it is necessary to synchronize the video data and audio data to ensure that the audio heard by the receiving end is synchronized with the content of the video screen being watched.

[0107] Step S14: Send the synchronous audio and video data to the receiving end.

[0108] In this embodiment, after obtaining the synchronous audio and video data, the data transmission device can send the synchronous audio and video data to the receiving end. When sending, it can be sent in real time or at a specified time. This embodiment does not specifically limit the sending method.

[0109] For example, user A is currently using an image transmission device to connect the video data collected by the surveillance camera to the visual network conference X for the members of the visual network conference X to watch. In order to facilitate the conference members to understand the content of the video more conveniently when watching the video, user A can record the voice commentary of the video content into the image transmission device through the audio acquisition component of the image transmission device itself, and send it to the receiving end of the visual network conference system synchronously with the video. For example, the video content may include content of multiple time periods, and user A can make voice commentary for the video content of each time period separately, so that the conference members can better understand the video content of each time period. In order to ensure that the audio data and video data sent by the image transmission device to the receiving end are synchronized, it is necessary to synchronize the video data and audio data according to the time sequence association relationship (for example: the corresponding relationship between the voice commentary of each time period and the video content of each time period mentioned above), obtain synchronized audio and video data, and then send the synchronized audio and video data to the receiving end of the visual network conference system.

[0110] In this embodiment, the data transmission device receives video data sent by an external video acquisition device, and collects audio data through its own audio acquisition component. The audio data has a time-series correlation with the video data, and then performs synchronization processing on the video data and the audio data according to the time-series correlation to obtain synchronized audio and video data, and finally sends the synchronized audio and video data to the receiving end. In this process, the synchronization processing of the externally collected video data and the audio data of the user at the data transmission device end is realized, ensuring that the receiving end can receive the corresponding audio data in real time when receiving the video data, so that the user at the receiving end can better understand the video content, effectively improving the user's experience.

[0111] In combination with the above embodiments, in one implementation manner, after step S14, the following steps may also be performed:

[0112] Receiving audio response data returned by the receiving end;

[0113] The audio response data is played.

[0114] In this embodiment, the data transmission device end can not only send the synchronous audio and video data to the receiving end, but also receive the audio data returned by the receiving end, play the audio data, so that after the user of the data transmission device end hears the audio data, he can promptly carry out the next step of processing, for example, adjust the sent video, the sending time, etc. For example, when the user of the receiving end watches a certain video content, he has doubts about the audio content, and can record the doubts into the receiving end by voice at any time, so that the receiving end sends the voice data to the data transmission device end, and the user of the data transmission device end can explain the doubts by voice, and send the explanation to the receiving end again through the data transmission device end.

[0115] In this embodiment, audio can be transmitted between the data transmission device and the receiving end, realizing real-time voice interaction during video transmission, which can effectively meet the communication needs of users in actual scenarios and enhance the user experience.

[0116] In combination with the above embodiments, in one implementation, step S13 may include:

[0117] Processing the audio data through the first thread to obtain an audio processing result;

[0118] Processing the video data through the second thread to obtain a video processing result;

[0119] Through a third thread, according to the audio processing result, the video processing result and the timing association relationship, synchronously processing the video data and the audio data to obtain the synchronized audio and video data;

[0120] Accordingly, step S14 may include:

[0121] Sending the synchronous audio and video data to a receiving end through the third thread;

[0122] The third thread is any one of the first thread and the second thread, or the third thread is different from the first thread and the second thread.

[0123] In this embodiment, the processing of audio data, video data, and synchronous audio and video data can be realized respectively through three threads. Specifically, the audio data can be processed through the first thread to obtain an audio processing result, the video data can be processed through the second thread to obtain a video processing result, and then the video data and the audio data can be synchronously processed through the third thread according to the audio processing result, the video processing result, and the timing association relationship to obtain synchronous audio and video data. Among them, the third thread can be either the first thread or the second thread, or any other thread different from the first thread or the second thread. It can be set according to actual needs, and this application does not make specific restrictions on this.

[0124] In this embodiment, multi-threading is used to achieve synchronous processing of video data and audio data, which speeds up the processing speed, thereby achieving rapid transmission of data from the data transmission device to the receiving end, improving the viewing experience of users on the receiving end, and at the same time improving the interactive experience of users on both ends.

[0125] In combination with the above embodiments, in one implementation manner, processing the audio data to obtain an audio processing result includes:

[0126] Encapsulating the audio data to obtain audio data packets, each of which carries a timestamp;

[0127] Processing the video data to obtain a video processing result includes:

[0128] Encapsulating the video data to obtain video data packets, each of which carries a timestamp;

[0129] According to the audio processing result, the video processing result and the timing association relationship, synchronously processing the video data and the audio data includes:

[0130] Determine a target audio data packet and a target video data packet according to the timestamp of the audio data packet, the timestamp of the video data packet, the time interval value of adjacent video data packets and a preset threshold;

[0131] The target audio data packet and the target video data packet are synchronously processed according to the sequence of timestamps.

[0132] In this embodiment, processing audio data specifically refers to: encapsulating audio data to obtain audio data packets, wherein each audio data packet is marked with a current timestamp during encapsulation; processing video data specifically refers to: encapsulating video data to obtain video data packets, wherein each video data packet is marked with a current timestamp during encapsulation; on this basis, target audio data packets and target video data packets can be obtained according to the timestamps of the audio data packets, the timestamps of the video data packets, the time interval value t3 of adjacent video data packets and the preset threshold t1, wherein the time interval value t3 of adjacent video data packets can be calculated according to the Fps of the acquired video, and the preset threshold t1 indicates the time range within which the human eye cannot observe the asynchrony phenomenon, which is usually within 300-500ms, and the specific value of the preset threshold t1 can be selected according to actual needs; after obtaining the target audio data packet and the target video data packet, synchronization processing is performed in the order of the timestamps they carry, so as to obtain synchronized audio and video data.

[0133] Specifically, determining a target audio data packet and a target video data packet according to the timestamp of the audio data packet, the timestamp of the video data packet, the time interval value of adjacent video data packets and a preset threshold value includes:

[0134] Determine the latest video data packet as the target video data packet;

[0135] Determine the audio data packets whose timestamps are greater than a preset difference value in all audio data packets as the first type of audio data packets, wherein the preset difference value is the difference between the timestamp of the target video data packet and the preset threshold value;

[0136] The target audio data packet is determined from the audio data packets of the first type according to a magnitude relationship between the time interval value and the preset threshold.

[0137] In this embodiment, a single processing method is adopted for the video data packet. Each time a video data packet (i.e., a target video data packet) is generated, a corresponding audio data packet (i.e., a target audio data packet) is obtained, and then the video data packet and the corresponding audio data packet are processed into synchronous audio and video data and sent to the receiving end.

[0138] In specific implementation, the newly generated video data packet can be determined as the target video data packet, and its timestamp is represented by T; then among all the audio data packets, the audio data packets with timestamps greater than a preset difference are determined as the first type of audio data packets, and the preset difference is the difference between the timestamp T of the target video data packet and the preset threshold t1; after obtaining the first type of audio data packets, according to the size relationship between the time interval value t3 and the preset threshold t1, the target audio data packet is obtained from the first type of audio data packets.

[0139] Specifically, determining the target audio data packet from the first type of audio data packets according to the size relationship between the time interval value and the preset threshold includes:

[0140] When the time interval value is less than the preset threshold, the audio data packets with timestamps less than the first sum value in the first type of audio data packets are determined as the target audio data packets, and the first sum value is the sum of the timestamp of the target video data packet and the time interval value;

[0141] When the time interval value is not less than the preset threshold, the audio data packets with timestamps less than the second sum value in the first type of audio data packets are determined as the target audio data packets, and the second sum value is the sum of the timestamp of the target video data packet and the preset threshold.

[0142] In this embodiment, when the time interval value t3 is less than the preset threshold t1, the audio data packets with timestamps less than the first sum value are obtained from the first type of audio data packets and determined as the target audio data packets, where the first sum value is the sum of the timestamp T of the target video data packet and the time interval value t3;

[0143] When the time interval value t3 is not less than the preset threshold t1, the audio data packets with timestamps less than the second sum value are obtained from the first type of audio data packets and determined as the target audio data packets, and the second sum value is the sum of the timestamp T of the target video data packet and the preset threshold t1.

[0144] In this embodiment, whenever a target video data packet is generated, first, the audio data packets with timestamps greater than T - t1 (i.e., the first type of audio data packets) are obtained from all the generated and unsent audio data packets, and then the sizes of t1 and t3 are judged. If t3 < t1, the audio data packets with timestamps less than T + t3 in the first type of audio data packets are selected as the target audio data packets. If t3 >= t1, the audio data packets with timestamps less than T + t1 in the first type of audio data packets are selected as the target audio data packets.

[0145] In combination with the above embodiments, in one implementation, synchronizing the target audio data packets and target video data packets in the order of timestamps includes:

[0146] Sorting the target audio data packets and the target video data packets in the order of timestamps;

[0147] Sending the synchronized audio-visual data to the receiving end, including:

[0148] Sequentially sending the target audio data packets with timestamps less than the timestamp of the target video data packet, the target video data packet, and the target audio data packets with timestamps not less than the timestamp of the target video data packet to the receiving end.

[0149] In this embodiment, after obtaining the target audio data packets and target video data packets, the target audio data packets and target video data packets are sorted in the order of timestamps. The sorting result is: the target audio data packets with timestamps less than the timestamp T of the target video data packet, the target video data packet, and the target audio data packets with timestamps not less than the timestamp T of the target video data packet. Among them, the target audio data packets with timestamps less than the timestamp T of the target video data packet are part of the audio data packets generated before the target video data packet, and the target audio data packets with timestamps not less than the timestamp T of the target video data packet are part of the audio data packets generated after the target video data packet. The timestamps of these two parts of audio data packets are within the time range where the human eye cannot observe asynchronous phenomena.

[0150] In this embodiment, according to whether the timestamp t of the audio-video packet satisfies the two conditions of t > T - t1 and t < T + t3 (or t < T + t1), the audio-video packets synchronized with the target video data packet can be screened out, and these audio-video packets are sequentially sent to the receiving end in the order of timestamps, so that the timestamp differences between the audio data packets and video data packets received by the receiving end are all within the time range where the human eye cannot observe asynchronous phenomena, realizing the equivalent synchronization of audio-visual data and improving the user experience.

[0151] In combination with the above embodiments, in one implementation, the method further includes:

[0152] Determining the audio data packets with timestamps not greater than the preset difference among all audio data packets as the second type of audio data packets;

[0153] Deleting the second type of audio data packets.

[0154] In this embodiment, since the audio data packets whose timestamps are not greater than the preset difference value are beyond the time range where the human eye cannot observe the asynchrony phenomenon, they do not need to be sent to the receiving end and can be deleted to avoid occupying memory.

[0155] In combination with the above embodiments, in one implementation manner, after encapsulating the audio data to obtain an audio data packet, the method further includes:

[0156] Buffering the audio data packet into an audio sending queue;

[0157] Periodically detecting the number of audio packets contained in the audio sending queue;

[0158] When the number of the audio packets is greater than a preset number, determining that the current network quality is poor;

[0159] Displays a prompt message indicating that the current network quality is poor.

[0160] In this embodiment, all generated audio data packets can be cached in an audio sending queue, and then the number of audio packets in the audio sending queue can be periodically detected. If the number of audio packets is greater than a preset number, it indicates that the current network quality is poor. Information about the poor current network quality can be displayed on the display screen of the data transmission device to prompt the user on the data transmission device to check the network connection status in time.

[0161] Figure 3 This is a schematic diagram of the overall process of the audio and video transmission method shown in an embodiment of the present application. Figure 3 The audio and video transmission method of the present application is generally described with a specific embodiment.

[0162] exist Figure 3 In the example, the image transmission device has its own audio hardware device, which collects the voice data of the operator (i.e., the user on the image transmission device end) through thread A, encodes the voice data, obtains multiple audio packets, and stores these audio packets in queue A. Each audio packet carries a timestamp At. Secondly, the time t for collecting an audio packet can be calculated according to the device sampling rate of the audio hardware device. If the audio and video time interval is within 300-500ms, the human eye cannot observe the asynchrony phenomenon, so the minimum audio and video time difference t1 can be obtained based on this.

[0163] Meanwhile, the video transmission device connects to a drone, a surveillance camera, a selfie stick, and a VR device, etc. through thread B, collects the video data therein, encodes the video data to obtain a video packet carrying a time stamp T, and calculates the time interval t3 between adjacent video packets according to the Fps of the collected video during collection, and t3 < t1. In this embodiment, thread B is used as the sending thread. Therefore, thread B accesses each audio packet in queue A, and sequentially determines whether T - A.t < t1 is satisfied. If T - A.t < t1 is not satisfied, the corresponding part of the audio packet is deleted. If T - A.t < t1 is satisfied, it continues to determine whether A.t < T + t3 is satisfied. If it is satisfied, this part of the audio packet is selected for sending.

[0164] It should be noted that, for the method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the embodiments of the present invention are not limited by the described action sequences, because according to the embodiments of the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily essential for the embodiments of the present invention.

[0165] This application also provides an audio - video transmission device 400, which is applied to a data transmission device, such as Figure 4 shown. Figure 4 is a structural block diagram of an audio - video transmission device shown in an embodiment of this application. Referring to Figure 4 , the audio - video transmission device 400 of this application may include:

[0166] A first receiving module 401, configured to receive video data sent by an external video acquisition device;

[0167] An obtaining module 402, configured to obtain audio data collected by an audio acquisition component of the data transmission device itself, and the audio data has a timing correlation relationship with the video data;

[0168] A processing module 403, configured to perform synchronization processing on the video data and the audio data according to the timing correlation relationship to obtain synchronized audio - video data;

[0169] A sending module 404, configured to send the synchronized audio - video data to a receiving end.

[0170] Optionally, the processing module 403 includes:

[0171] A first processing module, configured to process the audio data through a first thread to obtain an audio processing result;

[0172] A second processing module, used for processing the video data through a second thread to obtain a video processing result;

[0173] A third processing module, configured to synchronously process the video data and the audio data through a third thread according to the audio processing result, the video processing result and the timing association relationship, to obtain the synchronized audio and video data;

[0174] The sending module 404 includes:

[0175] A first sending submodule, used for sending the synchronous audio and video data to a receiving end through the third thread;

[0176] The third thread is any one of the first thread and the second thread, or the third thread is different from the first thread and the second thread.

[0177] Optionally, the first processing module includes:

[0178] A first encapsulation module, used to encapsulate the audio data to obtain audio data packets, each of which carries a timestamp;

[0179] The second processing module comprises:

[0180] A second encapsulation module is used to encapsulate the video data to obtain video data packets, each of which carries a timestamp;

[0181] The third processing module comprises:

[0182] A first determination module, configured to determine a target audio data packet and a target video data packet according to a timestamp of the audio data packet, a timestamp of the video data packet, a time interval between adjacent video data packets, and a preset threshold;

[0183] The first processing submodule is used to synchronously process the target audio data packet and the target video data packet according to the sequence of timestamps.

[0184] Optionally, the first determining module includes:

[0185] A first determination submodule, configured to determine the latest video data packet as the target video data packet;

[0186] A second determination submodule, configured to determine, in all audio data packets, audio data packets with timestamps greater than a preset difference as first-type audio data packets, wherein the preset difference is a difference between the timestamp of the target video data packet and the preset threshold;

[0187] The third determining submodule is configured to determine the target audio data packet from the audio data packets of the first type according to a magnitude relationship between the time interval value and the preset threshold.

[0188] Optionally, the third determining submodule includes:

[0189] A fourth determination submodule, configured to, when the time interval value is less than the preset threshold, determine an audio data packet of the first type of audio data packet with a timestamp less than a first sum value as the target audio data packet, wherein the first sum value is a sum of the timestamp of the target video data packet and the time interval value;

[0190] The fifth determination submodule is used to determine, when the time interval value is not less than the preset threshold, an audio data packet whose timestamp in the audio data packet of the first type is less than a second sum value as the target audio data packet, and the second sum value is the sum of the timestamp of the target video data packet and the preset threshold value.

[0191] Optionally, the first processing submodule includes:

[0192] A second processing submodule, used for sorting the target audio data packet and the target video data packet according to the order of timestamps;

[0193] The sending module 404 includes:

[0194] The second sending submodule is used to sequentially send a target audio data packet whose timestamp is smaller than the timestamp of the target video data packet, the target video data packet, and a target audio data packet whose timestamp is not smaller than the timestamp of the target video data packet to the receiving end.

[0195] Optionally, the device 400 further includes:

[0196] A second determining module, configured to determine, in all audio data packets, audio data packets whose timestamps are not greater than the preset difference as audio data packets of the second type;

[0197] The deleting module is used to delete the second type of audio data packet.

[0198] Optionally, the device 400 further includes:

[0199] A second receiving module, used for receiving the audio response data returned by the receiving end;

[0200] A playing module is used to play the audio response data.

[0201] Optionally, the device 400 further includes:

[0202] A cache module, used for caching the audio data packet into an audio sending queue;

[0203] A detection module, used for periodically detecting the number of audio packets contained in the audio sending queue;

[0204] A third determination module, configured to determine that the current network quality is poor when the number of the audio packets is greater than a preset number;

[0205] The display module is used to display prompt information indicating that the current network quality is poor.

[0206] Based on the same inventive concept, the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the steps in the audio and video transmission method described in any of the above embodiments of the present application.

[0207] Based on the same inventive concept, the present application provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the audio and video transmission method described in any of the above embodiments of the present application are implemented.

[0208] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0209] The visual Internet is an important milestone in the development of the Internet. It is a real-time network that can realize real-time transmission of high-definition video and push many Internet applications towards high-definition video and high-definition face-to-face communication.

[0210] The visual network adopts real-time high-definition video exchange technology. It can integrate the required services such as high-definition video conferencing, video monitoring, intelligent monitoring analysis, emergency command, digital radio and television, delayed TV, online teaching, live broadcast, VOD on demand, TV mail, personalized recording (PVR), intranet (self-operated) channels, intelligent video broadcast control, information release and dozens of other video, voice, picture, text, communication, data and other services on a single system platform, and realize high-definition quality video playback through TV or computer.

[0211] In order to enable those skilled in the art to better understand the embodiments of the present invention, the visual network is introduced as follows:

[0212] Some of the technologies used in the visual Internet of Things are as follows:

[0213] Network Technology

[0214] The network technology innovation of visual networking improves the traditional Ethernet to cope with the huge potential video traffic on the network. Different from simple network packet switching or network circuit switching, visual networking technology uses packet switching to meet streaming needs. Visual networking technology has the flexibility, simplicity and low price of packet switching, and the quality and security guarantee of circuit switching, realizing the seamless connection of the whole network switching virtual circuit and data format.

[0215] Switching Technology

[0216] Visual networking uses the two advantages of Ethernet: asynchronism and packet switching. It eliminates Ethernet defects under the premise of full compatibility, and has seamless end-to-end connection across the entire network, directly connects to user terminals, and directly carries IP data packets. User data does not require any format conversion within the entire network. Visual networking is a more advanced form of Ethernet and a real-time switching platform that can achieve large-scale real-time transmission of high-definition video across the entire network, which is currently unattainable on the Internet, and pushes many network video applications towards high definition and unification.

[0217] Server Technology

[0218] The server technology on the visual network and unified video platform is different from the traditional server. Its streaming media transmission is based on connection-oriented. Its data processing capability is independent of traffic and communication time. A single network layer can include signaling and data transmission. For voice and video services, the complexity of streaming media processing on the visual network and unified video platform is much simpler than data processing, and its efficiency is more than 100 times higher than that of traditional servers.

[0219] Storage Technology

[0220] The ultra-high-speed storage technology of the unified video platform adopts the most advanced real-time operating system to adapt to the media content with ultra-large capacity and ultra-large flow, mapping the program information in the server command to the specific hard disk space. The media content no longer passes through the server, but is directly delivered to the user terminal in an instant. The user generally waits for less than 0.2 seconds. The optimized sector distribution greatly reduces the mechanical movement of the hard disk head seeking, and the resource consumption accounts for only 20% of the same level of IP Internet, but it generates 3 times more concurrent traffic than the traditional hard disk array, and the overall efficiency is improved by more than 10 times.

[0221] Network Security Technology

[0222] The structural design of the visual network completely eradicates the network security issues that plague the Internet through separate licensing for each service and complete isolation of equipment and user data. Generally, anti-virus programs and firewalls are not required, and attacks by hackers and viruses are eliminated, providing users with a structurally worry-free and secure network.

[0223] Service Innovation Technology

[0224] The unified video platform integrates services and transmission. Whether it is a single user, a private network user or a network, it is just an automatic connection. User terminals, set-top boxes or PCs are directly connected to the unified video platform to obtain rich and colorful multimedia video services in various forms. The unified video platform uses a "recipe-style" table configuration mode to replace traditional complex application programming. Complex applications can be implemented with very little code, realizing "unlimited" new business innovation.

[0225] The networking of the visual network is as follows:

[0226] Visual networking is a centrally controlled network structure. The network can be a tree network, star network, ring network, etc., but on this basis, there needs to be a centralized control node in the network to control the entire network.

[0227] Figure 5 FIG. 1 is a schematic diagram of a visual network shown in an embodiment of the present application. Figure 5 As shown in the figure, the visual network is divided into two parts: access network and metropolitan area network.

[0228] The equipment in the access network can be divided into three categories: node server, access switch, and terminal (including various set-top boxes, encoding boards, storage, etc.). The node server is connected to the access switch, and the access switch can be connected to multiple terminals and can be connected to Ethernet.

[0229] The node server is a node in the access network that performs centralized control functions and can control access switches and terminals. The node server can be directly connected to the access switch or directly connected to the terminal.

[0230] Similarly, the devices in the metropolitan area network can also be divided into three categories: metropolitan area servers, node switches, and node servers. The metropolitan area servers are connected to the node switches, and the node switches can be connected to multiple node servers.

[0231] The node server is a node server of the access network part, that is, the node server belongs to both the access network part and the metropolitan area network part.

[0232] The metropolitan area server is a node with centralized control function in the metropolitan area network, which can control the node switch and node server. The metropolitan area server can be directly connected to the node switch or the node server.

[0233] It can be seen from this that the entire visual network is a hierarchical and centrally controlled network structure, and the network controlled by the node server and the metropolitan area server can be a tree, star, ring and other structures.

[0234] Figuratively speaking, the access network part can form a unified video platform (the part in the dotted circle), and multiple unified video platforms can form a visual network; each unified video platform can be interconnected through the metropolitan area and wide area visual networks.

[0235] Visual networking equipment classification

[0236] 1.1 The devices in the visual network of the embodiment of the present invention can be mainly divided into three categories: servers, switches (including Ethernet gateways), terminals (including various set-top boxes, encoding boards, storage devices, etc.). The visual network as a whole can be divided into a metropolitan area network (or a national network, a global network, etc.) and an access network.

[0237] 1.2 The equipment in the access network can be mainly divided into three categories: node server, access switch (including Ethernet gateway), terminal (including various set-top boxes, encoding boards, storage, etc.).

[0238] The specific hardware structure of each access network device is as follows:

[0239] Node server:

[0240] Figure 6 FIG. 1 is a schematic diagram of the hardware structure of a node server shown in an embodiment of the present application. Figure 6 As shown, it mainly includes a network interface module 601, a switching engine module 602, a CPU module 603, and a disk array module 604;

[0241] Among them, the packets coming from the network interface module 601, the CPU module 603, and the disk array module 604 all enter the switching engine module 602; the switching engine module 602 performs an operation of checking the address table 605 on the incoming packets, thereby obtaining the packet's guidance information; and according to the packet's guidance information, the packet is stored in the queue of the corresponding packet buffer 606; if the queue of the packet buffer 606 is close to full, it is discarded; the switching engine module 602 polls all packet buffer queues, and forwards if the following conditions are met: 1) the port send buffer is not full; 2) the queue packet counter is greater than zero. The disk array module 604 mainly implements the control of the hard disk, including the initialization, reading and writing of the hard disk; the CPU module 603 is mainly responsible for the protocol processing between the access switch and the terminal (not shown in the figure), the configuration of the address table 605 (including the downstream protocol packet address table, the upstream protocol packet address table, and the data packet address table), and the configuration of the disk array module 604.

[0242] Access switch:

[0243] Figure 7 FIG. 1 is a schematic diagram of the hardware structure of an access switch shown in an embodiment of the present application. Figure 7 As shown, it mainly includes a network interface module (a downlink network interface module 701, an uplink network interface module 702), a switching engine module 703 and a CPU module 704;

[0244] Among them, the packet (uplink data) coming from the downlink network interface module 701 enters the packet detection module 705; the packet detection module 705 detects whether the destination address (DA), source address (SA), data packet type and packet length of the packet meet the requirements. If they meet the requirements, the corresponding stream identifier (stream-id) is allocated and enters the switching engine module 703, otherwise it is discarded; the packet (downlink data) coming from the uplink network interface module 702 enters the switching engine module 703; the data packet coming from the CPU module 704 enters the switching engine module 703; the switching engine module 703 processes the incoming packet The address table 706 is looked up to obtain the packet's guidance information; if the packet entering the switching engine module 703 is from the downstream network interface to the upstream network interface, the packet is stored in the queue of the corresponding packet buffer 707 in combination with the stream identifier (stream-id); if the queue of the packet buffer 707 is nearly full, it is discarded; if the packet entering the switching engine module 703 is not from the downstream network interface to the upstream network interface, the data packet is stored in the queue of the corresponding packet buffer 707 according to the packet's guidance information; if the queue of the packet buffer 707 is nearly full, it is discarded.

[0245] The switching engine module 703 polls all packet buffer queues. In the embodiment of the present invention, there are two situations:

[0246] If the queue is from the downlink network interface to the uplink network interface, it is forwarded if the following conditions are met: 1) the port send buffer is not full; 2) the queue packet counter is greater than zero; 3) a token generated by the bit rate control module is obtained;

[0247] If the queue is not going from the downstream network interface to the upstream network interface, it is forwarded if the following conditions are met: 1) the port send buffer is not full; 2) the queue packet counter is greater than zero.

[0248] The code rate control module 208 is configured by the CPU module 204 to generate tokens for all packet buffer queues from the downstream network interface to the upstream network interface within a programmable interval to control the code rate of the upstream forwarding.

[0249] The CPU module 704 is mainly responsible for protocol processing with the node server, configuration of the address table 706 , and configuration of the bit rate control module 708 .

[0250] Ethernet protocol conversion gateway :

[0251] Figure 8 FIG. 1 is a schematic diagram of the hardware structure of an Ethernet protocol conversion gateway shown in an embodiment of the present application. Figure 8 As shown, it mainly includes a network interface module (a downlink network interface module 801, an uplink network interface module 802), a switching engine module 803, a CPU module 804, a packet detection module 805, a bit rate control module 808, an address table 806, a packet buffer 807, a MAC adding module 809, and a MAC deleting module 810.

[0252] The data packet from the downlink network interface module 801 enters the packet detection module 805; the packet detection module 805 detects whether the Ethernet MAC DA, Ethernet MAC SA, Ethernet length or frame type, visual network destination address DA, visual network source address SA, visual network data packet type and packet length of the data packet meet the requirements, and if they meet the requirements, the corresponding stream identifier (stream-id) is assigned; then, the MAC deletion module 810 subtracts the MAC DA, MAC SA, length or frame type (2byte) and enters the corresponding receiving buffer, otherwise it is discarded;

[0253] The downstream network interface module 801 detects the sending buffer of the port. If there is a packet, it obtains the Ethernet MAC DA of the corresponding terminal based on the visual network destination address DA of the packet, adds the Ethernet MAC DA of the terminal, the MACSA of the Ethernet protocol conversion gateway, and the Ethernet length or frame type, and sends it.

[0254] The functions of other modules in the Ethernet protocol conversion gateway are similar to those of the access switch.

[0255] terminal:

[0256] It mainly includes network interface module, business processing module and CPU module; for example, the set-top box mainly includes network interface module, audio and video encoding and decoding engine module, CPU module; the encoding board mainly includes network interface module, audio and video encoding engine module, CPU module; the memory mainly includes network interface module, CPU module and disk array module.

[0257] 1.3 The equipment in the metropolitan area network can be divided into two categories: node server, node switch, and metropolitan area server. Among them, the node switch mainly includes network interface module, switching engine module and CPU module; the metropolitan area server mainly includes network interface module, switching engine module and CPU module.

[0258] 2. Definition of Visual Network Data Packet

[0259] 2.1 Access Network Data Packet Definition

[0260] The data packet of the access network mainly includes the following parts: destination address (DA), source address (SA), reserved bytes, payload (PDU), and CRC.

[0261] As shown in the following table, the data packet of the access network mainly includes the following parts:

[0262] DA S Reserved Payload CRC

[0263] in:

[0264] The destination address (DA) consists of 8 bytes. The first byte indicates the type of data packet (such as various protocol packets, multicast data packets, unicast data packets, etc.), with a maximum of 256 possibilities. The second to sixth bytes are the metropolitan area network address, and the seventh and eighth bytes are the access network address.

[0265] The source address (SA) is also composed of 8 bytes and has the same definition as the destination address (DA).

[0266] The reserved bytes consist of 2 bytes;

[0267] The payload part has different lengths depending on the type of datagram. If it is a protocol packet, it is 64 bytes. If it is a single multicast data packet, it is 32+1024=1056 bytes. Of course, it is not limited to the above two types.

[0268] CRC consists of 4 bytes and its calculation method follows the standard Ethernet CRC algorithm.

[0269] 2.2 Definition of Metropolitan Area Network Data Packet

[0270] The topology of the metropolitan area network is a graph. There may be two or more connections between two devices, that is, there may be more than two connections between a node switch and a node server, a node switch and a node switch, and a node switch and a node server. However, the metropolitan area network address of a metropolitan area network device is unique. In order to accurately describe the connection relationship between metropolitan area network devices, a parameter: a label is introduced in the embodiment of the present invention to uniquely describe a metropolitan area network device.

[0271] The definition of labels in this manual is similar to the definition of labels in MPLS (Multi-Protocol Label Switch). Assuming that there are two connections between device A and device B, there are two labels for the data packet from device A to device B, and there are two labels for the data packet from device B to device A. Labels are divided into input labels and output labels. Assuming that the label of the data packet entering device A (input label) is 0x0000, the label of the data packet when leaving device A (output label) may become 0x0001. The access process of the metropolitan area network is a centralized access process, which means that the address allocation and label allocation of the metropolitan area network are dominated by the metropolitan area server, and the node switch and node server are only passively executed. This is different from the label allocation of MPLS, which is the result of mutual negotiation between the switch and the server.

[0272] As shown in the following table, the data packets of the metropolitan area network mainly include the following parts:

[0273] DA S Reserved Label Payload CRC

[0274] That is, destination address (DA), source address (SA), reserved byte (Reserved), label, payload (PDU), CRC. The label format can refer to the following definition: the label is 32 bits, of which the upper 16 bits are reserved and only the lower 16 bits are used. It is located between the reserved bytes and payload of the data packet.

[0275] Based on the above characteristics of the visual network, one of the core concepts of an embodiment of the present invention is proposed. In compliance with the protocol of the visual network, a data transmission device receives video data sent by an external video acquisition device, and acquires audio data through its own audio acquisition component. The audio data has a time sequence relationship with the video data. The video data and the audio data are then synchronously processed according to the time sequence relationship to obtain synchronized audio and video data, and finally the synchronized audio and video data are sent to a receiving end.

[0276] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0277] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0278] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0279] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0280] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0281] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0282] The above is a detailed introduction to an audio and video transmission method, device, electronic device and storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for a person skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An audio and video transmission method, It is characterized in that Applied to data transmission equipment, including: Receive video data sent by an external video acquisition device; Acquire audio data collected by the audio collection component of the data transmission device itself, wherein the audio data has a time-series correlation relationship with the video data; According to the timing association relationship, the video data and the audio data are synchronously processed to obtain synchronous audio and video data; Sending the synchronous audio and video data to a receiving end; Determine the audio data packets whose timestamps are not greater than a preset difference value in all audio data packets as the second type of audio data packets; the audio data packets are obtained by encapsulating the audio data, and each of the audio data packets carries a timestamp; deleting the second type of audio data packet; According to the audio processing result, the video processing result and the timing association relationship, the video data and the audio data are synchronously processed, including: Determine a target audio data packet and a target video data packet according to the timestamp of the audio data packet, the timestamp of the video data packet, the time interval value of adjacent video data packets and a preset threshold; Synchronously processing the target audio data packet and the target video data packet according to the order of timestamps; Determining a target audio data packet and a target video data packet according to the timestamp of the audio data packet, the timestamp of the video data packet, the time interval value of adjacent video data packets, and a preset threshold value, includes: Determine the latest video data packet as the target video data packet; Determine the audio data packets whose timestamps are greater than a preset difference value in all audio data packets as the first type of audio data packets, wherein the preset difference value is the difference between the timestamp of the target video data packet and the preset threshold value; Determining the target audio data packet from the audio data packets of the first type according to a magnitude relationship between the time interval value and the preset threshold; Determining the target audio data packet from the audio data packets of the first type according to the magnitude relationship between the time interval value and the preset threshold value includes: When the time interval value is less than the preset threshold, determining an audio data packet of the first type with a timestamp less than a first sum value as the target audio data packet, wherein the first sum value is a sum of the timestamp of the target video data packet and the time interval value; When the time interval value is not less than the preset threshold, an audio data packet of the first type whose timestamp is less than a second sum value is determined as the target audio data packet, and the second sum value is the sum of the timestamp of the target video data packet and the preset threshold.

2. The method according to claim 1, It is characterized in that According to the timing association relationship, the video data and the audio data are synchronously processed to obtain synchronized audio and video data, including: Processing the audio data through the first thread to obtain an audio processing result; Processing the video data through a second thread to obtain a video processing result; Through a third thread, according to the audio processing result, the video processing result and the timing association relationship, the video data and the audio data are synchronously processed to obtain the synchronous audio and video data: Sending the synchronous audio and video data to a receiving end includes: Sending the synchronous audio and video data to a receiving end through the third thread; The third thread is any one of the first thread and the second thread, or the third thread is different from the first thread and the second thread.

3. The method according to claim 2, It is characterized in that Processing the audio data to obtain an audio processing result includes: Encapsulating the audio data to obtain audio data packets, each of which carries a timestamp; Processing the video data to obtain a video processing result includes: The video data is encapsulated to obtain video data packets, each of which carries a timestamp.

4. The method according to claim 1, It is characterized in that The target audio data packet and the target video data packet are synchronously processed in a time stamp order, including: Sorting the target audio data packets and the target video data packets in the order of timestamps; Sending the synchronous audio and video data to a receiving end includes: The target audio data packet whose timestamp is less than the timestamp of the target video data packet, the target video data packet, and the target audio data packet whose timestamp is not less than the timestamp of the target video data packet are sequentially sent to the receiving end.

5. The method according to claim 1, It is characterized in that After sending the synchronous audio and video data to the receiving end, the method further includes: Receiving audio response data returned by the receiving end; The audio response data is played.

6. An audio and video transmission device, It is characterized in that Applied to data transmission equipment, including: A first receiving module, used for receiving video data sent by an external video acquisition device; An acquisition module, used to acquire audio data collected by the audio acquisition component of the data transmission device itself, wherein the audio data has a time-series correlation relationship with the video data; A processing module, configured to synchronously process the video data and the audio data according to the timing association relationship to obtain synchronous audio and video data; A sending module, used to send the synchronous audio and video data to a receiving end; A second determining module, configured to determine, in all audio data packets, audio data packets whose timestamps are not greater than a preset difference as audio data packets of a second type; the audio data packets are obtained by encapsulating the audio data, and each of the audio data packets carries a timestamp; A deleting module, used for deleting the second type of audio data packet; The third processing module includes: A first determination module, configured to determine a target audio data packet and a target video data packet according to a timestamp of the audio data packet, a timestamp of the video data packet, a time interval between adjacent video data packets, and a preset threshold; A first processing submodule, configured to synchronously process the target audio data packet and the target video data packet in a time stamp sequence; The first determining module comprises: A first determination submodule, configured to determine the latest video data packet as the target video data packet; A second determination submodule, configured to determine, in all audio data packets, audio data packets with timestamps greater than a preset difference as first-type audio data packets, wherein the preset difference is a difference between the timestamp of the target video data packet and the preset threshold; A third determining submodule, configured to determine the target audio data packet from the audio data packets of the first type according to a magnitude relationship between the time interval value and the preset threshold; The third determining submodule comprises: A fourth determination submodule, configured to, when the time interval value is less than the preset threshold, determine an audio data packet of the first type of audio data packet with a timestamp less than a first sum value as the target audio data packet, wherein the first sum value is a sum of the timestamp of the target video data packet and the time interval value; The fifth determination submodule is used to determine, when the time interval value is not less than the preset threshold, an audio data packet whose timestamp in the audio data packet of the first type is less than a second sum value as the target audio data packet, and the second sum value is the sum of the timestamp of the target video data packet and the preset threshold value.

7. An electronic device, It is characterized in that include: one or more processors; and One or more machine-readable media having instructions stored thereon, when executed by the one or more processors, enable the electronic device to perform the steps in the audio and video transmission method as described in any one of claims 1-5.

8. A computer-readable storage medium, It is characterized in that The computer program stored therein enables the processor to execute the steps in the audio and video transmission method as described in any one of claims 1-5.