Audio and video data processing method, electronic equipment and computer storage medium

Abstract

The invention discloses an audio and video data processing method, electronic equipment and a computer storage medium. The audio and video data processing method comprises the following steps: acquiring audio and video data; adding header attribute information to each frame of audio and video data; and sequentially writing each frame of audio and video data carrying the head attribute information into the same buffer area. According to the method, the audio data and the video data are sequentially written into the same buffer area, so that time synchronization of the audio data and the video data can be ensured, and an additional synchronization mechanism is not needed.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of audio and video technology, and in particular to an audio and video data processing method, an electronic device and a computer storage medium. BACKGROUND

[0002] In related technologies, when real-time audio and video encoding is cached, audio and video are usually stored in two independent caches respectively. This method increases the difficulty of audio and video synchronization. For example, in the case where the reading speed is less than the writing speed, the audio and video need to be selectively dropped, at which time an additional audio and video synchronization mechanism is required. SUMMARY

[0003] The present application aims to at least solve one of the technical problems in the prior art. To this end, one object of the present application is to provide an audio and video data processing method, which can ensure the time synchronization of audio and video data by sequentially writing audio and video data into the same buffer, without the need for an additional synchronization mechanism.

[0004] A second object of the present application is to provide an electronic device.

[0005] A third object of the present application is to provide a computer storage medium.

[0006] To solve the above problems, an audio and video processing method according to a first aspect of the present application comprises: obtaining audio and video data; adding header attribute information to each frame of the audio and video data; and sequentially writing each frame of the audio and video data carrying the header attribute information into the same buffer.

[0007] According to the audio and video data processing method of the present application, after obtaining the audio and video data, whether it is audio data or video data, it is sequentially written into the same buffer for storage, thereby ensuring the time synchronization of audio and video data when storing data. Readers only need to read in order, without the need for an additional synchronization mechanism, effectively solving the problem of audio and video synchronization in subsequent business. At the same time, the present application can also improve the scalability of data storage by adding header attribute information to each frame of audio and video data.

[0008] In some embodiments, sequentially writing each frame of the audio and video data carrying the header attribute information into the same buffer comprises: sequentially writing each frame of the audio and video data carrying the header attribute information into the same buffer using a serial communication method.

[0009] In some embodiments, the header attribute information adopts a binary data structure.

[0010] In some embodiments, the data type of the audio and video data is a union type.

[0011] In some embodiments, the header attribute information at least includes a time stamp.

[0012] In some embodiments, the audio / video data includes audio data and video data, the header attribute information of the audio data further includes an audio coding format, a sampling rate, and a data length, and the header attribute information of the video data further includes a video coding format, a frame rate, a code rate, and a data length.

[0013] In some embodiments, the buffer includes a cache layer for storing audio / video data, and the audio / video data carrying the header attribute information is sequentially written into the same buffer, including: under the condition that there is free cache space in the cache layer, the audio / video data carrying the header attribute information is sequentially written into the cache layer from the low address of the free cache space.

[0014] In some embodiments, the buffer is a circular buffer, and the buffer includes a cache layer for storing audio / video data, and the audio / video data carrying the header attribute information is sequentially written into the same buffer, including: under the condition that there is no free cache space in the cache layer, the historical audio / video data stored at the low address of the cache layer is sequentially cleared; and the audio / video data carrying the header attribute information is sequentially written into the address where the historical audio / video data has been cleared from the low address of the cache layer.

[0015] In some embodiments, the buffer at least includes an interface layer for providing read / write interfaces, and the audio / video data is obtained by: calling any interface in the interface layer that is not occupied; and obtaining the audio / video data through the any interface.

[0016] In some embodiments, the buffer further includes a management layer for managing audio / video data and a cache layer for storing audio / video data, and the method further includes: receiving a read data instruction; calling one of the interfaces in the interface layer that is not occupied; and sequentially reading the audio / video data in the cache layer through the one interface by the management layer.

[0017] In some embodiments, the method further includes: marking a key frame for the first frame of audio / video data in the audio / video data; under the condition that the reading speed of the one interface is lower than the writing speed of the any interface, obtaining a current reading position of the one interface; and when the current reading position is determined to be abnormal, jumping the data reading position of the one interface from the current reading position to the address position of the key audio / video data.

[0018] In some embodiments, the method further includes: under the condition that the reading speed of the one interface is lower than the writing speed of the any interface, allowing each frame of audio / video data read to be copied out of the cache layer.

[0019] An electronic device is provided in a second aspect of the present application, comprising: at least one processor; a memory communicatively connected with the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the at least one processor implements the audio and video data processing method according to the above-mentioned embodiments when executing the computer program.

[0020] The electronic device according to the embodiments of the present application can guarantee the time synchronization of the audio data and the video data by executing the computer program stored in the memory by the processor, and the audio and video processing method according to the above-mentioned embodiments can be implemented without an additional synchronization mechanism.

[0021] A computer storage medium is provided in a third aspect of the present application, wherein the computer program is executed by the processor to implement the audio and video data processing method according to the above-mentioned embodiments.

[0022] Additional aspects and advantages of the present application will be made apparent by the following description of embodiments of the present application, given as a non-restrictive example, with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0023] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood by referring to the following description, taken in conjunction with the accompanying drawings, in which:

[0024] Figure 1 is a flowchart of an audio and video data processing method according to an embodiment of the present application;

[0025] Figure 2 is a schematic diagram of a buffer according to an embodiment of the present application;

[0026] Figure 3 is a structural block diagram of a buffer according to an embodiment of the present application;

[0027] Figure 4 is a structural block diagram of an electronic device according to an embodiment of the present application.

[0028] REFERENCE NUMERALS

[0029] Buffer 100; electronic device 200;

[0030] Interface layer 10; management layer 20; cache layer 30; processor 40; memory 50. DETAILED DESCRIPTION

[0031] Embodiments of the present application are described in detail below, and the embodiments described with reference to the accompanying drawings are exemplary, and embodiments of the present application are described in detail below.

[0032] To solve the above problems, the first aspect of the present application provides an audio and video processing method, which writes audio and video data into the same buffer in sequence, so as to ensure the time synchronization of audio and video data without additional synchronization mechanism.

[0033] The following description is made with reference to the accompanying drawings, which show, by way of example, the principles of the present application. Figure 1 The audio and video processing method according to the present application is described below, which comprises at least steps S1-S3.

[0034] In step S1, audio and video data are acquired.

[0035] Specifically, the audio and video data include audio data and / or video data, which can be acquired by audio and / or video devices, such as microphones and cameras.

[0036] In step S2, header attribute information is added to each frame of audio and video data.

[0037] Specifically, additional information, i.e. header attribute information, is added to the beginning of each frame of audio and video data, which can be used to distinguish audio data from video data, and can also be used to extend other data besides audio and video data, thereby improving the scalability of data storage.

[0038] In step S3, each frame of audio and video data carrying header attribute information is written into the same buffer in sequence.

[0039] Specifically, the audio and video data are written into the buffer in the order of encoding, so that the audio and video data have a sequence when stored, i.e. the time synchronization of audio and video data is ensured, and when reading the audio and video data, they can be read in sequence without additional synchronization mechanism, effectively solving the problem of audio and video synchronization in subsequent services.

[0040] For example, referring to FIG. 1, Figure 2 As shown in FIG. 1, the audio information and video information carrying header attribute information, i.e. data header, are written into the buffer in the order of encoding, so that when reading the audio information and video information, they can be read in sequence without additional synchronization mechanism, thereby solving the problem of time synchronization of audio and video.

[0041] According to the audio and video data processing method of the present invention, after acquiring audio and video data, both audio and video data are sequentially written into the same buffer for storage. This ensures time synchronization between audio and video data during data storage, allowing readers to simply read the data in sequence without the need for additional synchronization mechanisms. This effectively solves the problem of audio and video synchronization in subsequent services. Furthermore, the present application can also improve the scalability of data storage by adding header attribute information to each frame of audio and video data.

[0042] In some embodiments, the audio and video data carrying header attribute information of each frame is sequentially written into the same buffer, including using a serial communication method to sequentially write the audio and video data carrying header attribute information of each frame into the same buffer.

[0043] For details, please refer to Figure 3 As shown, the buffer 100 also includes a management layer 20 for managing audio and video data. When audio and video data are written sequentially into the same buffer, the management layer 20 uses serial communication to write the audio and video data into the buffer's cache layer 30 in the order of their encoding. It should be noted that, as the amount of audio and video data increases, compared to using parallel communication, the serial communication method used in this application can effectively reduce system bandwidth pressure while ensuring the accuracy of data written in multi-writer scenarios.

[0044] In some embodiments, the header attribute information uses a binary data structure, which simplifies the header attribute information and reduces network transmission redundancy.

[0045] In some embodiments, the data type of audio and video data is a union type. By using the union for audio and video data, the data structure space can be reused, thus saving memory space.

[0046] In some embodiments, the header attribute information includes at least a timestamp, thereby ensuring data integrity and providing a time reference.

[0047] In some embodiments, audio and video data include audio data and video data. The header attribute information of the audio data further includes the audio encoding format, sampling rate, and data length, while the header attribute information of the video data further includes the video encoding format, frame rate, bit rate, and data length. Therefore, by adding the above information to the header attribute information, data transmission quality can be effectively improved, data transmission efficiency can be increased, data integrity can be guaranteed, and it is also easier to adapt to different transmission requirements.

[0048] In some embodiments, the buffer includes a cache layer for storing audio and video data, and the audio and video data carrying header attribute information of each frame is written to the same buffer sequentially, including writing the audio and video data carrying header attribute information of each frame sequentially to the cache layer starting from the low address of the free cache space, provided that there is free cache space in the cache layer.

[0049] Specifically, such as Figure 2 As shown, if Figure 2 The rightmost address is the lowest address of the cache layer. From right to left, the address of the cache layer gradually increases. When a writer writes audio and video data to the cache layer through any interface, the audio and video data will be written to the cache layer sequentially from the lowest address, thus occupying the cache space of the cache layer. Based on this, before writing each frame of audio and video data carrying header attribute information to the cache layer sequentially, the cache space of the cache layer needs to be checked. If there is free cache space, it means that the cache layer is not completely full. The writer's writing of audio and video data will not overwrite the original cached data, so the writing operation can continue. The write position is directly written and updated, that is, starting from the lowest address of the free cache space, each frame of audio and video data carrying header attribute information is written to the cache layer sequentially. At the same time, this process will not affect the reader's reading of data.

[0050] In some embodiments, the buffer is a circular buffer, which includes a cache layer for storing audio and video data. Each frame of audio and video data carrying header attribute information is sequentially written to the same buffer. This includes, when there is no free cache space in the cache layer, sequentially clearing the historical audio and video data stored at the low address starting from the low address of the cache layer; and sequentially writing each frame of audio and video data carrying header attribute information to the address where the historical audio and video data has been cleared, starting from the low address of the cache layer.

[0051] Specifically, the buffer is a circular buffer. The data structure of the circular buffer is special, and the size of the circular buffer cache space is fixed and does not change dynamically. The circular buffer connects the end of the buffer to the beginning position to form a closed loop, thereby realizing the circular access of data. Based on this, before writing each frame of audio and video data carrying header attribute information to the buffer layer in sequence, it is necessary to check the cache space of the buffer layer. If there is no free cache space, it means that the cache layer is completely full. Therefore, in order to enable the writer to continue writing data, it first starts from the low address of the cache layer and clears the historical audio and video data stored at the low address in sequence. After clearing the historical audio and video data, there is free cache space in the buffer layer. Then, each frame of audio and video data carrying header attribute information can be written to the address of the cleared historical audio data in sequence. That is to say, when writing data to overwrite the original data, the writer first records the oldest valid data position after the overwrite, i.e., the historical audio and video data at the low address, and then writes the newly written audio and video data directly and updates the write position.

[0052] In some embodiments, referring to Figure 3 As shown in the figure, the buffer 100 includes an interface layer 10 for providing read-write interfaces, and the obtaining of the audio and video data includes: calling any interface in the interface layer 10 that is not occupied; and obtaining the audio and video data through the any interface.

[0053] Specifically, referring to Figure 3 As shown in the figure, the interface layer 10 provides read-write interfaces of each module, and can record the reading position of each reader at the same time; the buffer 100 supports parallel operation of multiple readers and multiple writers, and supports two modes of data copying and direct use of cache address space when reading data, and thus can be compatible with all reading scenarios, and can reduce system bandwidth occupation caused by memory copying when the cache address space mode is used. Based on this, when obtaining the audio and video data, any interface in the interface layer 10 that is not occupied is called, and thus the audio and video data is written into the buffer 100 through the interface layer 10.

[0054] In some embodiments, the buffer further includes a management layer 20 for managing the audio and video data and a cache layer 30 for storing the audio and video data, and the method further includes: receiving a reading data instruction; calling one of the interfaces in the interface layer 10 that is not occupied; and the management layer 20 reading the audio and video data in the cache layer 30 through the one interface in sequence.

[0055] Specifically, the buffer supports modularization between the producer / consumer business and multi-threaded simultaneous reading and writing, and based on this, if an external program needs to use the audio and video data in the buffer 100, a reading data instruction is sent to the management layer 20, the management layer 20 calls one of the interfaces in the interface layer 10 that is not occupied, and thus the reading of the audio and video data stored in the cache layer 30 is completed through the interface layer 10, wherein the interface layer 10 can be multiple interfaces, and multiple audio and video data can be read from the cache layer 30 of the buffer 100 at the same time, wherein the reading processes of each audio and video data do not affect each other, and the weak coupling between businesses is reduced.

[0056] In some embodiments, the method of the present application further includes marking a key frame in the first frame of audio and video data; under the condition that the reading speed of one of the interfaces is lower than the writing speed of any interface, obtaining the current reading position of the one interface; and when it is determined that the current reading position is abnormal, jumping the data reading position of the one interface from the current reading position to the address position of the key audio and video data.

[0057] The cache size of the cache layer 30 is at least capable of storing video data of one GOP (Group of Picture, period of key frame); if pre-recording is needed, the cache size of the cache layer 30 is the pre-recording duration T*(audio bps+video bps). The audio bps is the bit number of audio data transmitted per unit time, and the video bps is the bit number of video data transmitted per unit time.

[0058] Specifically, the first frame of audio and video data in the audio and video data is marked as a key frame, so that the key frame can be quickly located, thereby ensuring the playing of the audio and video. When multiple audio and video data are read from the cache layer 30 of the buffer 100 and multiple audio and video data are written into the cache layer 30, the reading speed and the writing speed of each interface need to be monitored. If the reading speed of one interface is lower than the writing speed of any interface, that is, the reading speed of the reader is slower than the writing speed of the writer, the current reading position of the one interface is checked for abnormality. If it is determined that the current reading position is abnormal after the checking, the data reading position of the one interface is jumped from the current reading position to the address position of the key audio and video data, thereby ensuring the reading and writing efficiency and avoiding decoding abnormality and other errors. When the reading speed of the reader is slower than the writing speed of the writer, whether an abnormality occurs can be detected by comparing the reading position information of the reader managed in the management layer with the oldest valid data position of the cache layer, that is, the low address bit of the cache layer.

[0059] In addition, for the scenario that the reading speed of the reader is slower than the writing speed of the writer, the method abstracts the data storage, the reader and the writer as independent instances, so that multiple writers and readers do not affect each other and can read and write in parallel, thereby reducing the weak coupling between services, better accommodating the inconsistent scenarios of production speed and consumption speed, and without the need of additional audio and video synchronization mechanism.

[0060] In some embodiments, the method of the application further includes: under the condition that the reading speed of one interface is lower than the writing speed of any interface, allowing each frame of audio and video data read to be copied out of the cache layer 30.

[0061] Specifically, when the reading speed of the reader is slower than the writing speed of the writer, each frame of audio and video data read can be copied out of the cache layer 30, so that even if the reading speed is slow, the problem of loss due to the covering of audio and video data can be avoided.

[0062] It should be noted that when the reading speed of the reader is faster than the writing speed of the writer, the reader can directly use the cache address space without memory copying, thereby reducing the data copying of the subsequent service and optimizing the bandwidth pressure brought by reading to the system.

[0063] The second aspect of the present application provides an electronic device 200, referring to Figure 4 As shown in the figure, the electronic device 200 comprises at least one processor 40; a memory 50 connected with the at least one processor 40; wherein the memory 50 stores a computer program executable by the at least one processor 40, and the at least one processor 40 executes the computer program to realize the audio and video data processing method of the above-mentioned embodiments.

[0064] The electronic device 200 according to the embodiments of the present application executes the computer program stored in the memory 50 by the processor 40, realizes the audio and video processing method of the above-mentioned embodiments, and can guarantee the time synchronization of the audio data and the video data without additional synchronization mechanism.

[0065] The third aspect of the present application provides a computer storage medium, wherein the computer program is executed by the processor to realize the audio and video data processing method of the above-mentioned embodiments.

[0066] In the description of the present application, any process or method described in the flow chart or otherwise described herein can be understood as representing a module, a segment or a part of code including executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of the present application includes additional implementations in which the functions can be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0067] The logic and / or steps represented in the flowcharts and / or described herein, for example, can be considered as a sequence of instructions to implement logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device, such as a computer-based system, processor- based system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a computer- readable storage medium or a computer-readable signal medium. The computer- readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include the following: an electrical connection having one or more wires (electronic), a portable computer diskette (magnetic), a RAM (random access memory), a ROM (read-only memory), an EPROM (erasable programmable ROM), an EEPROM (electrically erasable programmable ROM), and a portable compact disc read-only memory (CD-ROM) (optical). In addition, the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via the optical scanner of a device or device or via the acoustical scanning of the paper or other medium, then electronically captured, interpreted, or processed in a suitable manner if necessary, and then stored in a computer memory.

[0068] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. As such, in some embodiments, the steps or methods can be implemented by a combination of hardware and software. If implemented in hardware, as in another embodiment, any of the above techniques can be implemented with or without the use of a programmable data processing apparatus, which can be a special purpose computer or general purpose computer. Such a computer can be programmed to perform one or more of the steps or methods described herein, and / or to perform one or more of the steps or methods described herein in combination with other steps or methods.

[0069] Those skilled in the art can understand that all or part of the steps carried out by the above-mentioned embodiment method can be completed by a program instructing the relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0070] In addition, each function unit in each embodiment of the present application can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The integrated module can be realized in the form of hardware, or in the form of a software function module. The integrated module, if realized in the form of a software function module and sold or used as an independent product, can also be stored in a computer readable storage medium.

[0071] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

[0072] In the description of the present specification, the description referring to the terms "one embodiment", "some embodiments", "exemplary embodiment", "example", "specific example" or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the exemplary description of the above terms does not necessarily mean the same embodiment or example.

[0073] Although the embodiments of the present application have been shown and described above, those skilled in the art can understand that various changes, modifications, replacements and variations can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the claims and their equivalents.

Claims

1. An audio / video data processing method, characterized in that, The method comprises: acquiring audio and video data; adding header attribute information to each frame of the audio and video data; writing each frame of the audio and video data carrying the header attribute information into the same buffer in sequence.

2. The audio-video data processing method of claim 1, wherein, The method of writing each frame of the audio and video data carrying the header attribute information into the same buffer in sequence comprises: writing each frame of the audio and video data carrying the header attribute information into the same buffer in sequence by using a serial communication mode.

3. The audio-video data processing method of claim 1, wherein, The header attribute information adopts a binary data structure.

4. The audio / video data processing method of claim 1, wherein, The data type of the audio and video data is a union type.

5. The audio / video data processing method of claim 1, wherein, The header attribute information at least comprises a timestamp.

6. The audio-video data processing method of claim 5, wherein, The audio and video data comprises audio data and video data, and the header attribute information of the audio data further comprises an audio coding format, a sampling rate and a data length, and the header attribute information of the video data further comprises a video coding format, a frame rate, a code rate and a data length.

7. The audio and video data processing method of any of claims 1-6, wherein, The buffer comprises a cache layer for storing audio and video data, and the method of writing each frame of the audio and video data carrying the header attribute information into the same buffer comprises: under the condition that there is a free cache space in the cache layer, writing each frame of the audio and video data carrying the header attribute information into the cache layer from a low address of the free cache space.

8. The audio and video data processing method according to any one of claims 1-6, characterized in that, The buffer is a circular buffer, and the buffer comprises a cache layer for storing audio and video data, and the method of writing each frame of the audio and video data carrying the header attribute information into the same buffer comprises: under the condition that there is no free cache space in the cache layer, sequentially clearing historical audio and video data stored at a low address of the cache layer from the low address; writing each frame of the audio and video data carrying the header attribute information into the address where the historical audio and video data have been cleared from the low address of the cache layer.

9. The audio and video data processing method of claim 1, wherein, The buffer at least comprises an interface layer for providing a read-write interface, and the method of acquiring audio and video data comprises: calling any interface in the interface layer that is not occupied; acquiring the audio and video data through the any interface.

10. The audio / video data processing method of claim 9, wherein, The buffer further comprises a management layer for managing audio and video data and a cache layer for storing audio and video data, and the method further comprises: receiving a read data instruction; calling one of the interfaces in the interface layer that is not occupied; the management layer sequentially reads the audio and video data in the cache layer through the one interface.

11. The audio video data processing method of claim 10, wherein, The method further comprises: labeling a key frame for a first frame of the audio and video data; under the condition that the reading speed of the one interface is lower than the writing speed of the any interface, acquiring a current reading position of the one interface; when it is determined that the current reading position is abnormal, jumping the data reading position of the one interface from the current reading position to an address position where the key audio and video data is located.

12. The audio video data processing method of claim 11, wherein, The method further comprises: under the condition that the reading speed of the one interface is lower than the writing speed of the any interface, allowing each frame of the audio and video data read to be copied out of the cache layer.

13. An electronic device, comprising: The device comprises: at least one processor; a memory in communication connection with the at least one processor; The memory stores a computer program executable by the at least one processor, and the at least one processor executes the computer program to implement the audio and video data processing method in any one of claims 1-12.

14. A computer storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the audio and video data processing method in any one of claims 1-12.

Images