Audio and video data storage method, device and computer equipment

Abstract

The application relates to a storage method and device of audio and video data, a computer device, a storage medium and a computer program product. The method comprises the following steps: receiving audio and video data, and obtaining a file starting identifier; in the case that the file starting identifier is a first state and the audio and video data is I frame data, writing the TS stream of the I frame data into a ring buffer, switching the file starting identifier from the first state to a second state; in the case that the file starting identifier is the second state and the usage rate of the ring buffer does not reach an upper limit value, writing the TS stream of the audio and video data into the ring buffer until the usage rate of the ring buffer reaches the upper limit value; in the case that the usage rate of the ring buffer reaches the upper limit value, discarding target data in the audio and video data; in the case that the discarded target data is P frame data, determining a picture group of the ring buffer, and uploading corresponding slice data to a server. The method can improve the efficiency of caching audio and video data.

Description

Methods, devices and computer equipment for storing audio and video data Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, computer equipment, storage medium, and computer program product for storing audio and video data. Background Technology

[0002] With the rapid development of smart cities and intelligent transportation, the security video surveillance field has generated massive amounts of audio and video data.

[0003] Currently, the large-scale networking of video surveillance, the development of high-definition and ultra-high-definition video, and improvements in intelligent video analysis technology have led to an increasing number of monitoring points and larger video bitrates in video surveillance systems. This has resulted in an explosive growth in security audio and video data, requiring longer storage times. However, when storing large amounts of audio and video data on cloud servers, if the data buffer reaches its cache limit, it often becomes impossible to effectively process newly received audio and video data. This makes caching audio and video data extremely difficult, thus impacting data storage.

[0004] Therefore, traditional technologies suffer from low storage efficiency for audio and video data. Summary of the Invention

[0005] Therefore, it is necessary to provide an audio and video data storage method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the efficiency of audio and video data storage in addressing the aforementioned technical problems.

[0006] A method for storing audio and video data, comprising:

[0007] Receive audio and video data from the receiving device and obtain the file start identifier of the TS file;

[0008] When the file start identifier is in the first state and the audio and video data are I-frame data, the TS stream of the I-frame data is written to the circular buffer, and the file start identifier is switched from the first state to the second state.

[0009] If the file start identifier is in the second state and the usage rate of the circular buffer has not reached the upper limit, write the TS stream of audio and video data to the circular buffer until the usage rate of the circular buffer reaches the upper limit.

[0010] When the usage rate of the circular buffer reaches its upper limit, the target data in the audio and video data is discarded; the target data includes data in the audio and video data other than I-frame data.

[0011] If the target data to be discarded is P-frame data, determine the frame group in the circular buffer and upload the corresponding fragment data of the frame group to the server.

[0012] In one embodiment, when the file start identifier is in the second state and the usage rate of the circular buffer has not reached the upper limit, the method further includes:

[0013] Get the fragment count identifier corresponding to the fragmented data;

[0014] Based on the fragment count identifier, determine whether the buffered data in the circular buffer has reached the data of a frame group;

[0015] If the cached data reaches the size of a screen group, the cached data will be uploaded to the server.

[0016] In one embodiment, prior to the step of writing the TS stream of audio and video data to the circular buffer, the method further includes:

[0017] Get the starting position of the used buffer in the circular buffer, the length of the used buffer in the circular buffer, the starting position of the buffer in the circular buffer, and the ending position of the buffer in the circular buffer;

[0018] Based on the start position of the used buffer, the length of the used buffer, the start position of the buffer, and the end position of the buffer, determine the current write position of the TS stream of audio and video data to the circular buffer.

[0019] In one embodiment, it further includes:

[0020] If the file start identifier is in the first state and the usage rate of the circular buffer has not reached the upper limit, a forced I-frame error code is returned to the device. The forced I-frame error code is used to indicate that the next frame of data that the device needs to push is I-frame data.

[0021] If the file start identifier is in the first state and the usage rate of the circular buffer reaches the upper limit, a buffer limit reached error code is returned to the device; the buffer limit reached error code is used to instruct the device to adjust the frequency of pushing TS streams.

[0022] In one embodiment, it further includes:

[0023] If the file start identifier is in the second state and the usage rate of the circular buffer reaches the upper limit, return a buffer limit reached error code to the device.

[0024] In one embodiment, uploading the segmented data corresponding to the screen group to the server includes:

[0025] Acquire the start position marker and length marker of the first part of the frame group data in the circular buffer; acquire the start position marker and length marker of the second part of the frame group data in the circular buffer; acquire the timestamp marker of the first frame data and the timestamp marker of the last frame data of the frame group.

[0026] Place the start position marker and length marker of the first part of the data, the start position marker and length marker of the second part of the data, the timestamp marker of the first frame of data, and the timestamp marker of the last frame of data into the structure queue;

[0027] Based on the tags in the structure queue, the segmented data corresponding to the screen group data is uploaded to the server.

[0028] In one embodiment, it further includes:

[0029] Based on the markers in the structure queue, clear the buffer in the circular buffer that caches the data of the screen group;

[0030] Once the buffer has been cleared, determine the current write position of the data for the next frame group in the circular buffer.

[0031] A storage device for audio and video data, comprising:

[0032] The receiving module is used to receive audio and video data transmitted from the device and to obtain the file start identifier of the TS file;

[0033] The switching module is used to write the TS stream of I-frame data to the circular buffer when the file start identifier is in the first state and the audio and video data is I-frame data, and to switch the file start identifier from the first state to the second state.

[0034] The write module is used to write the TS stream of audio and video data to the circular buffer when the file start identifier is in the second state and the usage rate of the circular buffer has not reached the upper limit, until the usage rate of the circular buffer reaches the upper limit.

[0035] The discard module is used to discard target data in the audio and video data when the usage rate of the circular buffer reaches the upper limit; the target data includes data in the audio and video data other than I-frame data.

[0036] The determination module is used to determine the frame group in the circular buffer when the target data to be discarded is P-frame data, and upload the fragment data corresponding to the frame group to the server.

[0037] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0038] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described above.

[0039] The aforementioned audio and video data storage method, apparatus, computer equipment, storage medium, and computer program product, by receiving audio and video data transmitted from the receiving device and obtaining the file start identifier of the TS file; when the file start identifier is in a first state and the audio and video data is I-frame data, writing the TS stream of I-frame data into a circular buffer, and switching the file start identifier from the first state to a second state; when the file start identifier is in the second state and the usage rate of the circular buffer has not reached the upper limit, writing the TS stream of audio and video data into the circular buffer until the usage rate of the circular buffer reaches the upper limit; when the usage rate of the circular buffer reaches the upper limit, discarding target data in the audio and video data; target data includes data in the audio and video data other than I-frame data; when the discarded target data is P-frame data, determining the frame group of the circular buffer, and uploading the segment data corresponding to the frame group to the server; thus, efficient management of the circular buffer memory can be achieved, and the frame dropping processing performed when caching the TS stream in the embedded device system can ensure the audio and video quality of the audio and video data transmitted to the server, which is beneficial for efficient caching of audio and video data. Attached Figure Description

[0040] Figure 1 is an application environment diagram of an audio and video data storage method in one embodiment;

[0041] Figure 2 is a flowchart illustrating a method for storing audio and video data in one embodiment;

[0042] Figure 3 is a flowchart of the storage process of audio and video data in one embodiment;

[0043] Figure 4 is a schematic diagram of a circular buffer structure in one embodiment;

[0044] Figure 5 is a flowchart illustrating a method for buffering TS streams and handling frame dropping in one embodiment.

[0045] Figure 6 is a flowchart illustrating a method for storing audio and video data in another embodiment;

[0046] Figure 7 is a structural block diagram of an audio and video data storage device in one embodiment;

[0047] Figure 8 is an internal structure diagram of a computer device in one embodiment. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0049] The audio and video data storage method provided in this application embodiment can be applied to the application environment shown in Figure 1. The embedded device 102 communicates with the server 104 via a network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated onto the server 104, or it can be located in the cloud or on another network server. Embedded device 102 receives audio and video data from the device side and obtains the file start identifier of the TS file. When the file start identifier is in the first state and the audio and video data is I-frame data, embedded device 102 writes the TS stream of the I-frame data to the circular buffer and switches the file start identifier from the first state to the second state. When the file start identifier is in the second state and the usage rate of the circular buffer has not reached the upper limit, embedded device 102 writes the TS stream of the audio and video data to the circular buffer until the usage rate of the circular buffer reaches the upper limit. When the usage rate of the circular buffer reaches the upper limit, embedded device 102 discards target data in the audio and video data; target data includes data in the audio and video data other than I-frame data. When the discarded target data is P-frame data, embedded device 102 determines the frame group in the circular buffer and uploads the corresponding segment data of the frame group to the server. Embedded device 102 can be, but is not limited to, various network cameras, network hard drive cameras, solar 4G panels, etc. Server 104 can be implemented using a standalone server or a server cluster composed of multiple servers.

[0050] In one embodiment, as shown in FIG2, a method for storing audio and video data is provided. Taking the application of this method to the embedded device 102 in FIG1 as an example, the method includes the following steps:

[0051] Step S202: Receive audio and video data from the device and obtain the file start identifier of the TS file.

[0052] The audio and video data can be video data in H264 / H265 format and audio data in AAC format.

[0053] Among them, TS files can be corresponding format files formed by slicing and packaging audio and video data.

[0054] The file start identifier can be a start identifier used to identify a GOP fragment.

[0055] In practice, the embedded device receives audio and video data from the device and obtains the file start identifier of the TS file. In actual applications, the file start identifier of the TS file is initialized to 0, and it changes depending on the data frame attributes of the incoming audio and video data and the usage rate of the circular buffer.

[0056] Step S204: When the file start identifier is in the first state and the audio and video data is I-frame data, write the TS stream of the I-frame data to the circular buffer and switch the file start identifier from the first state to the second state.

[0057] The first state can be a status identifier indicating that only I-frame data can be received at present. The first state can be represented by the identifier 1.

[0058] The second state can be a status identifier representing the current ability to receive audio frame data, I-frame data, P-frame data, and B-frame data. The second state can be represented by the identifier 0.

[0059] In the specific implementation, when the file start identifier is 1 and the incoming audio and video data is I-frame data, the embedded device slices the I-frame data and writes the TS stream of the I-frame data to the circular buffer. In addition, the embedded device switches the file start identifier of the TS file to 0.

[0060] Step S206: When the file start identifier is in the second state and the usage rate of the circular buffer has not reached the upper limit, write the TS stream of audio and video data to the circular buffer until the usage rate of the circular buffer reaches the upper limit.

[0061] The upper limit refers to the usable length of the circular buffer calculated as a preset percentage based on the total length of the circular buffer. This upper limit is used for frame dropping in weak network environments. For example, if the used length of the circular buffer is less than the upper limit, the raw audio and video streams are sliced ​​into TS segments, and the sliced ​​TS streams are written to the circular buffer. If the used length of the circular buffer is greater than or equal to the upper limit, only video I-frames are processed; the I-frames are sliced ​​into TS segments and written to the circular buffer, while audio frames and non-I-video frames are dropped.

[0062] In practice, when the file start identifier is 0 and the usage rate of the circular buffer has not reached the upper limit, the embedded device slices the incoming audio and video data and writes the sliced ​​data into the circular buffer until the usage rate of the circular buffer reaches the upper limit.

[0063] Step S208: When the usage rate of the circular buffer reaches the upper limit, discard the target data in the audio and video data; the target data includes data in the audio and video data other than I-frame data.

[0064] The target data can be audio frame data, P frame data, and B frame data, excluding I frame data.

[0065] In the specific implementation, when the usage rate of the circular buffer reaches the upper limit, if the incoming audio and video data is not I-frame data, the incoming data frame will be discarded. If the incoming audio and video data frame is I-frame data, the embedded device will slice the incoming I-frame data and write it into the circular buffer.

[0066] Step S210: If the target data to be discarded is P-frame data, determine the frame group of the circular buffer and upload the fragment data corresponding to the frame group to the server.

[0067] Here, a group of frames can refer to a group of frames (GOPs), which refers to the data between two I-frames in the audio and video data.

[0068] The fragmented data can be the data of each frame in a GOP (Group of Pictures).

[0069] In practice, when the embedded device determines that the discarded data is P-frame data, it forcibly terminates a GOP data and uploads the GOP data to the server.

[0070] To facilitate understanding by those skilled in the art, Figure 3 provides an exemplary flowchart of audio and video data storage. This flowchart can be described as follows:

[0071] When raw audio and video streams are input, the embedded device will separate them into audio raw streams and video raw streams.

[0072] For raw audio streams, the embedded device determines whether the usage rate of the circular buffer has reached its upper limit. If it has, the frame is discarded; otherwise, the frame is sliced ​​into TS segments and placed into the circular buffer, and the data is subsequently uploaded to the cloud via an upload thread.

[0073] For raw video streams, the embedded device determines whether the currently incoming data frame is an I-frame. If it is an I-frame, it checks if a GOP (Group of Pictures) is full. Once a GOP is full, it notifies the upload thread to upload the data to the cloud. If it is not an I-frame, it checks if the circular buffer has reached its upper limit. If it has, it checks if it is a P-frame. If it is, the P-frame is discarded, a GOP is forcibly terminated, and the data is uploaded to the cloud via the upload thread. If the upper limit has not been reached, the frame is sliced ​​into TS (Time-of-Sight) segments and placed into the circular buffer, and subsequently uploaded to the cloud via the upload thread.

[0074] Such embedded devices can store raw audio and video streams.

[0075] In practical applications, embedded device systems first request a certain number of bytes of memory and manage this memory using a circular buffer. The embedded device will perform different processing on the audio and video data based on the cache status flags. The cache status flags include the file start flag of the TS file and the fragment count flag used for uploading fragmented data.

[0076] For the start-of-frame (TOF) identifier of a TS file, when the TOF identifier is set to 1, the data frame pushed by the current device must be an I-frame before it will be TS-sliced ​​and written to the circular buffer. Other non-I-frame data will be discarded, and a forced I-frame error code will be returned to the device to indicate that the next frame to be pushed by the device must be an I-frame. After the I-frame data is successfully written to the circular buffer, the embedded device will set the TOF identifier of the TS file to 0. When the usage rate of the circular buffer has not reached the upper limit, the received audio and video data will be TS-sliced ​​and written to the circular buffer. When the usage rate of the circular buffer reaches the upper limit, only the sliced ​​I-frame data will be cached, and audio and video frames and non-I-frame video frames will be discarded. When the discarded video frame is a P-frame, a GOP data will be forcibly terminated, and the upload thread will be notified to read the GOP data from the circular buffer to the server.

[0077] For the fragment count flag of uploaded fragment data, the initial value of the fragment count flag is 0. If the data frame passed from the device is an I-frame, the fragment count flag is incremented by 1. The value of the fragment count flag is retrieved. When the value of the fragment count flag is 1, a GOP video data is terminated, the fragment count flag is reset to 0, and the GOP video data is uploaded to the server.

[0078] The above-described method for storing audio and video data involves receiving audio and video data from the receiving device and obtaining the file start identifier of the TS file. When the file start identifier is in the first state and the audio and video data is I-frame data, the TS stream of the I-frame data is written to the circular buffer, and the file start identifier is switched from the first state to the second state. When the file start identifier is in the second state and the usage rate of the circular buffer has not reached its upper limit, the TS stream of the audio and video data is written to the circular buffer until the usage rate of the circular buffer reaches its upper limit. When the usage rate of the circular buffer reaches its upper limit, target data in the audio and video data is discarded. Target data includes data in the audio and video data other than I-frame data. When the discarded target data is P-frame data, the frame group in the circular buffer is determined, and the corresponding fragment data of the frame group is uploaded to the server. This allows for efficient management of the circular buffer's memory. The frame dropping processing performed when caching the TS stream in the embedded device system ensures the audio and video quality of the data transmitted to the server, which is beneficial for efficient caching of audio and video data.

[0079] In another embodiment, when the file start identifier is in the second state and the usage rate of the circular buffer has not reached the upper limit, the method further includes: obtaining the fragment count identifier corresponding to the fragment data; determining whether the cached data in the circular buffer has reached the data of a screen group based on the fragment count identifier; and if the cached data has reached the data of a screen group, uploading the cached data to the server.

[0080] The fragment count identifier can be an identifier used to indicate the number of I-frames received by a GOP fragment.

[0081] One of the data for a group of screens can be a group of pictures (GOPs).

[0082] In the specific implementation, the embedded device obtains the fragment count identifier corresponding to the fragmented data. The fragment count identifier is initialized to 0. When the frame data passed by the device is an I-frame, the embedded device increments the fragment count identifier by 1. When the fragment count identifier is 1, the embedded device ends a GOP data and uploads the GOP data to the cloud server, and resets the fragment count identifier to 0.

[0083] The technical solution of this embodiment obtains the segment count identifier corresponding to the segmented data, and uploads the cached data to the server after the cached data reaches the data of a frame group according to whether the cached data in the circular buffer has reached the data of a frame group. This realizes the transmission of audio and video data to the server and ensures the audio and video quality of the audio and video data.

[0084] In another embodiment, before writing the TS stream of audio and video data to the circular buffer, the method further includes: obtaining the start position of the used buffer in the circular buffer, the length of the used buffer in the circular buffer, the buffer start position of the circular buffer, and the buffer end position of the circular buffer; and determining the current write position of the TS stream of audio and video data to the circular buffer based on the start position of the used buffer, the length of the used buffer, the buffer start position, and the buffer end position.

[0085] The current write position can be the specific write position where the TS stream of audio and video data is written to the circular buffer.

[0086] In the specific implementation, before writing the TS stream of audio and video data to the circular buffer, the embedded device also needs to obtain the starting position of the used buffer in the circular buffer, the length of the used buffer in the circular buffer, the starting position of the buffer, and the ending position of the circular buffer. Based on the starting position of the used buffer, the length of the used buffer, the starting position of the buffer, and the ending position of the buffer, the current writing position of the TS stream of audio and video data to the circular buffer is determined.

[0087] In practical applications, the method for determining the current write position is as follows:

[0088] Step 1: Determine the end position of the circular buffer (denoted as buffer_end). End position (denoted as buffer_end) = start position of the circular buffer (denoted as buffer_start) + total length of the circular buffer (denoted as total_length).

[0089] Step 2: Determine the current position of the data currently written to the circular buffer (denoted as current). The current position of the data currently written (current) = the starting position of the used circular buffer (used_start) + the length of the used circular buffer (used_length).

[0090] Step 3: Compare the values ​​of buffer_end and current. If current is greater than or equal to buffer_end, then the value of current is current minus the total length of the circular buffer; if current is less than buffer_end, then the value of the current write position is current.

[0091] Step 4: Compare the difference between the end position of the circular buffer (buffer_end) and the current position. If the difference is greater than or equal to the length of data to be written, the current write position is set to current, and the entire length of data is written starting from the current position. If the difference is less than the length of data to be written, the data length between buffer_end and current is written starting from the current position, and the remaining length of data is written starting from the beginning position of the circular buffer.

[0092] When writing to the cache, if the length of the data being written is greater than the remaining available length of the cache, a cache usage limit error code is returned to the device. The device can adjust the push frequency based on the return value, and the data will not be written to the cache.

[0093] The method for determining the current write position described above can also be described as follows:

[0094] Step 1: Get the sum S of the starting position of the used cache and the length of the used cache.

[0095] Step 2: If the sum S is greater than or equal to the end position of the circular buffer, then the current write buffer position C is equal to S minus the total length of the circular buffer; if the sum S is less than the end position of the circular buffer, then the current write buffer position C is equal to S.

[0096] Step 3: Based on the current write buffer position C, obtain the difference D between the end position of the circular buffer and the current write buffer position C.

[0097] Step 4: If the difference D is greater than or equal to the length L of the TS stream data, then copy L bytes to the current write position C of the circular buffer; if the difference D is less than the length L of the TS stream data, then first copy D bytes to the current write position C of the circular buffer, and copy the remaining LD bytes to the beginning position of the circular buffer.

[0098] Step 5: After successfully writing the sliced ​​TS stream to the circular buffer, the used length of the buffer needs to be updated. The used length of the buffer = the original used length of the buffer + the length of the TS stream.

[0099] The technical solution of this embodiment determines the current write position of the TS stream of audio and video data to the circular buffer by taking the starting position of the used buffer in the circular buffer, the length of the used buffer in the circular buffer, the starting position of the buffer and the ending position of the circular buffer. This can accurately determine the current write position of the TS stream, which is beneficial to improving the storage efficiency of audio and video data.

[0100] In another embodiment, the method further includes: when the file start identifier is in the first state and the usage rate of the circular buffer has not reached the upper limit, returning a forced I-frame error code to the device; the forced I-frame error code is used to indicate that the next frame of data to be pushed by the device is I-frame data; when the file start identifier is in the first state and the usage rate of the circular buffer has reached the upper limit, returning a buffer limit reached error code to the device; the buffer limit reached error code is used to instruct the device to adjust the frequency of pushing TS streams.

[0101] In the specific implementation, if the file start identifier is 1 and the usage rate of the circular buffer has not reached the upper limit, the embedded device returns a forced I-frame error code to the device side. The device side can find the I-frame data based on the return value of the forced I-frame error code. If the file start identifier is not 1 and the usage rate of the circular buffer has reached the upper limit, the embedded device returns a buffer limit reached error code to the device side. The device side can adjust the push frequency based on the return value of the buffer limit reached error code.

[0102] In this embodiment, when the file start identifier is in the first state and the usage rate of the circular buffer has not reached the upper limit, a forced I-frame error code is returned to the device. This indicates that the next frame of data that the device needs to push is I-frame data. When the file start identifier is in the first state and the usage rate of the circular buffer has reached the upper limit, a buffer limit error code is returned to the device. This indicates that the device can adjust the frequency of pushing TS streams. In this case, not returning a forced I-frame error code to the device can avoid the device from excessively searching for I-frame data. When the usage rate of the circular buffer reaches the upper limit, frame dropping processing can be performed on the incoming audio and video data, which is beneficial for high-quality storage of audio and video data.

[0103] In another embodiment, the method further includes: returning a cache limit reached error code to the device when the file start identifier is in the second state and the usage rate of the circular buffer reaches the upper limit.

[0104] In the specific implementation, when the file start mark is 0 and the usage rate of the circular buffer reaches the upper limit, the embedded device returns a buffer limit reached error code to the device side. The device side adjusts the push frequency according to the return value of the buffer limit reached error code.

[0105] In this embodiment, when the file start mark is in the second state and the usage rate of the circular buffer reaches the upper limit, a buffer limit error code is returned to the device. The device can adjust the streaming frequency according to the return value of the buffer limit error code. At this time, not returning a forced I-frame error code to the device can avoid the device from excessively searching for I-frame data, thus realizing frame dropping processing of the incoming audio and video data, which is beneficial to the high-quality storage of audio and video data.

[0106] In another embodiment, uploading the segmented data corresponding to the screen group to the server includes: obtaining the start position marker and length marker of the first part of the screen group data in the circular buffer; obtaining the start position marker and length marker of the second part of the screen group data in the circular buffer; and obtaining the timestamp marker of the first frame data and the timestamp marker of the last frame data of the screen group; placing the start position marker and length marker of the first part data, the start position marker and length marker of the second part data, the timestamp marker of the first frame data and the timestamp marker of the last frame data into a structure queue; and uploading the segmented data corresponding to the screen group data to the server according to the markers in the structure queue.

[0107] In one scenario, when the data of a frame group is cached in a circular buffer, there is a situation where the data of a frame group needs to be cached in two parts, namely, the data of a frame group is divided into a first part and a second part.

[0108] In specific implementation, when an embedded device uploads a GOP (Group of Pictures) of data to the server, it needs to determine the start position marker and length marker of the first part of the GOP data in the circular buffer, as well as the start position marker and length marker of the second part of the frame group data in the circular buffer, and the timestamp markers of the first and last frames of the frame group. Then, the embedded device puts the start position marker and length marker of the first part of the data, the start position marker and length marker of the second part of the data, the timestamp marker of the first frame data, and the timestamp marker of the last frame data into a structure queue. Based on the above markers in the structure queue, the embedded device uploads the GOP data to the cloud storage server.

[0109] Figure 4 provides an exemplary schematic diagram of a circular buffer structure. After the location information of a GOP (Group of Pictures) is placed into the structure queue, the `last_flush` pointer points to the end position of the GOP data in the circular buffer. After the GOP data is successfully uploaded to the cloud storage server or a timeout occurs, the `used_start` pointer points to the end position of the GOP data in the circular buffer, and `used_length` needs to be subtracted from the length of the GOP data.

[0110] In practical applications, it is necessary to determine the location of the GOP data in the circular buffer. The specific steps are as follows:

[0111] Step 1: Calculate the current write position. The current write position (denoted as current) is determined by the starting position of the used length of the circular buffer (denoted as used_start) and the used length of the circular buffer (denoted as used_length), that is, current = used_start + used_length.

[0112] Step 2: Determine the end position of the last uploaded data (denoted as last_flush).

[0113] Step 3: Compare the current write position (current) with the last upload position (last_flush).

[0114] If last_flush is less than current, it indicates that the GOP data does not cross the boundary of the circular buffer. The starting position of the first part of the GOP data in the circular buffer is equal to the ending position of the last uploaded data last_flush. The length of the first part of the data is equal to the current write position current minus the ending position of the last uploaded data last_flush. The position of the second part of the data is empty and the length is 0.

[0115] If last_flush is greater than current, it indicates that the GOP data crosses the boundary of the circular buffer. The starting position of the first part of the GOP data in the circular buffer is equal to the ending position of the last uploaded data (last_flush). The length of the first part of the data is equal to the starting position of the circular buffer (buffer_start) plus the total length of the circular buffer (total_length) minus the ending position of the last uploaded data (last_flush). The starting position of the second part of the data is equal to the starting position of the circular buffer (buffer_start). The length of the second part of the data is equal to the current write position in the buffer (current) minus the starting position of the circular buffer (buffer_start).

[0116] Step 4: Obtain the timestamp of the first frame and the timestamp of the last frame of the GOP data.

[0117] Step 5: Place the starting position of the first part of the GOP data in the cache, the length of the first part of the data, the starting position of the second part of the data in the cache, the length of the second part of the data, the timestamp of the first frame, and the timestamp of the last frame into a structure queue.

[0118] Step 6: Obtain the index of the position to be written to the structure queue, which is the latest accessed index of the structure queue (denoted as queue_pos) plus the length of the structure queue (denoted as queue_length). If this value is greater than the maximum length of the structure, subtract the maximum length of the structure from this value. After successfully putting the structure into the structure queue, increment the used length of the structure queue (queue_length) by 1, and notify the upload thread to retrieve these markers from the structure queue.

[0119] Step 7: Obtain the GOP data based on the markers from Step 6 and upload it to the cloud storage server. Obtain the GOP data based on the start position marker of the first part of the data in the circular buffer, the length marker of the first part of the data, the start position marker of the second part of the data in the circular buffer, and the length marker of the second part of the data; and obtain the TS filename based on the timestamp markers generated by the first and last frames of the GOP data. Use the timestamp generated by the first frame of the GOP data as the start time and the timestamp generated by the last frame as the end time, name the TS filename in the format of start time_end time, and upload the GOP data and the TS filename to the cloud storage server.

[0120] To facilitate understanding by those skilled in the art, Figure 5 provides an exemplary flowchart of a method for buffering TS streams and handling frame loss. The specific process can be described as follows:

[0121] Step 1: Input raw audio and video stream data from the embedded device.

[0122] Step 2: Obtain the TS file start identifier and the length of memory used in the circular buffer. If the TS file start identifier is 0 and the length of memory used has not reached the upper limit, the received audio and video raw stream data is TS-sliced ​​and placed into the circular buffer. If the TS file start identifier is 1 or the length of memory used has reached the upper limit, the incoming data must be an I-frame. The I-frame is TS-sliced ​​and placed into the buffer. Other audio frames and non-I-frame video frames will be discarded. If the GOP segment counter value is 1 or the discarded data is a P-frame, the TS file start identifier is reset to 0, and this GOP data is processed.

[0123] Step 3: Obtain the start position of the first part of the GOP data in the circular buffer, the length of the first part of the data, the start position of the second part of the data, the length of the second part of the data, the timestamp of the first video frame of the GOP data, and the timestamp of the last video frame of the GOP data, and put these markers into the structure queue.

[0124] Step 4: Notify the upload thread to retrieve the GOP data from the queue, including the starting position and length of the cached data, as well as the timestamps of the first and last frames, and upload the GOP data and the TS file name to the cloud storage server.

[0125] Step 5: After the upload to the cloud storage server is successful or times out, recalculate the starting position of the cache already used and the length of the cache already used.

[0126] The technical solution of this embodiment obtains the start position marker and length marker of the first part of the GOP data in the circular buffer, determines the start position marker and length marker of the second part of the frame group data in the circular buffer, and determines the timestamp marker of the first frame data and the timestamp marker of the last frame data of the frame group. Then, the start position marker and length marker of the first part of the data, the start position marker and length marker of the second part of the data, the timestamp marker of the first frame data and the timestamp marker of the last frame data are placed into a structure queue. According to the above markers in the structure queue, the GOP data is uploaded to the cloud storage server, thus achieving high-quality storage of audio and video data.

[0127] In another embodiment, the method further includes: clearing the buffer containing data of the screen group in the circular buffer according to each tag in the structure queue; and, after the buffer clearing is completed, determining the current write position of the data of the next screen group in the circular buffer.

[0128] In practice, after the embedded device uploads GOP data to the cloud storage server or times out, it needs to readjust the identifiers of the circular buffer and the structure queue, clear the GOP data cached in the circular buffer, and determine the current write position of the next screen group after the buffer data has been cleared.

[0129] In practical applications, after successfully uploading GOP data to the cloud storage server or after a timeout, it is necessary to readjust some flags of the circular buffer and the structure queue. The specific steps are as follows:

[0130] Step 1: Decrement the used length (queue_length) of the structure queue by 1, and increment the latest accessed index (queue_pos) by 1.

[0131] Step 2: Based on the starting position marker, length marker, and starting position marker of the first part of the GOP data within the circular buffer, and the length marker of the second part of the data within the circular buffer, clear the buffer occupied by the GOP data in the circular buffer. Note that only sequential buffer clearing is supported; clearing the middle portion of the buffer in use is not supported.

[0132] Step 3: Calculate the position of the next write to the buffer (next_start). If the second part of the data is empty at the beginning of the circular buffer, the position of the next write to the circular buffer is equal to the position of the first part of the data at the beginning of the circular buffer plus the length of the first part of the data; otherwise, the position of the next write to the circular buffer is equal to the position of the second part of the data at the beginning of the circular buffer plus the length of the second part of the data.

[0133] Step 4: Compare the value of the next write position to the buffer (denoted as next_start) with the starting position of the used length of the circular buffer (denoted as used_start).

[0134] If next_start is greater than used_start, then the used length of the circular buffer (used_length) is equal to next_start minus used_start; if next_start is less than or equal to used_start, then the used length of the circular buffer (used_length) is equal to the total length of the circular buffer (total_length) - (used_start - next_start), and the starting position of the used length of the circular buffer (used_start) is equal to (next_start).

[0135] If the value of used_start is greater than the end position of the circular buffer, then the end position of the circular buffer is the start position of the circular buffer (buffer_start) plus the total length of the circular buffer (total_length). The value of the used start position (used_start) of the circular buffer is equal to (used_start) minus the total length of the circular buffer (total_length).

[0136] The technical solution of this embodiment clears the buffer containing the data of the next video group in the circular buffer according to the markers in the structure queue. After the buffer is cleared, the current writing position of the data of the next video group in the circular buffer is determined, so that the circular buffer can efficiently store audio and video data and improve the caching efficiency of audio and video data.

[0137] In another embodiment, as shown in FIG6, a method for storing audio and video data is provided. Taking the application of this method to the embedded device 102 in FIG1 as an example, the method includes the following steps:

[0138] Step S602: Receive audio and video data from the device and obtain the file start identifier of the TS file.

[0139] Step S604: When the file start identifier is in the first state and the audio / video data is I-frame data, write the TS stream of the I-frame data to the circular buffer and switch the file start identifier from the first state to the second state.

[0140] Step S606: When the file start identifier is in the second state and the usage rate of the circular buffer has not reached the upper limit, obtain the starting position of the used cache in the circular buffer, the usage length of the used cache in the circular buffer, the buffer start position of the circular buffer, and the buffer end position of the circular buffer.

[0141] Step S608: Determine the current write position of the TS stream of the audio and video data to the circular buffer based on the start position of the used buffer, the length of the used buffer, the start position of the buffer, and the end position of the buffer.

[0142] Step S610: Write the TS stream of the audio and video data to the circular buffer until the usage rate of the circular buffer reaches the upper limit.

[0143] Step S612: If the usage rate of the circular buffer reaches the upper limit, discard the target data in the audio and video data; the target data includes data in the audio and video data other than I-frame data.

[0144] Step S614: If the discarded target data is P-frame data, determine the frame group of the circular buffer and upload the fragment data corresponding to the frame group to the server.

[0145] It should be noted that the specific limitations of the above steps can be found in the specific limitations of a method for storing audio and video data described above.

[0146] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0147] Based on the same inventive concept, this application also provides an audio / video data storage device for implementing the audio / video data storage method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more audio / video data storage device embodiments provided below can be found in the limitations of the audio / video data storage method described above, and will not be repeated here.

[0148] In one embodiment, as shown in FIG7, an audio and video data storage device is provided, comprising:

[0149] The receiving module 702 is used to receive audio and video data transmitted from the device and to obtain the file start identifier of the TS file;

[0150] The switching module 704 is used to write the TS stream of I-frame data to the circular buffer when the file start identifier is in the first state and the audio and video data is I-frame data, and to switch the file start identifier from the first state to the second state.

[0151] The writing module 706 is used to write the TS stream of audio and video data to the circular buffer when the file start identifier is in the second state and the usage rate of the circular buffer has not reached the upper limit, until the usage rate of the circular buffer reaches the upper limit.

[0152] The discard module 708 is used to discard target data in the audio and video data when the usage rate of the circular buffer reaches the upper limit; the target data includes data in the audio and video data other than I-frame data.

[0153] The determination module 710 is used to determine the frame group of the circular buffer when the discarded target data is P-frame data, and upload the fragment data corresponding to the frame group to the server.

[0154] In one embodiment, the writing module 706 is specifically used to obtain the fragment count identifier corresponding to the fragment data when the file start identifier is in the second state and the usage rate of the circular buffer has not reached the upper limit; determine whether the cached data in the circular buffer has reached the data of a screen group based on the fragment count identifier; if the cached data has reached the data of a screen group, then upload the cached data to the server.

[0155] In one embodiment, the writing module 706 is specifically used to obtain, before the step of writing the TS stream of audio and video data to the circular buffer, the starting position of the used buffer in the circular buffer, the length of the used buffer in the circular buffer, the starting position of the buffer and the ending position of the buffer in the circular buffer; and determine the current writing position of the TS stream of audio and video data to the circular buffer based on the starting position of the used buffer, the length of the used buffer, the starting position of the buffer and the ending position of the buffer.

[0156] In one embodiment, the apparatus further includes: a return module, specifically configured to return a forced I-frame error code to the device when the file start identifier is in the first state and the usage rate of the circular buffer has not reached the upper limit; the forced I-frame error code is used to indicate that the next frame of data to be pushed by the device is I-frame data; and to return a buffer limit reached error code to the device when the file start identifier is in the first state and the usage rate of the circular buffer has reached the upper limit; the buffer limit reached error code is used to instruct the device to adjust the frequency of pushing TS streams.

[0157] In one embodiment, the apparatus further includes a return module, specifically configured to return a cache limit reached error code to the device when the file start identifier is in the second state and the usage rate of the circular buffer reaches the upper limit.

[0158] In one embodiment, the determining module 710 is specifically used to obtain the start position marker and length marker of the first part of the screen group data in the circular buffer, and to obtain the start position marker and length marker of the second part of the screen group data in the circular buffer, and to obtain the timestamp marker of the first frame data and the timestamp marker of the last frame data of the screen group; to put the start position marker and length marker of the first part data, the start position marker and length marker of the second part data, the timestamp marker of the first frame data and the timestamp marker of the last frame data into a structure queue; and to upload the segmented data corresponding to the screen group data to the server according to the markers in the structure queue.

[0159] In one embodiment, the determining module 710 is specifically used to clear the buffer containing the data of the screen group in the circular buffer according to each mark in the structure queue; and after the buffer is cleared, to determine the current write position of the data of the next screen group in the circular buffer.

[0160] Each module in the aforementioned audio and video data storage device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0161] In one embodiment, a computer device, which may be a server, is provided, and its internal structure is shown in Figure 8. The computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores audio and video data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for storing audio and video data.

[0162] Those skilled in the art will understand that the structure shown in Figure 8 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or may combine certain components, or may have different component arrangements.

[0163] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, which, when executed by the processor, causes the processor to perform the steps of the audio / video data storage method described above. The steps of the audio / video data storage method described here can be steps from the audio / video data storage methods of the various embodiments described above.

[0164] In one embodiment, a computer-readable storage medium is provided, storing a computer program that, when executed by a processor, causes the processor to perform the steps of the audio / video data storage method described above. The steps of the audio / video data storage method described here may be steps from the audio / video data storage methods of the various embodiments described above.

[0165] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, causes the processor to perform the steps of the audio / video data storage method described above. The steps of the audio / video data storage method described here may be steps from the audio / video data storage methods of the various embodiments described above.

[0166] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0167] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0168] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0169] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for storing audio and video data, characterized in that, The method includes: receiving audio and video data from a device and obtaining a file start identifier of a TS file; when the file start identifier is in a first state and the audio and video data is I-frame data, writing the TS stream of the I-frame data to a circular buffer, and switching the file start identifier from the first state to a second state; the first state is used to indicate that only I-frame data can be received at present; the second state is used to indicate that audio frame data, I-frame data, P-frame data, and B-frame data can be received at present; when the file start identifier is in the second state and the usage rate of the circular buffer has not reached the upper limit, writing the TS stream of the audio and video data to the circular buffer until the usage rate of the circular buffer reaches the upper limit; when the usage rate of the circular buffer reaches the upper limit, discarding target data in the audio and video data; the target data includes the audio and video data. Based on the data excluding I-frame data; if the discarded target data is P-frame data, determine the frame group of the circular buffer, obtain the start position marker and length marker of the first part of the frame group data in the circular buffer, obtain the start position marker and length marker of the second part of the frame group data in the circular buffer, and obtain the timestamp marker of the first frame data and the timestamp marker of the last frame data of the frame group; put the start position marker and length marker of the first part data, the start position marker and length marker of the second part data, the timestamp marker of the first frame data and the timestamp marker of the last frame data into a structure queue; according to each marker in the structure queue, upload the fragment data corresponding to the frame group data to the server.

2. The method according to claim 1, characterized in that, When the file start identifier is in the second state and the usage rate of the circular buffer has not reached the upper limit, the method further includes: obtaining the fragment count identifier corresponding to the fragment data; determining whether the cached data of the circular buffer has reached the data of a screen group based on the fragment count identifier; if the cached data has reached the data of a screen group, then uploading the cached data to the server.

3. The method according to claim 1, characterized in that, Before the step of writing the TS stream of the audio and video data to the circular buffer, the method further includes: obtaining the starting position of the used buffer in the circular buffer, the used length of the used buffer in the circular buffer, the buffer start position of the circular buffer, and the buffer end position of the circular buffer; and determining the current writing position of the TS stream of the audio and video data to the circular buffer based on the starting position of the used buffer, the used length of the used buffer, the buffer start position, and the buffer end position.

4. The method according to claim 1, characterized in that, The method further includes: when the file start identifier is in the first state and the usage rate of the circular buffer has not reached the upper limit, returning a forced I-frame error code to the device; the forced I-frame error code is used to indicate that the next frame of data that the device needs to push is I-frame data; when the file start identifier is in the first state and the usage rate of the circular buffer has reached the upper limit, returning a buffer limit reached error code to the device; the buffer limit reached error code is used to instruct the device to adjust the frequency of pushing the TS stream.

5. The method according to claim 1, characterized in that, The method further includes: when the file start identifier is in the second state and the usage rate of the circular buffer reaches the upper limit, returning a cache limit reached error code to the device.

6. The method according to claim 1, characterized in that, The method further includes: clearing the buffer containing the data of the screen group in the circular buffer according to each marker in the structure queue; and determining the current write position of the data of the next screen group in the circular buffer after the buffer clearing is completed.

7. A storage device for audio and video data, characterized in that, The device includes: a receiving module for receiving audio and video data transmitted from a device and obtaining a file start identifier of a TS file; a switching module for writing the TS stream of the I-frame data to a circular buffer and switching the file start identifier from the first state to a second state when the file start identifier is in a first state and the audio and video data is I-frame data; the first state is used to indicate that only I-frame data can be received at present; the second state is used to indicate that audio frame data, I-frame data, P-frame data, and B-frame data can be received at present; a writing module for writing the TS stream of the audio and video data to a circular buffer when the file start identifier is in the second state and the usage rate of the circular buffer has not reached the upper limit, until the usage rate of the circular buffer reaches the upper limit; and a discarding module for discarding target data in the audio and video data when the usage rate of the circular buffer reaches the upper limit. The target data includes data other than I-frame data in the audio and video data; the determination module is used to determine the frame group of the circular buffer when the discarded target data is P-frame data, obtain the start position mark and length mark of the first part of the frame group data in the circular buffer, obtain the start position mark and length mark of the second part of the frame group data in the circular buffer, and obtain the timestamp mark of the first frame data and the timestamp mark of the last frame data of the frame group; put the start position mark and length mark of the first part data, the start position mark and length mark of the second part data, the timestamp mark of the first frame data and the timestamp mark of the last frame data into a structure queue; and upload the segmented data corresponding to the frame group data to the server according to each mark in the structure queue.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images