A method and device for synchronous acquisition and storage of multi-channel audio and video data

Abstract

The embodiment of the specification discloses a kind of synchronous acquisition storage methods and devices of multi-path audio and video data.The method includes after obtaining synchronization starting instruction, first by the time delay parameter of data source to the each road audio and video data continuously collected is carried out to remove delay acquisition processing, then by the delay time length of each road sensor to each road remove delay data is carried out starting frame synchronization alignment, finally audio and video encoding record and write disc storage, complete the synchronous acquisition storage of multi-path audio and video data.Through remove delay acquisition processing and source time error correction, the synchronous acquisition problem of multi-path audio and video data is solved, and by carrying out starting frame synchronization alignment processing, so that the starting time of each road video file obtained finally is synchronized and aligned, without needing to each road video frame by frame analysis and time sequence comparison in video decoding and playing process, just need to align the play progress time stamp of each road video data, reduce the complexity of software design and development, improve the synchronous playing efficiency of subsequent multi-path audio and video.

Description

Technical Field

[0001] This specification relates to the field of electronic information data processing technology, and in particular to a method and apparatus for synchronous acquisition and storage of multi-channel audio and video data. Background Technology

[0002] The synchronous recording and playback of audio and video in vehicle-mounted, shipborne, and airborne data storage systems is a crucial component of objectively inspecting system security. With the development of audio-visual and artificial intelligence technologies, systems are placing higher demands on the number of audio and video processing channels, the sound and picture quality of audio and video, and the synchronization performance between multiple audio and video streams. Therefore, a synchronous acquisition and storage method is needed to process multi-channel audio and video data.

[0003] Currently, multi-channel audio and video data synchronization is generally achieved through two methods: 1. Real-time acquisition and encoding of multi-channel audio and video data to ensure synchronization; 2. Adding frame timestamp information to each audio and video frame to achieve synchronized playback during decoding. However, with the increasing complexity of system environments, various audio and video interface types, the number of audio and video channels, and video resolutions are constantly increasing or improving. Furthermore, different types of video data may be processed by other devices before being acquired and stored, resulting in inconsistent transmission times from sensors to the acquisition and storage devices. This makes real-time acquisition and storage of audio and video data unable to solve the problem of asynchronous multi-channel audio and video data at the front end. Adding frame timestamp information to each audio and video frame to record data introduces a lag in subsequent synchronization processing because the timestamp information corresponds to the encoding completion time, rather than the data acquisition time. This increases the burden of video decoding and the complexity of software design and development, affecting the efficiency of subsequent multi-channel audio and video synchronized playback. Summary of the Invention

[0004] This specification provides a method and apparatus for synchronous acquisition and storage of multi-channel audio and video data, the technical solution of which is as follows:

[0005] In a first aspect, embodiments of this specification provide a method for synchronous acquisition and storage of multi-channel audio and video data, the method comprising:

[0006] Obtain a synchronization start command, and continuously collect at least two audio and video data based on the synchronization start command;

[0007] Based on the data source delay parameters of each audio and video data stream, delay-free acquisition processing is performed on each audio and video data stream to obtain delay-free data for each stream.

[0008] Based on the delay duration of each sensor, the start frame of each channel of the de-delayed data is synchronized and aligned to obtain each channel of valid data. The start frame synchronization alignment is used to synchronize and align the start frames in each channel of the de-delayed data.

[0009] The valid data from each channel is encoded and recorded using audio and video encoding, and the encoded data is written to a disk for storage.

[0010] Secondly, a device for synchronous acquisition and storage of multi-channel audio and video data is provided, the device comprising:

[0011] The acquisition module is used to acquire a synchronization start command and continuously collect at least two audio and video data based on the synchronization start command;

[0012] The processing module is used to perform delay-free acquisition processing on the audio and video data of each channel based on the data source delay parameters of each channel, so as to obtain the delay-free data of each channel.

[0013] The synchronization module is used to synchronize and align the start frames of the de-delayed data of each channel based on the delay time of each sensor to obtain valid data of each channel. The start frame synchronization and alignment is used to synchronize and align the start frames of the de-delayed data of each channel.

[0014] The encoding module is used to encode and record the valid data from each channel, and to write the encoded data to a disk for storage.

[0015] Thirdly, an electronic device is provided, including a device processor and a memory;

[0016] The device processor is connected to the memory;

[0017] The memory is used to store executable program code;

[0018] The device processor runs a program corresponding to the executable program code stored in the memory to perform the steps of the method provided as in the first aspect or any possible implementation thereof.

[0019] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, the computer-readable storage medium storing instructions that, when executed on a computer or device processor, cause the computer or device processor to perform the method provided as in the first aspect or any possible implementation thereof.

[0020] The beneficial effects of the technical solutions provided in some embodiments of this specification include at least the following:

[0021] In one or more embodiments of this specification, after obtaining the synchronization start command, the continuously acquired audio and video data is first processed by removing the delay through the data source delay parameter. Then, the start frame of each de-delayed data is synchronized and aligned using the delay duration of each sensor. Finally, the obtained valid data is encoded, recorded, and written to disk for storage, thus completing the synchronous acquisition and storage of multiple audio and video data. By removing the delay and correcting the source time error, the problem of synchronous acquisition of multiple audio and video data is solved. Furthermore, by performing start frame synchronization and alignment, the start times of the final video files are synchronized and aligned. In the video decoding and playback process, it is not necessary to perform frame-by-frame parsing and time sequence comparison of each video; only the playback progress timestamps of each video data need to be matched, which reduces the complexity of software design and development and improves the efficiency of subsequent synchronous playback of multiple audio and video data. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A schematic diagram of a system architecture for a method for synchronous acquisition and storage of multi-channel audio and video data provided in the embodiments of this specification;

[0024] Figure 2 A flowchart illustrating a method for synchronous acquisition and storage of multi-channel audio and video data provided in this specification's embodiments;

[0025] Figure 3 A schematic diagram of the structure of a multi-channel audio and video data synchronous acquisition and storage device provided in the embodiments of this specification;

[0026] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this specification. Detailed Implementation

[0027] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0028] The terms "first," "second," "third," etc., in the description, claims, and accompanying drawings are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0029] The following description provides examples and does not limit the scope, applicability, or examples set forth in the claims. Changes may be made to the function and arrangement of the described elements without departing from the scope of this specification. Various processes or components may be appropriately omitted, substituted, or added to the examples. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Furthermore, features described with respect to some examples may be combined into other examples.

[0030] Please see Figure 1 , Figure 1 This document illustrates a system architecture diagram of a method for synchronous acquisition and storage of multi-channel audio and video data provided in an embodiment of this specification.

[0031] like Figure 1 As shown, the system architecture of this method for synchronous acquisition and storage of multi-channel audio and video data can include at least a terminal 10, a server 20, and a network 30.

[0032] Terminal 10 includes, but is not limited to, electronic devices such as smartphones, desktop computers, tablets, laptops, smart speakers, digital assistants, and smart wearable devices, and may also be software running on the aforementioned electronic devices, such as applications. Optionally, the operating system running on the electronic device may include, but is not limited to, Android, iOS, Linux, and Windows. Optionally, terminal 10 provides synchronous data collection and storage services to the user. Terminal 10 can obtain synchronous data collection and storage instructions from the application programming interface and send synchronous data collection and storage requests to server 20.

[0033] Server 20 can provide background services for terminal 10. Based on the synchronous acquisition and storage request sent by terminal 10, server 20 will obtain a series of synchronous acquisition and storage instructions, and then transmit these instructions to other terminals 10 via network 30. Specifically, server 20 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0034] Network 30 is a medium used to provide a communication link between terminal 10 and server 20. Network 30 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0035] In addition, it should be noted that, Figure 1 The system shown is merely one example of the system provided in this disclosure. In practical applications, other systems may also be included, such as more terminals.

[0036] In the embodiments described in this specification, the terminal 10 and the server 20 can be directly or indirectly connected through wired or wireless communication, and this disclosure does not impose any restrictions.

[0037] Please refer to the following. Figure 2 , Figure 2 This document illustrates an overall flowchart of a method for synchronous acquisition and storage of multi-channel audio and video data provided in an embodiment of this specification. This method can be used in server 20.

[0038] like Figure 2 As shown, the method for synchronous acquisition and storage of multi-channel audio and video data may include at least the following steps:

[0039] Step 201: Obtain the synchronization start command, and continuously collect at least two audio and video data based on the synchronization start command.

[0040] In the embodiments of this specification, to meet the development needs of audio-visual and artificial intelligence technologies, it is often necessary to simultaneously acquire and play multiple audio and video signals through multiple sensors. Therefore, after receiving a capture start command sent by the target user via a button or mobile target terminal, the multi-channel audio and video data acquisition and storage system first obtains and parses the synchronization start command. After confirming that the command format and parameters are correct, it can send the synchronization start command to the audio and video acquisition module in the system. Then, after receiving the synchronization start command, the audio and video acquisition module opens the designated audio and video acquisition channel and continuously acquires multiple audio and video data.

[0041] In one possible implementation, before continuously acquiring at least two audio and video data based on the synchronous start command, the method further includes:

[0042] Determine the acquisition start instruction and the encoding start instruction in the synchronous start instruction;

[0043] Clear the data in the acquisition buffer of the acquisition device, and mark the acquisition timestamp for subsequent audio and video data;

[0044] Clear the encoding cache data in the encoder and restart / reset the encoder.

[0045] In the embodiments of this specification, after receiving the synchronization start command, the acquisition start command and encoding start command included in the synchronization start command are first determined. Next, to prevent historical data parameters and cached data from affecting the synchronization of this audio and video synchronous acquisition, the working status of the audio and video acquisition module is first determined. If it is determined that the audio and video acquisition module is in a working state, the data acquisition and forwarding process of the audio and video acquisition module is stopped, and the acquisition cache data in the acquisition unit's DDR is cleared. Further, according to the acquisition start command, the specified audio and video acquisition channel is restarted. Each audio and video data channel is stored in the DDR cache of a different channel, and subsequent acquired audio and video data is marked with a timestamp of the acquisition time for subsequent acquisition of the corresponding timestamp information. Simultaneously, after receiving the encoding start command, the audio and video encoding module clears the encoding cache data in the encoder and restarts the encoder's audio and video encoding function to ensure that after the system receives the synchronization start command, the first frame of data encoded by each video channel is an I-frame, that is, ensuring that the starting frame of each channel's data is a keyframe, eliminating the need for subsequent repositioning and cutting of invalid frame data.

[0046] Step 202: Based on the data source delay parameters of each audio and video data channel, perform delay-free acquisition processing on each audio and video data channel to obtain delay-free data for each channel.

[0047] In the embodiments of this specification, since the actual transmission environments of audio and video source data in different acquisition channels are different during the synchronous acquisition and storage of multi-channel audio and video data, different data source delays correspond to multiple different audio and video data sources. Therefore, in order to enable synchronous acquisition, encoding, and storage for subsequent synchronous decoding and playback, the delay parameters of different data sources corresponding to each audio and video data channel can be determined first. Then, a minimum delay duration is determined based on the delay parameters of all different data sources, and delayed acquisition processing is performed based on this delay duration. That is, all audio and video acquisition channels are started to acquire data, and after lagging behind the delay duration, data synchronization processing is performed to ensure that each acquisition channel has corresponding acquisition data, avoiding the phenomenon of missing audio and video data in a few acquisition channels due to data source delay.

[0048] Optionally, when determining a minimum latency duration using latency parameters from all different data sources, a preset latency calculation rule can be used for calculation, or a machine learning model can be used. This model can be trained on historical test latency data and then used to predict the latency parameters from all different data sources to obtain the minimum latency duration.

[0049] In one possible implementation, the delay-free acquisition processing of each audio and video data stream based on the data source delay parameters of each stream, to obtain delay-free data for each stream, includes:

[0050] Determine the minimum frame rate and maximum video source latency among all the audio and video data, and calculate the minimum frame rate and maximum video source latency based on the latency calculation rules to obtain the minimum acquisition delay duration;

[0051] Based on the minimum acquisition delay duration, delay-free acquisition processing is performed on each channel of audio and video data to obtain delay-free data for each channel.

[0052] In the embodiments of this specification, the data source latency parameters for each audio and video data stream can be determined first, including the frame rate and video source latency for each audio and video stream. Next, to ensure data synchronization can be performed even with the minimum frame rate and maximum video source latency, the frame rate and video source latency of each audio and video stream need to be compared to determine the minimum frame rate f. min and the maximum latency t of the video source max Furthermore, the determined minimum frame rate f is calculated using the following latency calculation rules. min and the maximum latency t of the video source max The minimum acquisition delay t0 is calculated as follows:

[0053] t0 = 1000 ÷ f min + t max

[0054] Finally, all audio and video acquisition channels are activated to acquire data. After a delay of the minimum acquisition delay duration, the acquired data is processed synchronously to achieve delay-free acquisition and processing of each audio and video data channel, resulting in delay-free data for each channel.

[0055] Step 203: Based on the delay duration of each sensor, perform start frame synchronization alignment on the de-delayed data of each channel to obtain the effective data of each channel.

[0056] The starting frame synchronization alignment is used to synchronize and align the starting frames in each of the de-delayed data streams.

[0057] In the embodiments described in this specification, due to differences in manufacturers, interface types, or the use of third-party audio / video processing, the time taken for each de-delayed data stream to be acquired from the underlying sensors and transmitted to the synchronization system varies, resulting in different delays for each sensor stream. As an example, when an AI recognition device is connected in series before a certain video stream is acquired, and the AI ​​recognition device has an output delay t... a Therefore, during synchronization processing, the actual time of frame data generation is the acquisition timestamp minus the output delay t. a Therefore, in order to ensure that multiple audio and video data can be decoded simultaneously, it is necessary to first synchronize and align the starting frames in each de-delayed data stream by using the delay time of each sensor, so as to ensure that the actual generation time of the starting frames of each data stream is the same when synchronous encoding is performed at the beginning.

[0058] Optionally, when performing start frame synchronization alignment, source time error correction can be performed first, and then start frame synchronization alignment can be performed using a determined buffer reference time. Alternatively, a timing arrangement method can be used to arrange the de-delayed data according to their actual generation time, and start frame synchronization alignment can be performed based on the same actual generation time.

[0059] In one possible implementation, the step of synchronizing and aligning the de-delayed data of each channel based on the delay duration of each sensor channel to obtain valid data for each channel includes:

[0060] Based on the delay time of each sensor, the source time error of each channel's de-delayed data is corrected to obtain the corrected data for each channel.

[0061] Determine the cache reference time corresponding to each path of the corrected data, and perform start frame synchronization alignment on each path of the corrected data based on the cache reference time to obtain each path of valid data.

[0062] In the embodiments of this specification, when performing start frame synchronization alignment on each channel of de-delayed data, the source time error of the de-delayed data can first be corrected by using the sensor delay duration corresponding to each channel of data. This eliminates the error between the actual generation time and the acquisition time of each frame of data, resulting in corrected data for each channel that has had the influence of sensor delay removed. Next, a buffer reference time corresponding to all the obtained corrected data is determined, and this buffer reference time is used as the start frame time line. Other data exceeding this start frame time line are filtered out, completing the start frame synchronization alignment and obtaining valid data for each channel. This eliminates the need for frame-by-frame parsing and timing comparison of each audio and video channel during subsequent data decoding and playback.

[0063] In one possible implementation, the step of correcting the source time error of each channel of the de-delayed data based on the delay duration of each sensor channel to obtain corrected data for each channel includes:

[0064] Determine the acquisition timestamp and sensor delay duration corresponding to each audio and video data stream;

[0065] The difference between the acquisition timestamp and the sensor delay duration is calculated, and the source time error of each channel of the de-delayed data is corrected based on the difference calculation result to obtain the corrected data for each channel.

[0066] In the embodiments of this specification, the acquisition timestamps and sensor delay durations for each acquisition channel when acquiring audio and video data in the acquisition module are first determined. Next, the difference between each acquisition timestamp and the corresponding sensor delay duration is calculated to obtain the actual generation time corresponding to each de-delayed data. Further, the source time of the de-delayed data is corrected by adjusting the actual generation time, thus obtaining corrected data for each channel with accurate source time.

[0067] If a portion of the data packets in the obtained de-delayed data streams contains the real-time acquisition timestamps from the sensors, then the corresponding source time is the actual generation time, and no source time error correction is needed for this portion of the de-delayed data. As an example, when the audio / video input interface is SRIO, ARINC818, PCIE, Ethernet, etc., the data contains the timestamp information of the sensor acquisition time, and no correction is needed. When the audio / video input interface is a standard interface such as SDI, PAL, HDMI, VGA, etc., the input data does not contain the timestamp information of the sensor acquisition time, and therefore source time error correction is required.

[0068] In one possible implementation, determining the cache reference time corresponding to each path of the corrected data, and performing start frame synchronization alignment on each path of the corrected data based on the cache reference time to obtain each path of valid data, includes:

[0069] Obtain the starting frame time corresponding to each path of the corrected data, and determine the maximum value among the starting frame times of each path based on the extreme value algorithm to obtain the cache reference time;

[0070] Remove invalid data from each path of the corrected data that is less than the cache reference time, and synchronize and align each path of the corrected data based on the cache reference time to obtain valid data for each path.

[0071] In the embodiments of this specification, when synchronizing and aligning the starting frames of each corrected data stream, the starting frame time corresponding to each corrected data stream can be obtained first, and arranged according to the same video acquisition timeline. Next, the maximum value among all starting frame times is determined using an extreme value algorithm, and the time corresponding to this maximum value is used as the buffer reference time. Further, using this buffer reference time as a standard, all invalid data in each corrected data stream with acquisition times less than this buffer reference time are removed, and the remaining data is synchronized and aligned with this buffer reference time as the starting frame to obtain each valid data stream.

[0072] Step 204: Encode and record the valid data from each channel using audio and video, and then write the encoded data to a disk for storage.

[0073] In the embodiments of this specification, after synchronizing and aligning the start frames of each audio and video data channel to obtain valid data for each channel, the valid data for each channel can first be synchronously encoded and recorded to obtain encoded data. Then, the encoded data is written to disk and stored to obtain video files with the same start time, thus completing the synchronous acquisition and storage of multiple audio and video data channels. This ensures that after each synchronization start command is received, the start time of the video files corresponding to each audio and video data channel is synchronously aligned. In subsequent multi-channel synchronous playback, it is only necessary to match the playback progress timestamps.

[0074] If a video channel is missing data due to signal source issues before encoding, the channel is filled with blue screen data and a timestamp of the current time is added before encoding. This ensures that once the audio and video signal source is restored, the audio and video channel can maintain the same starting frame synchronization with other audio and video channels.

[0075] In one possible implementation, the step of encoding and recording the valid data from each channel into audio and video format, and then writing and storing the encoded data to a disk, includes:

[0076] Determine the collection timestamp information in the valid data from each path;

[0077] The valid data from each channel is encoded and recorded based on the timestamp information of each acquisition.

[0078] The encoded data is written to disk and stored based on the collected timestamp information and the cache reference time.

[0079] In the embodiments of this specification, since the acquisition module clears the acquisition buffer data after receiving the acquisition start command, it will mark the subsequent acquired audio and video data with acquisition timestamps. Therefore, the acquisition timestamp information in each valid data channel can be determined first, and then the determined acquisition timestamp information is added to the bitstream information and combined with each valid data channel for encoding. The completed encoded data is then forwarded to the data recording module. Furthermore, after receiving the encoded data, the data recording module extracts the acquisition timestamp information and the buffer reference time from the encoded data as recording time information and sends the encoded data to the standard audio and video file I / O interface for disk writing and storage.

[0080] When encoding audio and video data, the video encoding format supports H.264, H.265 and other formats, and the audio encoding format supports LPCM, G711A, G711U, AAC and other formats. When writing to disk, the standard audio and video files support mp4, mkv, avi and other container formats.

[0081] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0082] Please refer to the following. Figure 3 , Figure 3 A schematic diagram of a multi-channel audio and video data synchronous acquisition and storage device provided in an embodiment of this specification is shown. It should be noted that... Figure 3 The multi-channel audio and video data synchronous acquisition and storage device shown is used to execute this application. Figure 2 The methods shown in the embodiments are for illustrative purposes only, illustrating the parts relevant to the embodiments of this application. For specific technical details not disclosed, please refer to this application. Figure 2 The example shown.

[0083] like Figure 3 As shown, the synchronous acquisition and storage device for multi-channel audio and video data may include at least:

[0084] The acquisition module 301 is used to acquire a synchronization start command and continuously collect at least two audio and video data based on the synchronization start command;

[0085] Processing module 302 is used to perform delay-free acquisition processing on each audio and video data based on the data source delay parameters of each audio and video data, to obtain each delay-free data.

[0086] Synchronization module 303 is used to perform start frame synchronization alignment on each of the de-delayed data based on the delay duration of each sensor to obtain valid data for each channel. The start frame synchronization alignment is used to synchronize the start frames of each of the de-delayed data.

[0087] The encoding module 304 is used to encode and record the valid data from each channel, and to write the encoded data to a disk for storage.

[0088] In one possible implementation, the acquisition module 301 is specifically used for:

[0089] Determine the acquisition start instruction and the encoding start instruction in the synchronous start instruction;

[0090] Clear the data in the acquisition buffer of the acquisition device, and mark the acquisition timestamp for subsequent audio and video data;

[0091] Clear the encoding cache data in the encoder and restart / reset the encoder.

[0092] In one possible implementation, the processing module 302 is specifically used for:

[0093] Determine the minimum frame rate and maximum video source latency among all the audio and video data, and calculate the minimum frame rate and maximum video source latency based on the latency calculation rules to obtain the minimum acquisition delay duration;

[0094] Based on the minimum acquisition delay duration, delay-free acquisition processing is performed on each channel of audio and video data to obtain delay-free data for each channel.

[0095] In one possible implementation, the synchronization module 303 is specifically used for:

[0096] Based on the delay time of each sensor, the source time error of each channel's de-delayed data is corrected to obtain the corrected data for each channel.

[0097] Determine the cache reference time corresponding to each path of the corrected data, and perform start frame synchronization alignment on each path of the corrected data based on the cache reference time to obtain each path of valid data.

[0098] In one possible implementation, the synchronization module 303 is further configured to:

[0099] Determine the acquisition timestamp and sensor delay duration corresponding to each audio and video data stream;

[0100] The difference between the acquisition timestamp and the sensor delay duration is calculated, and the source time error of each channel of the de-delayed data is corrected based on the difference calculation result to obtain the corrected data for each channel.

[0101] In one possible implementation, the synchronization module 303 is further configured to:

[0102] Obtain the starting frame time corresponding to each path of the corrected data, and determine the maximum value among the starting frame times of each path based on the extreme value algorithm to obtain the cache reference time;

[0103] Remove invalid data from each path of the corrected data that is less than the cache reference time, and synchronize and align each path of the corrected data based on the cache reference time to obtain valid data for each path.

[0104] In one possible implementation, the encoding module 304 is specifically used for:

[0105] Determine the collection timestamp information in the valid data from each path;

[0106] The valid data from each channel is encoded and recorded based on the timestamp information of each acquisition.

[0107] The encoded data is written to disk and stored based on the collected timestamp information and the cache reference time.

[0108] Those skilled in the art will clearly understand that the technical solutions of the embodiments of this application can be implemented by means of software and / or hardware. In this specification, "unit" and "module" refer to software and / or hardware that can independently complete or cooperate with other components to complete a specific function, wherein the hardware may be, for example, a field-programmable gate array (FPGA), an integrated circuit (IC), etc.

[0109] Each processing unit and / or module in the embodiments of this application can be implemented by an analog circuit that implements the functions described in the embodiments of this application, or by software that executes the functions described in the embodiments of this application.

[0110] Please refer to the following. Figure 4 , Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of this specification is shown.

[0111] like Figure 4 As shown, the electronic device 400 may include: at least one device processor 401, at least one network interface 404, user interface 403, memory 405, and at least one communication bus 402.

[0112] The communication bus 402 can be used to realize the connection and communication of the above components.

[0113] The user interface 403 may include buttons, and the optional user interface may also include a standard wired interface or a wireless interface.

[0114] Among them, network interface 404 may include, but is not limited to, Bluetooth module, NFC module, Wi-Fi module, etc.

[0115] The device processor 401 may include one or more processing cores. The device processor 401 connects to various parts within the electronic device 400 using various interfaces and lines. It executes various functions and processes data of the electronic device 400 by running or executing instructions, programs, code sets, or instruction sets stored in the memory 405, and by calling data stored in the memory 405. Optionally, the device processor 401 may be implemented using at least one hardware form of DSP, FPGA, or PLA. The device processor 401 may integrate one or more of the following: CPU, GPU, and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the device processor 401 and may be implemented as a separate chip.

[0116] The memory 405 may include RAM or ROM. Optionally, the memory 405 may include a non-transitory computer-readable medium. The memory 405 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 405 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 405 may also be at least one storage device located remotely from the aforementioned device processor 401. Figure 4 As shown, the memory 405, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and program instructions.

[0117] Specifically, the device processor 401 can be used to call the synchronous acquisition and storage application for multi-channel audio and video data stored in the memory 405, and specifically perform the following operations:

[0118] Obtain a synchronization start command, and continuously collect at least two audio and video data based on the synchronization start command;

[0119] Based on the data source delay parameters of each audio and video data stream, delay-free acquisition processing is performed on each audio and video data stream to obtain delay-free data for each stream.

[0120] Based on the delay duration of each sensor, the start frame of each channel of the de-delayed data is synchronized and aligned to obtain each channel of valid data. The start frame synchronization alignment is used to synchronize and align the start frames in each channel of the de-delayed data.

[0121] The valid data from each channel is encoded and recorded using audio and video encoding, and the encoded data is written to a disk for storage.

[0122] As an optional embodiment of this specification, before continuously acquiring at least two audio and video data based on the synchronous start command, the method further includes:

[0123] Determine the acquisition start instruction and the encoding start instruction in the synchronous start instruction;

[0124] Clear the data in the acquisition buffer of the acquisition device, and mark the acquisition timestamp for subsequent audio and video data;

[0125] Clear the encoding cache data in the encoder and restart / reset the encoder.

[0126] As an optional embodiment of this specification, the step of performing delay-free acquisition processing on each channel of audio and video data based on the data source delay parameters of each channel of audio and video data to obtain each channel of delay-free data includes:

[0127] Determine the minimum frame rate and maximum video source latency among all the audio and video data, and calculate the minimum frame rate and maximum video source latency based on the latency calculation rules to obtain the minimum acquisition delay duration;

[0128] Based on the minimum acquisition delay duration, delay-free acquisition processing is performed on each channel of audio and video data to obtain delay-free data for each channel.

[0129] As an optional embodiment of this specification, the step of synchronizing and aligning the de-delayed data of each channel based on the delay duration of each sensor channel to obtain valid data for each channel includes:

[0130] Based on the delay time of each sensor, the source time error of each channel's de-delayed data is corrected to obtain the corrected data for each channel.

[0131] Determine the cache reference time corresponding to each path of the corrected data, and perform start frame synchronization alignment on each path of the corrected data based on the cache reference time to obtain each path of valid data.

[0132] As an optional embodiment of this specification, the step of correcting the source time error of each channel of the de-delayed data based on the delay duration of each sensor channel to obtain corrected data for each channel includes:

[0133] Determine the acquisition timestamp and sensor delay duration corresponding to each audio and video data stream;

[0134] The difference between the acquisition timestamp and the sensor delay duration is calculated, and the source time error of each channel of the de-delayed data is corrected based on the difference calculation result to obtain the corrected data for each channel.

[0135] As an optional embodiment of this specification, the step of determining the cache reference time corresponding to each path of the corrected data, and performing start frame synchronization alignment on each path of the corrected data based on the cache reference time to obtain each path of valid data includes:

[0136] Obtain the starting frame time corresponding to each path of the corrected data, and determine the maximum value among the starting frame times of each path based on the extreme value algorithm to obtain the cache reference time;

[0137] Remove invalid data from each path of the corrected data that is less than the cache reference time, and synchronize and align each path of the corrected data based on the cache reference time to obtain valid data for each path.

[0138] As an optional embodiment of this specification, the step of recording audio and video encoding of each channel of valid data and writing the encoded data to disk includes:

[0139] Determine the collection timestamp information in the valid data from each path;

[0140] The valid data from each channel is encoded and recorded based on the timestamp information of each acquisition.

[0141] The encoded data is written to disk and stored based on the collected timestamp information and the cache reference time.

[0142] This specification also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0143] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0144] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0145] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0146] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0147] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0148] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0149] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0150] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

Claims

1. A method for synchronous acquisition and storage of multi-channel audio and video data, characterized in that, The method includes: Obtain a synchronization start command, and continuously collect at least two audio and video data based on the synchronization start command; Based on the data source delay parameters of each audio and video data stream, delay-free acquisition processing is performed on each audio and video data stream to obtain delay-free data for each stream. Based on the delay duration of each sensor, the start frame of each channel of the de-delayed data is synchronized and aligned to obtain each channel of valid data. The start frame synchronization alignment is used to synchronize and align the start frames in each channel of the de-delayed data. The valid data from each channel is encoded and recorded using audio and video encoding, and the encoded data is written to disk for storage. The process of performing delay-free acquisition on the audio and video data of each channel based on the data source delay parameters of each channel to obtain delay-free data for each channel includes: Determine the minimum frame rate and maximum video source latency among all the audio and video data, and calculate the minimum frame rate and maximum video source latency based on the latency calculation rules to obtain the minimum acquisition delay duration; Based on the minimum acquisition delay duration, delay-free acquisition processing is performed on each channel of audio and video data to obtain delay-free data for each channel. The method of synchronizing and aligning the de-delayed data of each channel based on the delay duration of each sensor channel to obtain the effective data of each channel includes: Based on the delay time of each sensor, the source time error of each channel's de-delayed data is corrected to obtain the corrected data for each channel. Determine the cache reference time corresponding to each path of the corrected data, and perform start frame synchronization alignment on each path of the corrected data based on the cache reference time to obtain each path of valid data; The source time error correction is performed on each channel of the de-delayed data based on the delay duration of each sensor channel to obtain the corrected data for each channel, including: Determine the acquisition timestamp and sensor delay duration corresponding to each audio and video data stream; The difference between the acquisition timestamp and the sensor delay duration is calculated, and the source time error of each channel of the de-delayed data is corrected based on the difference calculation result to obtain the corrected data for each channel. The process involves determining the cache reference time corresponding to each path of the corrected data, and performing start frame synchronization alignment on each path of the corrected data based on the cache reference time to obtain valid data for each path, including: Obtain the starting frame time corresponding to each path of the corrected data, and determine the maximum value among the starting frame times of each path based on the extreme value algorithm to obtain the cache reference time; Remove invalid data from each path of the corrected data that is less than the cache reference time, and synchronize and align each path of the corrected data based on the cache reference time to obtain valid data for each path.

2. The method according to claim 1, characterized in that, Before continuously acquiring at least two audio and video data streams based on the synchronous start command, the process also includes: Determine the acquisition start instruction and the encoding start instruction in the synchronous start instruction; Clear the data in the acquisition buffer of the acquisition device, and mark the acquisition timestamp for subsequent audio and video data; Clear the encoding cache data in the encoder and restart / reset the encoder.

3. The method according to claim 1, characterized in that, The process of encoding and recording the valid data from each channel into audio and video formats, and then writing and storing the encoded data to disk, includes: Determine the collection timestamp information in the valid data from each path; The valid data from each channel is encoded and recorded based on the timestamp information of each acquisition. The encoded data is written to disk and stored based on the collected timestamp information and the cache reference time.

4. A device for synchronous acquisition and storage of multi-channel audio and video data, characterized in that, The device includes: The acquisition module is used to acquire a synchronization start command and continuously collect at least two audio and video data based on the synchronization start command; The processing module is used to perform delay-free acquisition processing on the audio and video data of each channel based on the data source delay parameters of each channel, so as to obtain the delay-free data of each channel. The synchronization module is used to synchronize and align the start frames of the de-delayed data of each channel based on the delay time of each sensor to obtain valid data of each channel. The start frame synchronization and alignment is used to synchronize and align the start frames of the de-delayed data of each channel. The encoding module is used to encode and record the valid data from each channel, and to write and store the encoded data to a disk. The processing module is specifically used for: Determine the minimum frame rate and maximum video source latency among all the audio and video data, and calculate the minimum frame rate and maximum video source latency based on the latency calculation rules to obtain the minimum acquisition delay duration; Based on the minimum acquisition delay duration, delay-free acquisition processing is performed on each channel of audio and video data to obtain delay-free data for each channel. The synchronization module is specifically used for: Based on the delay time of each sensor, the source time error of each channel's de-delayed data is corrected to obtain the corrected data for each channel. Determine the cache reference time corresponding to each path of the corrected data, and perform start frame synchronization alignment on each path of the corrected data based on the cache reference time to obtain each path of valid data; The synchronization module is also specifically used for: Determine the acquisition timestamp and sensor delay duration corresponding to each audio and video data stream; The difference between the acquisition timestamp and the sensor delay duration is calculated, and the source time error of each channel of the de-delayed data is corrected based on the difference calculation result to obtain the corrected data for each channel. The synchronization module is also specifically used for: Obtain the starting frame time corresponding to each path of the corrected data, and determine the maximum value among the starting frame times of each path based on the extreme value algorithm to obtain the cache reference time; Remove invalid data from each path of the corrected data that is less than the cache reference time, and synchronize and align each path of the corrected data based on the cache reference time to obtain valid data for each path.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1-3.

6. A computer-readable storage medium having a computer program stored thereon, the computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform the steps of the method as claimed in any one of claims 1-3.

Images