Synchronous acquisition and storage method and device for multi-channel audio and video data

Abstract

The embodiment of the invention discloses a synchronous acquisition and storage method and device for multi-channel audio and video data. The method comprises the steps that after a synchronous starting instruction is obtained, dedelay collection processing is carried out on all paths of continuously collected audio and video data through data source delay parameters, starting frame synchronous alignment is carried out on all paths of dedelay data through delay duration of all paths of sensors, and finally audio and video coding recording and disk writing storage are carried out. And synchronous acquisition and storage of multi-channel audio and video data are completed. The problem of synchronous acquisition of multi-channel audio and video data is solved through de-delay acquisition processing and source time error correction, initial time of each channel of finally obtained video file is synchronously aligned through initial frame synchronous alignment processing, frame-by-frame analysis and time sequence comparison do not need to be carried out on each channel of video in a video decoding and playing process, and the video decoding and playing efficiency is improved. And only the playing progress timestamps of all paths of video data need to be aligned, so that the software design and development complexity is reduced, and the synchronous playing efficiency of subsequent multiple paths of audios and videos is improved.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the technical field of electronic information data processing, and in particular, to a method and device for synchronously collecting and storing multi-channel audio and video data. Background Art

[0002] The audio and video synchronous recording and playback function of vehicle-mounted, ship-mounted, and airborne data storage systems is an important part of the objective inspection system safety. With the development of audio and video and artificial intelligence technology, the system has put forward higher requirements on the number of audio and video processing channels, the sound and picture quality of audio and video, and the synchronization performance between multi-channel audio and multi-channel video. Therefore, a synchronous acquisition and storage method is needed to process multi-channel audio and video data.

[0003] At present, the following two methods are generally used to synchronize the processing of multi-channel audio and video data: 1. Real-time acquisition and real-time encoding of multi-channel audio and video data to ensure the synchronization of multi-channel audio and video; 2. Adding frame timestamp information to each audio and video frame to achieve synchronous playback of decoding. However, with the increase in the complexity of the system environment, various types of audio and video interfaces, audio and video channels, video resolution, etc. are constantly increasing or improving, and various types of video data may be processed by other devices before being acquired and stored. The time from the sensor to the acquisition and storage device is inconsistent, resulting in the real-time acquisition and storage of audio and video data cannot solve the problem of asynchronous multi-channel audio and video data at the front end. The method of adding frame timestamp information to each audio and video frame to record data has a certain lag in the subsequent synchronization processing because its timestamp information is the corresponding encoding completion time, rather than the timestamp information corresponding to the data acquisition time, which increases the burden of video decoding and the complexity of software design and development, and affects the efficiency of subsequent multi-channel audio and video synchronous playback. Summary of the invention

[0004] The embodiment of this specification provides a method and device for synchronously collecting and storing multi-channel audio and video data, and its technical solution is as follows: In a first aspect, an embodiment of the present specification provides a method for synchronously collecting and storing multi-channel audio and video data, the method comprising: Obtaining a synchronization start instruction, and continuously collecting at least two channels of audio and video data based on the synchronization start instruction; Performing delay-removing collection processing on the audio and video data of each channel based on the data source delay parameter of the audio and video data of each channel to obtain the delay-removing data of each channel; Based on the delay time of each sensor, the start frame of each channel of the de-delayed data is synchronously aligned to obtain each channel of valid data, and the start frame synchronization alignment is used to synchronously align the start frames in each channel of the de-delayed data; The valid data of each channel is recorded in audio and video encoding, and the encoded data is written to a disk for storage.

[0005] In a second aspect, a device for synchronously collecting and storing multi-channel audio and video data is provided, the device comprising: An acquisition module, used to acquire a synchronization start instruction, and continuously collect at least two channels of audio and video data based on the synchronization start instruction; A processing module, used for performing delay-removing collection processing on each channel of the audio and video data based on a data source delay parameter of each channel of the audio and video data to obtain each channel of delay-removing data; A synchronization module, used for performing start frame synchronization alignment on each channel of the de-delayed data based on the delay time of each channel of sensors to obtain each channel of valid data, wherein the start frame synchronization alignment is used for performing synchronization alignment on the start frames of each channel of the de-delayed data; The encoding module is used to perform audio and video encoding and recording of each channel of valid data, and write the encoded data to a disk for storage.

[0006] In a third aspect, an electronic device is provided, including a device processor and a memory; The device processor is connected to the memory; The memory is used to store executable program code; The device processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the steps of the method provided in the first aspect or any possible implementation manner of the first aspect.

[0007] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and the computer-readable storage medium stores instructions, which, when the instructions are executed on a computer or a device processor, cause the computer or the device processor to execute the method provided in the first aspect or any possible implementation of the first aspect.

[0008] The beneficial effects brought by the technical solutions provided by some embodiments of this specification include at least: In one or more embodiments of the present specification, after obtaining the synchronous start instruction, the audio and video data collected continuously are first processed for delay removal through the data source delay parameter, and then the start frame synchronization alignment of each channel of delay removal data is performed through the delay time of each sensor, and finally the obtained valid data is recorded by audio and video encoding and written to disk for storage, so as to complete the synchronous collection and storage of multiple channels of audio and video data. The problem of synchronous collection of multiple channels of audio and video data is solved by delay removal and source time error correction, and the start frame synchronization alignment processing is performed so that the start time of each channel of video files finally obtained is synchronized and aligned. In the video decoding and playback process, there is no need to perform frame-by-frame analysis and timing comparison on each channel of video, and only the playback progress timestamps of each channel of video data need to be matched, which reduces the complexity of software design and development and improves the efficiency of subsequent synchronous playback of multiple channels of audio and video. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0010] Figure 1 A schematic diagram of the system architecture of a method for synchronously collecting and storing multi-channel audio and video data provided in an embodiment of this specification; Figure 2 A flowchart of a method for synchronously collecting and storing multi-channel audio and video data provided by an embodiment of this specification; Figure 3 A schematic diagram of the structure of a device for synchronously collecting and storing multi-channel audio and video data provided in an embodiment of this specification; Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0011] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0012] The terms "first", "second", "third", etc. in the description and claims of this specification and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0013] The following description provides examples and does not limit the scope, applicability or examples set forth in the claims. Changes may be made to the functions and arrangements of the elements described without departing from the scope of the present specification. Various processes or components may be appropriately omitted, substituted or added to each example. For example, the described method may be performed in an order different from the order described, and various steps may be added, omitted or combined. In addition, features described with respect to some examples may be combined in other examples.

[0014] See also Figure 1 , Figure 1 A schematic diagram of the system architecture of a method for synchronously collecting and storing multi-channel audio and video data provided by an embodiment of this specification is shown.

[0015] like Figure 1 As shown, the system architecture of the method for synchronously collecting and storing multi-channel audio and video data may at least include a terminal 10 , a server 20 and a network 30 .

[0016] The terminal 10 includes but is not limited to electronic devices such as smart phones, desktop computers, tablet computers, laptops, smart speakers, digital assistants, smart wearable devices, etc., and may also be software running on the above electronic devices, such as applications, etc. Optionally, the operating system running on the electronic device may include but is not limited to Android system, IOS system, Linux, Windows, etc. Optionally, the terminal 10 provides a synchronous acquisition storage service to the user, and the terminal 10 may obtain a synchronous acquisition storage instruction of an application program interface and send a synchronous acquisition storage request to the server 20.

[0017] The server 20 can provide background services for the terminal 10. According to the synchronous acquisition and storage request sent by the terminal 10, the server 20 will obtain a series of synchronous acquisition and storage instructions, and the server 20 will transmit the synchronous acquisition and storage instructions to other terminals 10 through the network 30. Specifically, the server 20 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0018] The network 30 is used to provide a medium for a communication link between the terminal 10 and the server 20. The network 30 may include various connection types, such as wired or wireless communication links or optical fiber cables.

[0019] In addition, it should be noted that Figure 1 What is shown is only one system provided by the present disclosure. In practical applications, other systems may also be included, for example, more terminals may be included.

[0020] In the embodiments of the present specification, the terminal 10 and the server 20 may be directly or indirectly connected via wired or wireless communication, which is not limited in the present disclosure.

[0021] See next Figure 2 , Figure 2 The overall flow chart of a method for synchronously collecting and storing multi-channel audio and video data provided by an embodiment of the present specification is shown. The method for synchronously collecting and storing multi-channel audio and video data can be used in the server 20.

[0022] like Figure 2 As shown, the method for synchronously collecting and storing multi-channel audio and video data may at least include the following steps: Step 201: Obtain a synchronization start instruction, and continuously collect at least two channels of audio and video data based on the synchronization start instruction.

[0023] In the embodiments of this specification, in order to meet the development needs of audio, video and artificial intelligence technology, it is often necessary to collect and play multiple audio and video through multiple sensors at the same time. Therefore, after receiving the collection start instruction sent by the target user through a button or a mobile target terminal, the multi-channel audio and video data collection and storage system first obtains the synchronization start instruction, parses the synchronization start instruction, and confirms that the instruction format and parameters are correct. Then, the synchronization start instruction can be sent to the audio and video collection module in the system. Then, after receiving the synchronization start instruction, the audio and video collection module opens the specified audio and video collection channel and continuously collects multiple audio and video data.

[0024] In one possible implementation, before continuously collecting at least two channels of audio and video data based on the synchronization start instruction, the method further includes: Determine the acquisition start instruction and the encoding start instruction in the synchronization start instruction; Clear the acquisition cache data in the collector and add acquisition timestamps to the subsequent audio and video data; Clear the encoding cache data in the encoder and restart and reset the encoder.

[0025] In the embodiment of the present specification, after obtaining the synchronous start instruction, the acquisition start instruction and the encoding start instruction included in the synchronous start instruction are first determined. Then, in order to prevent the historical record parameters and cache data from affecting the synchronization of this audio and video synchronous acquisition, the working state of the audio and video acquisition module is first determined. If it is determined that the audio and video acquisition module is in a working state, the data acquisition and forwarding process of the audio and video acquisition module is first stopped, and the acquisition cache data in the collector DDR is cleared. Further, according to the acquisition start instruction, the specified audio and video acquisition channel is reopened, and each channel of audio and video data is stored in the DDR cache of different channels, and the subsequent acquisition audio and video data is marked with the acquisition time timestamp so as to obtain the corresponding acquisition timestamp information later. At the same time, after the audio and video encoding module receives the encoding start instruction, the encoding cache data in the encoder is cleared, and the audio and video encoding function of the encoder is restarted for the audio and video encoding module to ensure that after the system receives the synchronous start instruction, the first frame data encoded by each video channel is an I frame, that is, the starting frame of each channel data is guaranteed to be a key frame, and there is no need to reposition and cut invalid frame data later.

[0026] Step 202: De-delay the audio and video data based on the data source delay parameters of the audio and video data to obtain the de-delayed data.

[0027] In the embodiments of the present specification, since the actual transmission environment of the audio and video source data in different acquisition channels is different during the actual synchronous acquisition and storage process of multiple channels of audio and video data, different data source delays are caused to correspond to multiple different audio and video data sources. Therefore, in order to be able to synchronously acquire, encode and store them in the future, so as to perform synchronous decoding and playback in the future, the delay parameters of different data sources corresponding to each channel of audio and video data can be determined first. Then, a minimum delay duration is determined through the delay parameters of all different data sources, and delayed acquisition processing is performed based on the delay duration, that is, all audio and video acquisition channels are opened to acquire data, and after the delay duration is delayed, data synchronization processing is performed to ensure that each acquisition channel has corresponding acquisition data, and avoid the phenomenon of missing audio and video data in a few acquisition channels due to data source delays.

[0028] Optionally, when determining a minimum delay duration through the delay parameters of all different data sources, a preset delay calculation rule can be adopted for calculation, or a machine learning model can be used to first train it through trained historical test delay data, and then the trained machine learning model is used to predict the delay parameters of all different data sources to obtain the minimum delay duration.

[0029] In one possible implementation, performing delay removal collection processing on each channel of the audio and video data based on the data source delay parameter of each channel of the audio and video data to obtain each channel of delay removal data includes: Determine the minimum frame rate and the maximum delay of the video source in all the audio and video data, calculate the minimum frame rate and the maximum delay of the video source based on the delay calculation rule, and obtain the minimum acquisition delay duration; De-delaying and collecting processing is performed on each channel of the audio and video data based on the minimum collection delay time to obtain each channel of de-delayed data.

[0030] In the embodiments of this specification, the data source delay parameters of each channel of audio and video data can be determined first, that is, the frame rate and video source delay of each channel of audio and video data can be included. Then, in order to ensure that data synchronization can be performed under the condition of minimum frame rate and maximum video source delay, it is necessary to compare the frame rate and video source delay of each channel of audio and video data to determine the minimum frame rate f min and the maximum delay of the video source t max Furthermore, the minimum frame rate f is determined by the following delay calculation rule: min and the maximum delay of the video source t max Calculate the minimum acquisition delay time t0: t0 = 1000 ÷ f min + t max Finally, all audio and video acquisition channels are opened to collect data, and synchronous processing of the collected data is performed after the minimum acquisition delay time is delayed, so as to realize the de-delayed acquisition processing of each channel of audio and video data and obtain the de-delayed data of each channel.

[0031] Step 203: Perform start frame synchronization alignment on each channel of the de-delayed data based on the delay time of each channel of sensors to obtain each channel of valid data.

[0032] The start frame synchronization alignment is used to synchronize the start frames in each channel of the de-delayed data.

[0033] In the embodiments of this specification, due to different manufacturers, different interface types, or each channel of de-delayed data processed by third-party audio and video, the time from the underlying sensor to the synchronization system is different, and the corresponding sensor delays are different. As an example, when a video is connected in series with an AI recognition device before acquisition, and the AI ​​recognition device has an output delay t a , then when performing synchronization processing, the actual generation time of the frame data is the acquisition timestamp minus the output delay t a Therefore, in order to ensure that multiple audio and video data can be decoded simultaneously in the future, it is necessary to first synchronize the start frames of each channel of de-delayed data using the delay time of each sensor to ensure that when synchronous encoding is performed at the start time, the actual generation time corresponding to the start frames of each channel of data is the same.

[0034] Optionally, when performing start frame synchronization alignment, the source time error correction can be performed first, and then the start frame synchronization alignment can be performed based on the determined cache reference time. Alternatively, the time sequence arrangement method can be used to directly arrange the delay-free data of each channel according to the actual generation time, and the start frame synchronization alignment can be performed based on the same actual generation time.

[0035] In one possible implementation, the step of performing start frame synchronization alignment on each channel of the de-delayed data based on the delay time of each channel of sensors to obtain each channel of valid data includes: Performing source time error correction on each channel of the de-delayed data based on the delay time of each channel of sensors to obtain each channel of corrected data; The cache reference time corresponding to each channel of the correction data is determined, and the start frame synchronization alignment of each channel of the correction data is performed based on the cache reference time to obtain each channel of valid data.

[0036] In the embodiments of this specification, when performing start frame synchronization alignment on each channel of de-delayed data, the source time error correction of the de-delayed data can be performed first by using the sensor delay time corresponding to each channel of data, eliminating the error between the actual generation time and the acquisition time of each frame of data, and obtaining each channel of corrected data without the influence of each sensor delay. Then, a cache reference time corresponding to all the corrected data obtained is determined, and the cache reference time is used as the start frame time line, and other data exceeding the start frame time line is filtered out to complete the start frame synchronization alignment and obtain each channel of valid data, so that there is no need to perform frame-by-frame analysis and timing comparison on each channel of audio and video in the subsequent data decoding and playback process.

[0037] In one possible implementation manner, performing source time error correction on each channel of the de-delayed data based on the delay time of each channel of sensors to obtain each channel of corrected data includes: Determine the acquisition timestamp and sensor delay duration corresponding to each channel of audio and video data; The acquisition timestamp and the sensor delay duration are subjected to a difference calculation, and source time error correction is performed on each channel of the de-delayed data based on the difference calculation result to obtain each channel of corrected data.

[0038] In the embodiments of this specification, the acquisition timestamp corresponding to the mark when the acquisition module acquires each channel of audio and video data and the sensor delay duration of each acquisition channel can be determined first. Then, the difference between each acquisition timestamp and the corresponding sensor delay duration is calculated to obtain the actual generation time corresponding to each de-delayed data. Furthermore, the source time of the de-delayed data is corrected by each actual generation time, so that the corrected data of each channel with correct source time can be obtained.

[0039] Among them, if part of the data in each channel of de-delayed data contains the real-time acquisition timestamp of the sensor, the corresponding source time is the actual generation time, and there is no need to correct the source time error of this part of the de-delayed data. As an example, when the audio and video input interface is SRIO, ARINC818, PCIE, Ethernet and other interfaces, the data contains the timestamp information of the sensor acquisition time, and no correction is required. When the audio and video input interface is a standard interface such as SDI, PAL, HDMI, VGA, etc., the input data does not contain the timestamp information of the sensor acquisition time, and it is necessary to correct the source time error.

[0040] In one possible implementation manner, determining the cache reference time corresponding to each channel of the corrected data, and performing start frame synchronization alignment on each channel of the corrected data based on the cache reference time to obtain each channel of valid data includes: Obtaining the start frame time corresponding to each channel of the correction data, and determining the maximum value of the start frame time of each channel based on an extreme value algorithm to obtain a cache reference time; Invalid data in each channel of the correction data that is younger than the cache reference time is cleared, and the correction data in each channel is synchronously aligned based on the cache reference time to obtain valid data in each channel.

[0041] In the embodiments of the present specification, when performing synchronous alignment of the start frames of the correction data of each channel, the start frame time corresponding to each correction data channel can be obtained first, and arranged according to the same video acquisition time axis. Then, the maximum value of all the start frame times is determined by the extreme value algorithm, and the time corresponding to the maximum value is used as the cache reference time. Furthermore, with the cache reference time as the standard line, all invalid data in each correction data whose acquisition time is less than the cache reference time is removed, and the remaining data is synchronously aligned with the cache reference time as the start frame to obtain valid data of each channel.

[0042] Step 204: record the valid data in audio and video encoding, and write the encoded data to a disk for storage.

[0043] In the embodiment of the present specification, after the start frame synchronization alignment of each channel of audio and video data is performed to obtain each channel of valid data, the audio and video synchronization encoding and recording of each channel of valid data can be performed first to obtain the encoded data, and then the encoded data is written to the disk for storage to obtain the video files with the same start time, thereby completing the synchronous acquisition and storage of multiple channels of audio and video data. It is ensured that the start time of the video file corresponding to each channel of audio and video data is synchronously aligned after receiving the synchronous start instruction each time, and when performing multiple-channel synchronous playback in the subsequent time, it is only necessary to correspond the playback progress timestamp.

[0044] Before encoding, if there is no data in one of the video channels due to reasons such as the signal source, the blue screen data will be supplemented for the channel, and the timestamp information of the current time will be added before encoding to ensure that after the audio and video signal source of the channel is restored, the audio and video of the channel can still maintain the starting frame synchronization relationship with other audio and video channels.

[0045] In one possible implementation, the step of performing audio and video encoding and recording on each channel of valid data, and writing and storing the encoded data on a disk, includes: Determine the acquisition timestamp information in each channel of the valid data; Encoding and recording each channel of the valid data based on each acquisition timestamp information; The encoded data is written to a disk for storage based on each of the acquisition timestamp information and the cache reference time.

[0046] In the embodiment of the present specification, since the acquisition module will mark the subsequent acquired audio and video data with an acquisition timestamp after receiving the acquisition start instruction to clear the acquisition cache data, the acquisition timestamp information in each channel of valid data can be determined first, and then the determined acquisition timestamp information is added to the bitstream information to be encoded in combination with each channel of valid data, and the completed encoded data is forwarded to the data recording module. Further, after receiving the encoded data, the data recording module extracts the acquisition timestamp information and the cache reference time in the encoded data as the recording time information and sends the encoded data to the standard audio and video file IO interface for disk writing and storage.

[0047] Among them, when encoding audio and video data, the video encoding format supports H.264, H.265 and other formats, the audio encoding format supports LPCM, G711A, G711U, AAC and other formats, and the standard audio and video files support mp4, mkv, avi and other packaging formats when writing to disk.

[0048] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0049] See next Figure 3 , Figure 3 The structure diagram of a synchronous acquisition and storage device for multi-channel audio and video data provided by an embodiment of this specification is shown. It should be noted that: Figure 3 The synchronous acquisition and storage device of multi-channel audio and video data shown is used to execute the present application Figure 2 For the convenience of explanation, only the part related to the embodiment of the present application is shown. For the specific technical details not disclosed, please refer to the present application. Figure 2 The embodiment shown.

[0050] like Figure 3 As shown, the synchronous acquisition and storage device of multi-channel audio and video data may at least include: An acquisition module 301 is used to acquire a synchronization start instruction and continuously collect at least two channels of audio and video data based on the synchronization start instruction; The processing module 302 is used to perform delay-removal collection processing on each channel of the audio and video data based on the data source delay parameter of each channel of the audio and video data to obtain each channel of delay-removal data; A synchronization module 303 is used to perform start frame synchronization alignment on each channel of the de-delayed data based on the delay time of each channel of sensors to obtain each channel of valid data, wherein the start frame synchronization alignment is used to synchronize the start frames of each channel of the de-delayed data; The encoding module 304 is used to perform audio and video encoding and recording on each channel of the valid data, and write the encoded data to a disk for storage.

[0051] In one possible implementation, the acquisition module 301 is specifically used for: Determine the acquisition start instruction and the encoding start instruction in the synchronization start instruction; Clear the acquisition cache data in the collector and add acquisition timestamps to the subsequent audio and video data; Clear the encoding cache data in the encoder and restart and reset the encoder.

[0052] In one embodiment, the processing module 302 is specifically configured to: Determine the minimum frame rate and the maximum delay of the video source in all the audio and video data, calculate the minimum frame rate and the maximum delay of the video source based on the delay calculation rule, and obtain the minimum acquisition delay duration; De-delaying and collecting processing is performed on each channel of the audio and video data based on the minimum collection delay time to obtain each channel of de-delayed data.

[0053] In one possible implementation, the synchronization module 303 is specifically used for: Performing source time error correction on each channel of the de-delayed data based on the delay time of each channel of sensors to obtain each channel of corrected data; The cache reference time corresponding to each channel of the correction data is determined, and the start frame synchronization alignment of each channel of the correction data is performed based on the cache reference time to obtain each channel of valid data.

[0054] In one possible implementation, the synchronization module 303 is further configured to: Determine the acquisition timestamp and sensor delay duration corresponding to each channel of audio and video data; The acquisition timestamp and the sensor delay duration are subjected to a difference calculation, and source time error correction is performed on each channel of the de-delayed data based on the difference calculation result to obtain each channel of corrected data.

[0055] In one possible implementation, the synchronization module 303 is further configured to: Obtaining the start frame time corresponding to each channel of the correction data, and determining the maximum value of the start frame time of each channel based on an extreme value algorithm to obtain a cache reference time; Invalid data in each channel of the correction data that is younger than the cache reference time is cleared, and the correction data in each channel is synchronously aligned based on the cache reference time to obtain valid data in each channel.

[0056] In one possible implementation, the encoding module 304 is specifically used for: Determine the acquisition timestamp information in each channel of the valid data; Encoding and recording each channel of the valid data based on each acquisition timestamp information; The encoded data is written to a disk for storage based on each of the acquisition timestamp information and the cache reference time.

[0057] Those skilled in the art can clearly understand that the technical solutions of the embodiments of the present application can be implemented with the help of software and / or hardware. The "unit" and "module" in this specification refer to software and / or hardware that can independently complete or cooperate with other components to complete specific functions, where the hardware can be, for example, a field programmable gate array (FPGA), an integrated circuit (IC), etc.

[0058] Each processing unit and / or module of the embodiments of the present application may be implemented by an analog circuit that implements the functions described in the embodiments of the present application, or may be implemented by software that executes the functions described in the embodiments of the present application.

[0059] See next Figure 4 , Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of this specification is shown.

[0060] like Figure 4 As shown, the electronic device 400 may include: at least one device processor 401 , at least one network interface 404 , a user interface 403 , a memory 405 , and at least one communication bus 402 .

[0061] The communication bus 402 may be used to realize the connection and communication among the above-mentioned components.

[0062] The user interface 403 may include buttons, and the optional user interface may also include a standard wired interface or a wireless interface.

[0063] The network interface 404 may include, but is not limited to, a Bluetooth module, an NFC module, a Wi-Fi module, etc.

[0064] Among them, the device processor 401 may include one or more processing cores. The device processor 401 uses various interfaces and lines to connect various parts within the entire electronic device 400, and executes various functions and processes data of the electronic device 400 by running or executing instructions, programs, code sets or instruction sets stored in the memory 405, and calling data stored in the memory 405. Optionally, the device processor 401 can be implemented in at least one hardware form of DSP, FPGA, and PLA. The device processor 401 can integrate one or a combination of CPU, GPU, modem, etc. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to handle wireless communications. It can be understood that the above-mentioned modem may not be integrated into the device processor 401, and it can be implemented separately through a chip.

[0065] The memory 405 may include a RAM or a ROM. Optionally, the memory 405 includes a non-transitory computer-readable medium. The memory 405 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 405 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned method embodiments, etc.; the data storage area may store data involved in the above-mentioned method embodiments, etc. The memory 405 may optionally be at least one storage device located away from the aforementioned device processor 401. As Figure 4 As shown, the memory 405 as a computer storage medium may include an operating system, a network communication module, a user interface module, and program instructions.

[0066] Specifically, the device processor 401 may be used to call the synchronous acquisition and storage application for multiple channels of audio and video data stored in the memory 405, and specifically perform the following operations: Obtaining a synchronization start instruction, and continuously collecting at least two channels of audio and video data based on the synchronization start instruction; Performing delay-removing collection processing on the audio and video data of each channel based on the data source delay parameter of the audio and video data of each channel to obtain the delay-removing data of each channel; Based on the delay time of each sensor, the start frame of each channel of the de-delayed data is synchronously aligned to obtain each channel of valid data, and the start frame synchronization alignment is used to synchronously align the start frames in each channel of the de-delayed data; The valid data of each channel is recorded in audio and video encoding, and the encoded data is written to a disk for storage.

[0067] As an optional embodiment of this specification, before continuously collecting at least two channels of audio and video data based on the synchronous start instruction, the method further includes: Determine the acquisition start instruction and the encoding start instruction in the synchronization start instruction; Clear the acquisition cache data in the collector and add acquisition timestamps to the subsequent audio and video data; Clear the encoding cache data in the encoder and restart and reset the encoder.

[0068] As an optional embodiment of this specification, the de-delaying and collecting processing of each channel of the audio and video data based on the data source delay parameter of each channel of the audio and video data to obtain each channel of de-delayed data includes: Determine the minimum frame rate and the maximum delay of the video source in all the audio and video data, calculate the minimum frame rate and the maximum delay of the video source based on the delay calculation rule, and obtain the minimum acquisition delay duration; De-delaying and collecting processing is performed on each channel of the audio and video data based on the minimum collection delay time to obtain each channel of de-delayed data.

[0069] As an optional embodiment of the present specification, the starting frame synchronization alignment of each channel of the de-delayed data is performed based on the delay time of each sensor to obtain each channel of valid data, including: Performing source time error correction on each channel of the de-delayed data based on the delay time of each channel of sensors to obtain each channel of corrected data; The cache reference time corresponding to each channel of the correction data is determined, and the start frame synchronization alignment of each channel of the correction data is performed based on the cache reference time to obtain each channel of valid data.

[0070] As an optional embodiment of the present specification, the source time error correction of each channel of the de-delayed data is performed based on the delay time of each channel of the sensor to obtain each channel of corrected data, including: Determine the acquisition timestamp and sensor delay duration corresponding to each channel of audio and video data; The acquisition timestamp and the sensor delay duration are subjected to a difference calculation, and source time error correction is performed on each channel of the de-delayed data based on the difference calculation result to obtain each channel of corrected data.

[0071] As an optional embodiment of this specification, the determining of the cache reference time corresponding to each channel of the correction data, and performing start frame synchronization alignment on each channel of the correction data based on the cache reference time to obtain each channel of valid data includes: Obtaining the start frame time corresponding to each channel of the correction data, and determining the maximum value of the start frame time of each channel based on an extreme value algorithm to obtain a cache reference time; Invalid data in each channel of the correction data that is younger than the cache reference time is cleared, and the correction data in each channel is synchronously aligned based on the cache reference time to obtain valid data in each channel.

[0072] As an option of the embodiment of this specification, the step of performing audio and video encoding and recording of each channel of valid data, and writing and storing the encoded data on a disk includes: Determine the acquisition timestamp information in each channel of the valid data; Encoding and recording each channel of the valid data based on each acquisition timestamp information; The encoded data is written to a disk for storage based on each of the acquisition timestamp information and the cache reference time.

[0073] The embodiments of this specification also provide a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the above method are implemented. The computer-readable storage medium may include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a micro drive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nanosystem (including a molecular memory IC), or any type of medium or device suitable for storing instructions and / or data.

[0074] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0075] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0076] In the several embodiments provided in the present application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0077] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0078] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0079] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, disk or optical disk and other media that can store program codes.

[0080] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by entering a program to instruct related hardware, and the program may be stored in a computer-readable memory, which may include a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0081] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A method for synchronously collecting and storing multi-channel audio and video data, characterized in that: The method comprises: Obtaining a synchronization start instruction, and continuously collecting at least two channels of audio and video data based on the synchronization start instruction; Performing delay-removing collection processing on the audio and video data of each channel based on the data source delay parameter of the audio and video data of each channel to obtain the delay-removing data of each channel; Based on the delay time of each sensor, the start frame of each channel of the de-delayed data is synchronously aligned to obtain each channel of valid data, and the start frame synchronization alignment is used to synchronously align the start frames in each channel of the de-delayed data; The valid data of each channel is recorded in audio and video encoding, and the encoded data is written to a disk for storage.

2. The method according to claim 1, characterized in that Before continuously collecting at least two channels of audio and video data based on the synchronous start instruction, the method further includes: Determine the acquisition start instruction and the encoding start instruction in the synchronization start instruction; Clear the acquisition cache data in the collector and mark the acquisition timestamp for the subsequent audio and video data; Clear the encoding cache data in the encoder and restart and reset the encoder.

3. The method according to claim 1, characterized in that The step of performing delay removal collection processing on each channel of the audio and video data based on the data source delay parameter of each channel of the audio and video data to obtain each channel of delay removal data includes: Determine the minimum frame rate and the maximum delay of the video source in all the audio and video data, calculate the minimum frame rate and the maximum delay of the video source based on the delay calculation rule, and obtain the minimum acquisition delay duration; De-delaying and collecting processing is performed on each channel of the audio and video data based on the minimum collection delay time to obtain each channel of de-delayed data.

4. The method according to claim 1, characterized in that: The step of performing start frame synchronization alignment on each channel of the de-delayed data based on the delay time of each channel of the sensor to obtain each channel of valid data includes: Performing source time error correction on each channel of the de-delayed data based on the delay time of each channel of sensors to obtain each channel of corrected data; The cache reference time corresponding to each channel of the correction data is determined, and the start frame synchronization alignment of each channel of the correction data is performed based on the cache reference time to obtain each channel of valid data.

5. The method according to claim 4, characterized in that The source time error correction of each channel of the de-delayed data is performed based on the delay time of each channel of the sensor to obtain each channel of corrected data, including: Determine the acquisition timestamp and sensor delay duration corresponding to each channel of audio and video data; The acquisition timestamp and the sensor delay duration are subjected to difference calculation, and source time error correction is performed on each channel of the de-delayed data based on the difference calculation result to obtain each channel of corrected data.

6. The method according to claim 4, characterized in that The determining of the cache reference time corresponding to each channel of the correction data, and performing start frame synchronization alignment on each channel of the correction data based on the cache reference time to obtain each channel of valid data, includes: Obtaining the start frame time corresponding to each channel of the correction data, and determining the maximum value of the start frame time of each channel based on an extreme value algorithm to obtain a cache reference time; Invalid data in each channel of the correction data that is younger than the cache reference time is cleared, and the correction data in each channel is synchronously aligned based on the cache reference time to obtain valid data in each channel.

7. The method according to claim 4, characterized in that The step of recording audio and video encoding of each channel of valid data and writing the encoded data to a disk for storage includes: Determine the acquisition timestamp information in each channel of the valid data; Encoding and recording each channel of the valid data based on each acquisition timestamp information; The encoded data is written to a disk for storage based on each of the acquisition timestamp information and the cache reference time.

8. A synchronous acquisition and storage device for multi-channel audio and video data, characterized in that: The device comprises: An acquisition module, used to acquire a synchronization start instruction, and continuously collect at least two channels of audio and video data based on the synchronization start instruction; A processing module, used for performing delay-removing collection processing on each channel of the audio and video data based on a data source delay parameter of each channel of the audio and video data to obtain each channel of delay-removing data; A synchronization module, used for performing start frame synchronization alignment on each channel of the de-delayed data based on the delay time of each channel of sensors to obtain each channel of valid data, wherein the start frame synchronization alignment is used for performing synchronization alignment on the start frames of each channel of the de-delayed data; The encoding module is used to perform audio and video encoding and recording of each channel of valid data, and write the encoded data to a disk for storage.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, wherein the computer-readable storage medium has instructions stored therein, and when the instructions are executed on a computer or a processor, the computer or the processor executes the steps of the method according to any one of claims 1 to 7.

Images