Audio and video file generation method and system, vehicle terminal and storage medium

Abstract

The application provides an audio and video file generation method and system, a vehicle-mounted terminal and a storage medium. The method comprises the following steps: obtaining driving record data, wherein the driving record data comprises video frames and audio frames, and the video frames and the audio frames are both configured with time stamps; storing the video frames and the audio frames into a video cache pool and an audio cache pool respectively based on the time stamps according to the first-in first-out principle, wherein the video cache pool and the audio cache pool are both configured with an overlay storage period, and the overlay storage period is set according to a preset audio and video duration; and generating an audio and video file according to the video data in the video cache pool and the audio data in the audio cache pool in response to a received audio and video file generation signal. The overlay storage period of the cache pool is set according to the fixed audio and video duration requirement, and the fixed duration audio and video file is generated in the frame cache technology, so that the driving recorder can still generate the fixed duration audio and video file in the environment with the actual frame rate not fixed, and the user experience is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to an audio and video file generation method and system, a vehicle terminal and a storage medium. BACKGROUND

[0002] A driving recorder is an instrument for recording images and sounds and other related information during vehicle driving. The driving recorders on the market are mostly designed in an integrated manner of a driving recorder host, a camera and an audio acquisition device. After the driving recorder is installed on a vehicle, the driving recorder can record videos, images and sounds during the whole driving process of the vehicle, can restore events in dangerous events or emergency situations, and can be used for event backtracking when a traffic accident or collision occurs.

[0003] In the related art, the driving recorder first adds watermarks to original data captured by a camera and original data collected by an audio acquisition device, then encodes video data and audio data respectively, and then performs audio and video synthesis to obtain an audio and video file to be written into a digital media file. However, in the process of generating the audio and video file, if a fixed time length file is to be generated, there is a problem that the actual time length of the generated audio and video file is greater than the set time length. SUMMARY

[0004] In view of the above-mentioned defects of the prior art, the present application discloses an audio and video file generation method and system, a vehicle terminal and a storage medium, which are used to solve the technical problem that the driving recorder in the prior art generates an audio and video file based on a fixed time length but the actual time length is greater than the set time length.

[0005] In a first aspect, the present application provides an audio and video file generation method, which comprises: acquiring driving record data, wherein the driving record data comprises video frames and audio frames, and the video frames and the audio frames are both configured with time stamps; storing the video frames and the audio frames into a video cache pool and an audio cache pool respectively based on the time stamps according to a first-in-first-out principle, wherein the video cache pool and the audio cache pool are both configured with an overlay storage period, and the overlay storage period is set according to a preset audio and video time length; and generating an audio and video file according to video data in the video cache pool and audio data in the audio cache pool in response to a received audio and video file generation signal

[0006] In an embodiment of the present application, the setting mode of the cover storage period comprises: determining a time length compensation factor according to a second-level integer rule of the audio / video time length, the second-level integer rule being downward integer, and the time length compensation factor being smaller than a second-level unit time length; and determining the cover storage period according to the audio / video time length and the time length compensation factor, the cover storage period being a sum of the audio / video time length and the time length compensation factor, so that an actual time length of the generated audio / video file is the audio / video time length.

[0007] In an embodiment of the present application, the storing of the video frame and the audio frame into the video cache pool and the audio cache pool according to the timestamps based on the first-in-first-out principle comprises: determining a first time difference between the video frame and a video starting frame in the video cache pool according to the timestamp of the video frame and a first historical timestamp of the video starting frame, and determining a second time difference between the audio frame and an audio starting frame in the audio cache pool according to the timestamp of the audio frame and a second historical timestamp of the audio starting frame; caching the video frame into the video cache pool, and removing the video starting frame from the video cache pool if the first time difference is greater than the cover storage period; and caching the audio frame into the audio cache pool, and removing the audio starting frame from the audio cache pool if the second time difference is greater than the cover storage period.

[0008] In an embodiment of the present application, the storing of the video frame and the audio frame into the video cache pool and the audio cache pool according to the timestamps based on the first-in-first-out principle comprises: sequentially calculating a third time difference between the timestamp of the video frame and a timestamp of each historical video frame in the video cache pool according to the timestamp sequence, and sequentially calculating a fourth time difference between the timestamp of the audio frame and a timestamp of each historical audio frame in the audio cache pool; caching the video frame into the video cache pool, and removing any historical video frame from the video cache pool if the third time difference of the historical video frame is greater than the cover storage period; and caching the audio frame into the audio cache pool, and removing any historical audio frame from the audio cache pool if the fourth time difference of the historical audio frame is greater than the cover storage period.

[0009] In an embodiment of the present application, the generating the audio-video file according to the video data in the video cache pool and the audio data in the audio cache pool comprises: obtaining timestamps of first and last frames in the video data, and denoting the timestamps as a first timestamp and a second timestamp respectively; removing audio frames in the audio data with timestamps earlier than the first timestamp, and / or removing audio frames in the audio data with timestamps later than the second timestamp, to obtain target audio data; and synchronously integrating the video data and the target audio data according to timestamps of each video frame in the video data and timestamps of each audio frame in the target audio data, to obtain the audio-video file.

[0010] In an embodiment of the present application, the video frame configuration type identifier is used to distinguish between key frames and non-key frames; and before the generating the audio-video file according to the video data in the video cache pool and the audio data in the audio cache pool, the method further comprises: if the first frame of the video data is a key frame, entering the operation of generating the audio-video file; and if the first frame of the video data is a non-key frame, setting the first frame of the video data as a key frame, and entering the operation of generating the audio-video file.

[0011] In an embodiment of the present application, before the storing the video frames and the audio frames into the video cache pool and the audio cache pool respectively based on the timestamps according to the first-in-first-out principle, the method further comprises: setting a frame type encoding strategy, wherein the frame type encoding strategy is setting each video frame as a key frame when encoding the video frame.

[0012] In a second aspect, the present application provides an audio-video file generation system, which comprises: an obtaining module configured to obtain driving record data, wherein the driving record data comprises video frames and audio frames, and the video frames and the audio frames are both configured with timestamps; a cache module configured to store the video frames and the audio frames into a video cache pool and an audio cache pool respectively based on the timestamps according to the first-in-first-out principle, wherein the video cache pool and the audio cache pool are both configured with an overwrite storage period, and the overwrite storage period is set according to a preset audio-video duration; and a generating module configured to generate an audio-video file according to video data in the video cache pool and audio data in the audio cache pool in response to a received audio-video file generation signal.

[0013] In a third aspect, the present application provides a vehicle-mounted terminal, which comprises: one or more processors; and a storage device configured to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the vehicle-mounted terminal implements the audio-video file generation method as described in the first aspect.

[0014] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program. When the computer program is executed by a processor of a computer, the computer program causes the computer to execute the audio and video file generation method according to the first aspect.

[0015] As described above, the audio and video file generation method, system, vehicle-mounted terminal and storage medium provided by the embodiments of the present application have the following beneficial effects:

[0016] First, the driving record data is acquired, including video frames and audio frames. The video frames and the audio frames are both configured with time stamps. Then, the video frames and the audio frames are respectively stored into a video cache pool and an audio cache pool based on the time stamps according to the first-in-first-out principle. The video cache pool and the audio cache pool are both configured with an overlay storage period. The overlay storage period is set according to a preset audio and video duration. If an audio and video file generation signal is received, the video data in the video cache pool and the audio data in the audio cache pool are used to generate an audio and video file in response to the signal. By setting the overlay storage period of the cache pool according to the fixed audio and video duration requirement, the video data and the audio data in the cache pool can be kept for a fixed duration. In this way, the frame cache technology with the set overlay storage period is used to generate the audio and video file with the fixed duration. The driving recorder can still generate the audio and video file with the fixed duration in the environment with the actual frame rate that is not fixed, and is not affected by the system performance, thereby improving the user experience.

[0017] It should be understood that the general description above and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0018] The drawings incorporated into the specification and forming a part thereof, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor. In the drawings:

[0019] Figure 1 is a schematic diagram of an implementation environment of an audio and video file generation system according to an exemplary embodiment of the present application;

[0020] Figure 2 is a flowchart of an audio and video file generation method according to an exemplary embodiment of the present application;

[0021] Figure 3 is a schematic diagram of storing video frames in a video cache pool according to an exemplary embodiment of the present application;

[0022] Figure 4is a schematic diagram of storing video frames in another video buffer pool according to an example embodiment of the present application;

[0023] Figure 5 is a schematic diagram of video file information according to an example embodiment of the present application;

[0024] Figure 6 is a schematic diagram of video data structure according to an example embodiment of the present application;

[0025] Figure 7 is a schematic diagram of another video data structure according to an example embodiment of the present application;

[0026] Figure 8 is a flow chart of a specific audio / video file generation method according to an example embodiment of the present application;

[0027] Figure 9 is a block diagram of an audio / video file generation system according to an example embodiment of the present application;

[0028] Figure 10 is a schematic diagram of a vehicle terminal according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] Other advantages and effects of the present application will be easily understood by those skilled in the art from the above description of the preferred embodiments of the present application. The present application can be implemented or applied in other different specific embodiments, and the details in the present specification can be modified or changed based on different views and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, but not for limiting the protection scope of the present application.

[0030] It should be noted that the diagrams provided in the following embodiments only schematically illustrate the basic concept of the present application, and thus only the components related to the present application are shown in the diagrams, but not the number, shape and size of the components in actual implementation, and the shape, number and ratio of the components in actual implementation can be arbitrarily changed, and the layout form of the components can be more complex.

[0031] In the following description, a large number of details are discussed to provide a more thorough explanation of the embodiments of the present application, however, it is obvious for those skilled in the art that the embodiments of the present application can be implemented without these specific details, and in other embodiments, the well-known structures and devices are shown in the form of block diagrams instead of details, to avoid making the embodiments of the present application difficult to understand.

[0032] The vehicle recorder first adds a watermark to the original data captured by the camera and the original data collected by the audio collection device, then encodes the video data and the audio data respectively, and then performs audio and video synthesis to obtain an audio and video file to write into a digital media file. For example, in the vehicle machine system of the Android operating system, the original data is added with a watermark through the OpenGL (Open Graphics Library) technology, and then encoded through the MediaCodec (multimedia codec), and then the audio and video are synthesized through the MediaMuxer (audio and video packaging class), and the synthesized data is written into the mp4 file.

[0033] The inventors have found that in the process of generating an audio and video file, the audio and video file is synthesized according to the number of frames, and if a fixed-length file is to be generated, there may be a problem that the actual length of the generated audio and video file is greater than the set length. For example, in order to generate an audio and video file with a fixed length of 30S, if the frame rate is 25 frames per second, a total of 30*25=750 frames of data are required, and the audio and video file is synthesized according to 750 frames of data. If the ideal situation (the vehicle machine system runs stably) is 25 frames per second, due to the influence of the stability of the vehicle machine system, the frame rate of the video frame and the audio frame will be unstable, and the ideal situation is 25 frames per second, the actual frame rate will fluctuate around 25 frames per second, for example, between 15 and 25 frames per second, that is, the vehicle machine system is a little bit lagging, and it may be around 20 frames per second, and if it is more lagging, it may be around 15 frames per second, so the number of frames per second will be reduced compared to the ideal situation, and the audio and video file synthesized according to 750 frames of data will be greater than 30S, and the total length of the audio and video file is not fixed due to the uncertainty of the system running stably.

[0034] If the length of the audio and video file generated by the vehicle recorder is not fixed, at least the following problems exist. For example, storage management is complicated, if the file size and length are inconsistent, it will be more difficult to manage the storage space; retrieval and playback are inconvenient, when an accident or event occurs, it is necessary to quickly find the audio and video file of the relevant period, if the length is not fixed, it will be complicated to locate the segment of a specific time point; data analysis is difficult, for the automatic analysis of vehicle recorder data, the fixed length will make the automatic data processing and analysis more complicated, and even the algorithm needs to be adjusted for different lengths of files.

[0035] Therefore, please refer to Figure 1 , Figure 1 is an example embodiment of the present application, which shows an implementation environment diagram of an audio and video file generation system. As Figure 1As shown, the implementation environment includes a vehicle 110 and an audio and video file generation system 120, wherein the audio and video file generation system 120 is embedded in the vehicle 110 for realizing generation of an audio and video file of a driving recorder in the vehicle 110, the audio and video file generation system 120 includes but is not limited to a car machine system, a vehicle-mounted computer, etc., by setting a fixed audio and video time length requirement, a coverage storage period of a cache pool is set, so that video data and audio data in the cache pool can be kept in the set fixed time length, in this way, the frame cache technology with the set coverage storage period is used to generate the audio and video file with the fixed time length, which can ensure that the driving recorder can still generate the audio and video file with the fixed time length in the environment where the actual frame rate is not fixed, is not affected by system performance, and improves user experience.

[0036] Please refer to Figure 2 , Figure 2 is a flowchart of an audio and video file generation method according to an example embodiment of the present application. The method can be applied to Figure 1 the implementation environment as shown, it should be understood that the method can also be applied to other example implementation environments, and the present embodiment does not limit the implementation environment to which the method is applied.

[0037] In addition, the audio and video file generation method can be applied to a car machine system under various operating systems, including but not limited to a car machine system under an Android operating system.

[0038] As shown in Figure 2 in an example embodiment, the audio and video file generation method at least includes steps S210 to S230, which are described in detail as follows:

[0039] Step S210, driving record data is acquired, the driving record data includes video frames and audio frames, and the video frames and the audio frames are both configured with time stamps.

[0040] Among them, the video frames are acquired from a camera of the driving recorder, and the audio frames are acquired from a corresponding recording device of the driving recorder; the time stamps configured for the video frames and the audio frames use system relative time, rather than acquiring the current system time, so as to avoid the problem that if the user actively modifies the system time, the acquired time and the configured time stamp will be wrong.

[0041] In addition, the video frames and the audio frames are acquired at the same time, and in an ideal case, the time stamps configured for the video frames and the audio frames are the same, since the video frames and the audio frames come from different devices, although the video frames and the audio frames are acquired at the same time, there may be errors in the time stamps marked by the two.

[0042] In step S220, the video frames and the audio frames are respectively stored into the video cache pool and the audio cache pool based on the timestamps according to the first-in-first-out principle, and the video cache pool and the audio cache pool are both configured with an overlay storage period, and the overlay storage period is set according to a preset audio-video duration.

[0043] In this embodiment, if the driving recorder is started, the data in the video cache pool and the audio cache pool is in a state of continuous updating, that is, the video frames and the audio frames are continuously acquired and stored into the video cache pool and the audio cache pool based on the timestamps according to the first-in-first-out principle. Therefore, the duration of the data in the video cache pool and the audio cache pool is always the duration corresponding to the overlay storage period, that is, the required audio-video duration is maintained, so that the frame cache technology of setting the overlay storage period achieves the purpose of generating an audio-video file with a fixed duration.

[0044] In a possible embodiment, after the video frames are acquired from the camera of the driving recorder and the audio frames are acquired from the corresponding recording device of the driving recorder, the video frames and the audio frames are respectively cached in the corresponding total cache pool. In the total cache pool, the overlay storage principle of first-in-first-out is not involved, but all the video frames or audio frames captured after the driving recorder is started are stored, and all the frame data is saved. If the cleaning condition of the data in the total cache pool is reached, the data is cleaned. The cleaning condition includes but is not limited to that the data memory reaches a preset value, the time reaches a cleaning time, and a user triggers a cleaning request, etc.

[0045] As a possible embodiment, the video frames are stored in the corresponding total cache pool, in which case the data in the total cache pool is cleaned only when the cleaning condition is reached, and the video frames are also stored in the video cache pool, in which case the video frames are stored into the video cache pool based on the timestamps according to the first-in-first-out principle. The audio frames are the same, and will not be described in detail. The data in the video cache pool and the data in the audio cache pool are used to generate an audio-video file with a fixed duration in response to an audio-video file generation signal, and the data in the total cache pool can also be taken out to generate an audio-video file according to specific requirements, for example, data of other duration or some time period can be taken out to generate an audio-video file.

[0046] In an embodiment, the setting mode of the overlay storage period includes: determining a duration compensation factor according to a second-level integer rule of the audio-video duration, the second-level integer rule being downward integer, and the duration compensation factor being less than a second-level unit duration; determining the overlay storage period according to the audio-video duration and the duration compensation factor, the overlay storage period being the sum of the audio-video duration and the duration compensation factor, so that the actual duration of the generated audio-video file is the audio-video duration.

[0047] In this embodiment, considering that if the cover storage period is set as a preset audio / video duration, when the video frames and the audio frames are respectively stored into the video cache pool and the audio cache pool based on timestamps according to the first-in-first-out principle due to system lag, the video data in the video cache pool and / or the audio data in the audio cache pool may be less than the preset audio / video duration, therefore, a duration compensation factor is introduced, the duration compensation factor is a duration value, the cover storage period is determined according to the audio / video duration and the duration value, and under the rule of taking the second level of the audio / video duration as an integer, in the case of system instability, the actual duration of the generated audio / video file will not exceed the preset audio / video duration, nor will it be lower than the preset audio / video duration, and the duration of the generated audio / video file each time is the preset audio / video duration.

[0048] Taking the generation of an audio / video file with a fixed duration of 30S as an example, due to system lag, the timestamp difference between two frames may be large, at this time, after the previous frame greater than the cover storage period is removed, the timestamp difference between the next frame and the tail frame in the cache pool may be less than 30S, that is, the duration of the video data in the video cache pool and the duration of the audio data in the audio cache pool may be less than 30S. Therefore, according to the rule of taking the second level of the audio / video duration as an integer, that is, if the duration of the data is in [30, 31), the duration of the generated audio / video file is 30S, therefore, the duration value of the duration compensation factor is a value greater than 0S and less than 1S (second level unit duration), in this way, the cover storage period is a period greater than 30S and less than 31S, which can effectively avoid the problem that the duration of the video data and the duration of the audio data are less than the preset audio / video duration due to system lag.

[0049] In an embodiment, storing the video frames and the audio frames into the video cache pool and the audio cache pool based on timestamps according to the first-in-first-out principle comprises: determining a first time difference between the video frame and a video starting frame according to the timestamp of the video frame and a first historical timestamp of the video starting frame in the video cache pool, and determining a second time difference between the audio frame and an audio starting frame according to the timestamp of the audio frame and a second historical timestamp of the audio starting frame in the audio cache pool; caching the video frame into the video cache pool, and removing the video starting frame from the video cache pool under the condition that the first time difference is greater than the cover storage period; caching the audio frame into the audio cache pool, and removing the audio starting frame from the audio cache pool under the condition that the second time difference is greater than the cover storage period.

[0050] In this embodiment, the new video frame is put at the end of the video cache pool and compared with the timestamp of the video start frame in the video cache pool. If the time difference between the two is greater than the cover storage period, the video start frame is removed from the video cache pool. Similarly, the new audio frame is put at the end of the audio cache pool and compared with the timestamp of the audio start frame in the audio cache pool. If the time difference between the two is greater than the cover storage period, the audio start frame is removed from the video cache pool. In this way, the new data frame is compared with the start frame. In the video cache pool and the audio cache pool, the first-in first-out of the data frame is realized based on the timestamp and the cover storage period. Regardless of the actual frame rate of the system, the video data in the cache pool and the audio data in the audio cache pool can be kept at a fixed audio-video time length.

[0051] Please refer to Figure 3 , Figure 3 is an exemplary embodiment of the present application, which shows a schematic diagram of storing video frames in a video cache pool. As Figure 3 shown, after the new video frame enters the video cache pool, the time difference between the new video frame and the start frame is calculated. The new video frame is stored at the end of the video cache pool. If the calculated time difference is greater than the cover storage period, the start frame is removed from the video cache pool.

[0052] In an embodiment, the video frame and the audio frame are respectively stored in the video cache pool and the audio cache pool based on the timestamp according to the first-in first-out principle, comprising: calculating the third time difference between the timestamp of the video frame and the timestamp of each historical video frame in the video cache pool in order according to the timestamp sequence, and calculating the fourth time difference between the timestamp of the audio frame and the timestamp of each historical audio frame in the audio cache pool in order; caching the video frame to the video cache pool. If the third time difference of any historical video frame is greater than the cover storage period, any historical video frame is removed from the video cache pool; caching the audio frame to the audio cache pool. If the fourth time difference of any historical audio frame is greater than the cover storage period, any historical audio frame is removed from the audio cache pool.

[0053] In this embodiment, in the case of severe system lag, that is, the time stamp of the new data frame and the time stamp of the previous frame have a large difference, the time stamp of the new data frame and the time stamp of the data frame at the head of the cache pool are both greater than the cover storage period, so the time difference between the new data frame and the data frame at the head is calculated, and the video frame with a time difference greater than the cover storage period is removed from the cache pool until there is no case where the time difference is greater than the cover storage period. In this way, the new data frame is compared with each data frame from the head, and the first-in first-out of the data frame is realized in the video cache pool and the audio cache pool based on the time stamp and the cover storage period. Regardless of the actual frame rate of the system, the video data in the cache pool and the audio data in the audio cache pool can be kept at a fixed audio-video time length.

[0054] Please refer to Figure 4 , Figure 4 is another schematic diagram of storing video frames in a video cache pool according to an example embodiment of the present application. As shown in Figure 4 , after the new video frame enters the video cache pool, the time difference between the new video frame and each video frame at the head is calculated, and the video frame with a time difference greater than the cover storage period is removed from the video cache pool.

[0055] Step S230, in response to the received audio-video file generation signal, the video data in the video cache pool and the audio data in the audio cache pool are used to generate an audio-video file.

[0056] The audio-video file generation signal is generated based on an event trigger, and the event includes but is not limited to sudden acceleration or deceleration (such as collision or sudden braking), user operation, and preset generation mechanism reaching conditions (such as timing generation, periodic generation, etc.).

[0057] In this embodiment, if the audio-video file generation signal is received, the video data in the video cache pool and the audio data in the audio cache pool are read in response to the signal, and an audio-video file is synthesized. Based on the frame buffer technology of the video cache pool and the audio cache pool, the generated audio-video file always maintains a preset audio-video time length.

[0058] In an embodiment, generating an audio-video file according to the video data in the video cache pool and the audio data in the audio cache pool includes: obtaining the time stamps of the head and tail frames in the video data, which are respectively denoted as the first time stamp and the second time stamp; removing the audio frames in the audio data with a time stamp earlier than the first time stamp, and / or removing the audio frames in the audio data with a time stamp later than the second time stamp to obtain target audio data; and synchronously integrating the video data and the target audio data according to the time stamps of the video frames in the video data and the time stamps of the audio frames in the target audio data to obtain an audio-video file.

[0059] In this embodiment, considering that the video data and the audio data are separately encoded and buffered, the read-out video data and audio data can have the problem that the head and tail frame timestamps do not correspond, that is, the audio and video are not synchronized, so it is necessary to synchronize and integrate the video data and the audio data to ensure the quality of the audio-video file.

[0060] As a possible embodiment, the audio data is corrected based on the video data, that is, the audio frames with timestamps earlier than the head frame timestamp in the video data are removed, and / or the audio frames with timestamps later than the tail frame timestamp in the video data are removed, and then the video data and the target audio data are synchronized and integrated according to the timestamps of the video frames in the video data and the timestamps of the audio frames in the target audio data, so as to obtain an audio-video file. In this way, in the synthesized audio-video file, there will be no problem of having sound but no picture and the sound and picture being out of synchronization, and on the basis of ensuring that the audio-video file remains at a fixed time length, the quality of the audio-video file is also ensured, and the user experience is improved.

[0061] In addition, in this embodiment, the video data and the audio data both remain at a fixed time length, and due to the error in data acquisition, the cases in which the audio data and the video data cannot be aligned in the time period based on the video data include: the timestamp of the head frame of the audio data is earlier than the timestamp of the head frame of the video data, the timestamp of the tail frame of the audio data is later than the timestamp of the tail frame of the video data, and the timestamp of the head frame of the audio data is both earlier than the timestamp of the head frame of the video data and the timestamp of the tail frame of the audio data is later than the timestamp of the tail frame of the video data. Therefore, the correction of the audio data includes: removing the audio frames with timestamps earlier than the head frame timestamp in the video data, removing the audio frames with timestamps later than the tail frame timestamp in the video data, and removing both the audio frames with timestamps earlier than the head frame timestamp in the video data and the audio frames with timestamps later than the tail frame timestamp in the video data.

[0062] In actual audio-video file synthesis, the frame buffering technology can avoid the generated audio-video file exceeding the fixed audio-video time length, but through checking the detailed information of the generated audio-video file, the problem of the actual time length being less than the audio-video time length can occur. Please refer to Figure 5 , Figure 5 is a schematic diagram of a video file information according to an exemplary embodiment of the present application. As shown in Figure 5 , taking 30S as the fixed audio-video time length, if the total frame number of the 30S video data before the audio-video file synthesis is 570 frames, the actual frame number can be only 550 frames. When the actual frame number is only 550 frames, the time length of the synthesized audio-video file can be less than 30S.

[0063] The above situation occurs because when a video frame encoder is created before encoding a video frame, a frame rate and a key frame interval parameter need to be set, for example, the key frame interval is set to 1 second, that is, the video frame encoder sets a key frame every 1 second. When synthesizing an audio and video file, if the video data is a non-key frame at the beginning, the non-key frame data needs to be discarded until a key frame is found before starting synthesis. Among them, the key frame is a completely self-contained video frame, which does not depend on other frames for decoding, and each key frame contains all the information of the picture at that moment and can be decoded and displayed independently. The non-key frame is a reference to the previous key frame or non-key frame and represents the difference between the current frame and the reference frame through predictive coding. For example, in the car system under the Android operating system, after researching the audio and video encoding MediaCodec and audio and video synthesis MediaMuxer, it is found that when synthesizing audio and video, if the first frame obtained is not a key frame, MediaMuxer will discard these data until the obtained frame is a key frame, and then start data synthesis and writing.

[0064] Please refer to Figure 6 , Figure 6 is a structure diagram of video data shown by an exemplary embodiment of the present application. As Figure 6 indicated, the key frame is represented by I frame, and the non-key frame is represented by P frame. If the first several frames of the video data header are P frames, these P frames need to be discarded until an I frame is found to start synthesis.

[0065] Because the audio and video file generation signal is random, it cannot be ensured that the first frame of the video data is a key frame. Therefore, if a series of non-key frames are discarded, the originally fixed-length video data will have the problem that the actual length after synthesis is lower than the set fixed length.

[0066] In an embodiment, the video frame configuration type identifier is used to distinguish between key frames and non-key frames. Before generating an audio and video file according to the video data in the video cache pool and the audio data in the audio cache pool, the method further includes: if the first frame of the video data is a key frame, entering the generation operation of the audio and video file; if the first frame of the video data is a non-key frame, setting the first frame of the video data as a key frame, and entering the generation operation of the audio and video file.

[0067] As a possible embodiment, before generating the audio and video file, first determine whether the first frame of the video data is a key frame. If so, there is no frame loss at this time, and directly enter the audio and video file generation operation. If not, avoid discarding non-key frames, set the first frame as a key frame, and then enter the audio and video file generation operation. In this way, there will be no frame loss when the audio and video files are synthesized, avoiding the problem that the actual length of the audio and video files will be lower than the fixed length.

[0068] See Figure 7 , Figure 7 FIG. 1 is a schematic diagram showing another structure of video data according to an exemplary embodiment of the present application. Figure 7 As shown, in the video data, if the first frame of the video data is a P frame, the frame is set as an I frame.

[0069] In one embodiment, before the video frames and audio frames are stored in the video cache pool and the audio cache pool respectively based on the timestamp and the first-in-first-out principle, it also includes: setting a frame type encoding strategy, and the frame type encoding strategy is to set each video frame as a key frame when encoding the video frame.

[0070] As a possible embodiment, in order to avoid the problem that the initial non-key frames in the video data will be discarded when synthesizing audio and video files, each frame is set as a key frame when encoding the video frame, that is, the frame type encoding strategy of the video frame encoder is configured to set each video frame as a key frame when encoding the video frame. In this way, no frames will be lost when synthesizing audio and video files, and the problem that the actual length of the audio and video files will be lower than the fixed length is avoided.

[0071] In a possible embodiment, the received audio and video file generation signal carries the event occurrence time and time requirements, that is, when the audio and video file is generated based on an event trigger, the event also sets the time requirements for the audio and video file. The time requirement can be understood as the required audio and video files are audio and video of a period of time before and after the event occurs. At this time, based on the event occurrence time, continue to wait for the data in the cache pool to meet the time requirements, and then read the data from the video cache pool and the audio cache pool to synthesize the audio and video file.

[0072] As a possible embodiment, audio and video files with different needs can be generated based on time requirements, thereby improving user experience.

[0073] For example, the fixed length of the audio and video file is 30 seconds. When a vehicle collides, 15 seconds of audio and video are required before and after the collision. After the received audio and video file generates a signal, the data in the video cache pool and the audio cache pool are updated for another 15 seconds, and then the data is read out to synthesize the audio and video file.

[0074] SeeFigure 8 , Figure 8 is a flow chart of a specific audio and video file generation method shown in an exemplary embodiment of the present application. As shown in Figure 8 , taking the car machine system under the Android operating system as an example, the specific audio and video file generation method is described as follows: after the car machine system and the driving recorder are started, the driving recorder opens the camera to obtain the video frame, and opens the recording to obtain the audio frame; after the video frame and the audio frame are obtained, the watermark is added through OpenGL; then the video frame is encoded through MediaCodec, including setting the timestamp and type identifier, and the audio frame is encoded through MediaCodec, including setting the timestamp; then the encoded video frame is cached to VideoQueue (video cache pool), and the encoded audio frame is cached to AudioQueue (audio cache pool), and the video frame and the audio frame are respectively stored in VideoQueue and AudioQueue according to the coverage storage period; if the audio and video file generation signal is monitored, MediaMuxer obtains the video data in VideoQueue and the audio data in AudioQueue, and corrects the audio data based on the video data; then it is judged whether the first frame of the video data is a key frame, if yes, the audio and video file is directly generated according to the video data and the audio data, if not, the first frame of the video data is set as a key frame, and then the audio and video file is generated according to the video data and the audio data.

[0075] In the exemplary embodiment, the timing between the correction operation of the audio data and the judgment operation of whether the first frame of the video data is a key frame is not limited.

[0076] In this way, by combining the frame cache technology, the key frame setting strategy and the synchronization integration strategy between the video data and the audio data, it is ensured that the driving recorder can record an audio and video file with a fixed time length in the environment where the actual frame rate is not fixed, and is not affected by the performance of the system, so as to facilitate the storage management, retrieval and playback of the audio and video file, data analysis and the like, and improve the user experience

[0077] The audio and video file generation method described above first acquires driving record data, including video frames and audio frames, and the video frames and the audio frames are both configured with time stamps, then the video frames and the audio frames are respectively stored into a video cache pool and an audio cache pool based on the time stamps according to the first-in first-out principle, wherein the video cache pool and the audio cache pool are both configured with an overlay storage period, and the overlay storage period is set according to a preset audio and video duration, if an audio and video file generation signal is received, then in response to the signal, an audio and video file is generated according to the video data in the video cache pool and the audio data in the audio cache pool, by setting the overlay storage period of the cache pool according to the fixed audio and video duration requirement, the video data and the audio data in the cache pool can be kept at the set fixed duration, in this way, the frame cache technology with the set overlay storage period is used to generate the audio and video file with the fixed duration, which can ensure that the driving recorder can still generate the audio and video file with the fixed duration in the environment with the actual frame rate not fixed, is not affected by the system performance, and the user experience is improved.

[0078] See Figure 9 , Figure 9 is a block diagram of a command line based ECU access system according to an example embodiment of the present application. The system can be applied in the implementation environment shown in Figure 1 It should be understood that the system can also be applied to other example implementation environments, and the implementation environment to which the system is applied is not limited by the present embodiment.

[0079] As shown in Figure 9 , in an example embodiment, the audio and video file generation system 900 at least includes an acquisition module 910, a cache module 920 and a generation module 930, which are described in detail as follows:

[0080] The acquisition module 910 is configured to acquire driving record data, and the driving record data includes video frames and audio frames, and the video frames and the audio frames are both configured with time stamps;

[0081] The cache module 920 is configured to store the video frames and the audio frames into a video cache pool and an audio cache pool respectively based on the time stamps according to the first-in first-out principle, and the video cache pool and the audio cache pool are both configured with an overlay storage period, and the overlay storage period is set according to a preset audio and video duration;

[0082] The generation module 930 is configured to generate an audio and video file according to the video data in the video cache pool and the audio data in the audio cache pool in response to a received audio and video file generation signal.

[0083] It should be noted that the audio and video file generation system provided by the above embodiment and the audio and video file generation method provided by the above embodiment belong to the same concept, and the content of the operation of each module has been described in detail in the method embodiment, which will not be described here.

[0084] Please refer to Figure 10 , Figure 10 is a structural schematic diagram of a vehicle terminal provided by an embodiment of the present application. Figure 10 A structural schematic diagram of a computer system of a vehicle terminal suitable for implementing embodiments of the present application is shown. It should be noted that Figure 10 The computer system 1000 of the vehicle terminal shown is only an example and should not bring any limitation to the functions and use range of embodiments of the present application.

[0085] As shown in Figure 10 , the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage portion 1008 into a random access memory (RAM) 1003, such as performing the methods in the above embodiments. Various programs and data required for system operation are also stored in the RAM 1003. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other through a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0086] The following components are connected to the I / O interface 1005: an input portion 1006 including a keyboard, a mouse, and the like; an output portion 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), and the like, and a speaker, and the like; a storage portion 1008 including a hard disk, and the like; and a communication portion 1009 including a network interface card such as a LAN (Local Area Network) card, a modem, and the like. The communication portion 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as necessary. A removable recording medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like is attached to the drive 1010 as necessary, so that a computer program read therefrom is installed in the storage portion 1008 as necessary.

[0087] In particular, according to embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing instructions for carrying out the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication section 1009, and / or installed from the detachable medium 1011. When the computer program is executed by the central processing unit (CPU) 1001, various functions defined in the system of the present application are executed.

[0088] The present application also provides a computer readable storage medium, which stores a computer program, and the computer program, when executed by a processor of a computer, causes the computer to perform the audio / video file generation method as described above. The computer readable storage medium can be included in the vehicle terminal described in the above embodiments, or can exist separately and not be assembled into the vehicle terminal.

[0089] It should be noted that the computer readable medium shown in the embodiments of the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable signal medium can include a data signal carried in a baseband or as a carrier wave in a propagated data signal, which carries the computer readable computer program. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium, which can send, propagate or transmit the program for use by or in connection with an instruction execution system, device or apparatus. The computer program contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to wireless, wired, or the like, or any suitable combination of the above.

[0090] The flow and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flow and block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the flow and block diagrams can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may be executed in the reverse order, depending on the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or combinations of hardware and software.

[0091] The units described in the embodiments of the present application can be implemented by software, or can be implemented by hardware, and the units described can also be arranged in a processor. In some cases, the names of the units do not constitute a limitation on the units themselves.

[0092] The above-described embodiments only illustrate the principles of the present application and its effects, and are not used to limit the present application. Any modification or change made by any person skilled in the art without departing from the spirit and scope of the present application shall be covered by the claims of the present application.

Claims

1. A method for generating an audio and video file, characterized in that: The method comprises: Acquire driving record data, the driving record data including video frames and audio frames, and the video frames and the audio frames are both configured with timestamps; The video frame and the audio frame are stored in a video buffer pool and an audio buffer pool respectively according to a first-in-first-out principle based on the timestamp, and both the video buffer pool and the audio buffer pool are configured with an overlay storage period, and the overlay storage period is set according to a preset audio and video duration; In response to the received audio and video file generation signal, generating an audio and video file according to the video data in the video buffer pool and the audio data in the audio buffer pool; The method for setting the overwriting storage period includes: Determine a duration compensation factor according to a second-level rounding rule for the audio and video duration, wherein the second-level rounding rule is rounding down, and the duration compensation factor is less than the second-level unit duration; The coverage storage period is determined based on the audio and video duration and the duration compensation factor, and the coverage storage period is the sum of the audio and video duration and the duration compensation factor, so that the actual duration of the generated audio and video file is the audio and video duration.

2. The method for generating an audio and video file according to claim 1, wherein: The step of storing the video frame and the audio frame in a video buffer pool and an audio buffer pool respectively based on the timestamp and in accordance with a first-in-first-out principle includes: Determining a first time difference between the video frame and the video start frame according to the timestamp of the video frame and a first historical timestamp of a video start frame in the video buffer pool, and determining a second time difference between the audio frame and the audio start frame according to the timestamp of the audio frame and a second historical timestamp of an audio start frame in the audio buffer pool; caching the video frame into the video cache pool, and moving the video start frame out of the video cache pool under the condition that the first time difference is greater than the coverage storage period; The audio frame is cached in the audio cache pool, and under the condition that the second time difference is greater than the coverage storage period, the audio start frame is moved out of the audio cache pool.

3. The method for generating an audio and video file according to claim 1, wherein: The step of storing the video frame and the audio frame in a video buffer pool and an audio buffer pool respectively based on the timestamp and in accordance with a first-in-first-out principle includes: Calculating, in sequence according to the order of the timestamps, a third time difference between the timestamp of the video frame and the timestamp of each historical video frame in the video buffer pool, and calculating, in sequence, a fourth time difference between the timestamp of the audio frame and the timestamp of each historical audio frame in the audio buffer pool; caching the video frame in the video cache pool, and if the third time difference between the video frame and any historical video frame is greater than the overwriting storage period, removing the any historical video frame from the video cache pool; The audio frame is cached in the audio cache pool, and if the fourth time difference between the audio frame and any historical audio frame is greater than the overwriting storage period, the any historical audio frame is removed from the audio cache pool.

4. The method for generating an audio and video file according to claim 1, wherein: Generating an audio and video file according to the video data in the video buffer pool and the audio data in the audio buffer pool includes: Obtaining timestamps of the first and last frames in the video data, which are recorded as a first timestamp and a second timestamp respectively; removing audio frames whose timestamps are earlier than the first timestamp in the audio data, and / or removing audio frames whose timestamps are later than the second timestamp in the audio data, to obtain target audio data; The video data and the target audio data are synchronously integrated according to the timestamp of each video frame in the video data and the timestamp of each audio frame in the target audio data to obtain the audio and video file.

5. The method for generating an audio and video file according to any one of claims 1 to 4, characterized in that: The video frame is configured with a type identifier, where the type identifier is used to distinguish between key frames and non-key frames; Before generating the audio and video file according to the video data in the video buffer pool and the audio data in the audio buffer pool, the method further includes: If the first frame of the video data is the key frame, the process proceeds to the generation operation of the audio and video file; If the first frame of the video data is the non-key frame, the first frame of the video data is set as the key frame, and the generation operation of the audio and video file is started.

6. The method for generating an audio and video file according to any one of claims 1 to 4, characterized in that: Before storing the video frame and the audio frame in the video buffer pool and the audio buffer pool respectively according to the first-in-first-out principle based on the timestamp, the method further includes: A frame type encoding strategy is set, wherein each video frame is set as a key frame during video frame encoding.

7. A system for generating audio and video files, characterized in that: The system comprises: An acquisition module, configured to acquire driving record data, wherein the driving record data includes video frames and audio frames, and both the video frames and the audio frames are configured with timestamps; A cache module, configured to store the video frames and the audio frames in a video cache pool and an audio cache pool, respectively, based on the timestamps and in a first-in-first-out principle, wherein both the video cache pool and the audio cache pool are configured with an overlay storage period, and the overlay storage period is set according to a preset audio and video duration; A generating module, configured to generate a signal in response to a received audio and video file, and generate an audio and video file according to the video data in the video buffer pool and the audio data in the audio buffer pool; The cache module is further specifically used for: Determine a duration compensation factor according to a second-level rounding rule for the audio and video duration, wherein the second-level rounding rule is rounding down, and the duration compensation factor is less than the second-level unit duration; The coverage storage period is determined based on the audio and video duration and the duration compensation factor, and the coverage storage period is the sum of the audio and video duration and the duration compensation factor, so that the actual duration of the generated audio and video file is the audio and video duration.

8. A vehicle-mounted terminal, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the vehicle-mounted terminal to implement the audio and video file generation method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer is caused to execute the method for generating an audio and video file as claimed in any one of claims 1 to 6.

Images