Audio and video file generation method and system, vehicle-mounted terminal and storage medium

Abstract

The invention provides an audio and video file generation method and system, a vehicle-mounted terminal and a storage medium, and the method comprises the steps: obtaining driving record data, the driving record data comprises a video frame and an audio frame, and the video frame and the audio frame are each provided with a timestamp; the video frames and the audio frames are respectively stored in a video cache pool and an audio cache pool based on timestamps according to a first-in first-out principle, the video cache pool and the audio cache pool are both configured with coverage storage periods, and the coverage storage periods are set according to preset audio and video duration; and in response to a received audio and video file generation signal, generating an audio and video file according to the video data in the video cache pool and the audio data in the audio cache pool. According to the method, the coverage storage period of the cache pool is set according to the fixed audio and video duration demand, and the audio and video file with the fixed duration is generated in a frame cache technology mode, so that the automobile data recorder can still generate the audio and video file with the fixed duration in an environment with an unfixed actual frame rate, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method and system for generating audio and video files, a vehicle-mounted terminal and a storage medium. Background Art

[0002] A driving recorder is an instrument that records images, sounds and other related information while a vehicle is driving. Most driving recorders on the market use an integrated design of a driving recorder host, camera and audio acquisition equipment. After installing a driving recorder on a vehicle, it can record the video, images and sounds of the entire driving process of the vehicle, and can restore dangerous events or emergency situations, especially when the vehicle is involved in a traffic accident or collision, it can be used for event backtracking.

[0003] In the related art, the driving recorder first adds watermarks to the original data captured by the camera and the original data collected by the audio acquisition device, then encodes the video data and audio data respectively, and then performs audio and video synthesis to obtain audio and video files to be written into digital media files. However, in the process of generating audio and video files, if you want to generate a file with a fixed length, there is still a problem that the actual length of the generated audio and video files is greater than the set length. Summary of the invention

[0004] In view of the shortcomings of the prior art mentioned above, the present application discloses a method, system, vehicle-mounted terminal and storage medium for generating audio and video files, which are used to solve the technical problem in the prior art that a driving recorder generates audio and video files based on a fixed duration but the actual duration is greater than the set duration.

[0005] In a first aspect, the present application provides a method for generating an audio and video file, the method comprising: obtaining driving record data, the driving record data comprising video frames and audio frames, the video frames and the audio frames being both configured with timestamps; storing the video frames and the audio frames in a video cache pool and an audio cache pool respectively based on the timestamps according to a first-in-first-out principle, the video cache pool and the audio cache pool being both configured with a coverage storage period, the coverage storage period being set according to a preset audio and video duration; generating an audio and video file in response to a received audio and video file generation signal, according to the video data in the video cache pool and the audio data in the audio cache pool.

[0006] In one embodiment of the present application, the setting method of the coverage storage period includes: determining the duration compensation factor according to the second-level rounding rule of the audio and video duration, the second-level rounding rule is rounding down, and the duration compensation factor is less than the second-level unit duration; determining the coverage storage period according to the audio and video duration and the duration compensation factor, the coverage storage period is the sum of the audio and video duration and the duration compensation factor, so that the actual duration of the generated audio and video file is the audio and video duration.

[0007] In one embodiment of the present application, the video frame and the audio frame are stored in a video cache pool and an audio cache pool respectively based on the timestamp according to the first-in-first-out principle, including: determining a first time difference between the video frame and the video start frame according to the timestamp of the video frame and the first historical timestamp of the video start frame in the video cache pool, and determining a second time difference between the audio frame and the audio start frame according to the timestamp of the audio frame and the second historical timestamp of the audio start frame in the audio cache pool; caching the video frame in the video cache pool, and moving the video start frame out of the video cache pool under the condition that the first time difference is greater than the overlay storage period; caching the audio frame in the audio cache pool, and moving the audio start frame out of the audio cache pool under the condition that the second time difference is greater than the overlay cache period.

[0008] In one embodiment of the present application, the video frames and the audio frames are stored in a video cache pool and an audio cache pool respectively based on the timestamps according to the first-in-first-out principle, including: calculating the third time difference between the timestamp of the video frame and the timestamp of each historical video frame in the video cache pool in sequence according to the order of timestamps, and calculating the fourth time difference between the timestamp of the audio frame and the timestamp of each historical audio frame in the audio cache pool in sequence; caching the video frame in the video cache pool, and if the third time difference of any of the historical video frames is greater than the overwriting storage period, then moving any of the historical video frames out of the video cache pool; caching the audio frame in the audio cache pool, and if the fourth time difference of any of the historical audio frames is greater than the overwriting storage period, then moving any of the historical audio frames out of the audio cache pool.

[0009] In one embodiment of the present application, the generation of an audio and video file based on the video data in the video cache pool and the audio data in the audio cache pool includes: obtaining the timestamps of the first and last frames in the video data, recorded as a first timestamp and a second timestamp, respectively; removing the audio frames in the audio data whose timestamps are earlier than the first timestamp, and / or removing the audio frames in the audio data whose timestamps are later than the second timestamp, to obtain target audio data; and synchronously integrating the video data and the target audio data according to the timestamps of each video frame in the video data and the timestamps of each audio frame in the target audio data to obtain the audio and video file.

[0010] In one embodiment of the present application, the video frame is configured with a type identifier, and the type identifier is used to distinguish between key frames and non-key frames; before generating an audio and video file based on the video data in the video cache pool and the audio data in the audio cache pool, it also includes: if the first frame of the video data is the key frame, then entering the audio and video file generation operation; if the first frame of the video data is the non-key frame, then setting the first frame of the video data as the key frame, and entering the audio and video file generation operation.

[0011] In one embodiment of the present application, before storing the video frame and the audio frame in the video cache pool and the audio cache pool respectively according to the first-in-first-out principle based on the timestamp, it also includes: setting a frame type encoding strategy, and the frame type encoding strategy is to set each video frame as a key frame when encoding the video frame.

[0012] In the second aspect, the present application provides an audio and video file generation system, the system comprising: an acquisition module, used to acquire driving record data, the driving record data comprising video frames and audio frames, the video frames and the audio frames are both configured with timestamps; a cache module, used to store the video frames and the audio frames in a video cache pool and an audio cache pool respectively based on the timestamps according to the first-in-first-out principle, the video cache pool and the audio cache pool are both configured with a coverage storage period, and the coverage storage period is set according to a preset audio and video duration; a generation module, used to generate a signal in response to a received audio and video file, and generate an audio and video file according to the video data in the video cache pool and the audio data in the audio cache pool.

[0013] In a third aspect, the present application provides a vehicle-mounted terminal, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by one or more processors, the vehicle-mounted terminal implements the audio and video file generation method described in the first aspect.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor of a computer, the computer executes the method for generating an audio and video file as described in the first aspect.

[0015] As described above, the audio and video file generation method, system, vehicle terminal and storage medium provided by the embodiments of the present application have the following beneficial effects:

[0016] First, the driving record data is obtained, including video frames and audio frames, both of which are configured with timestamps, and then the video frames and audio frames are stored in the video cache pool and the audio cache pool respectively according to the first-in-first-out principle based on the timestamps, wherein the video cache pool and the audio cache pool are both configured with a coverage storage period, and the coverage storage period is set according to the preset audio and video duration. If an audio and video file generation signal is received, then in response to the signal, an audio and video file is generated according to the video data in the video cache pool and the audio data in the audio cache pool. According to the fixed audio and video duration requirement, the coverage storage period of the cache pool is set, so that the video data and audio data in the cache pool can be maintained at the set fixed duration. In this way, the frame cache technology with a coverage storage period is used to generate audio and video files with a fixed duration, which can ensure that the driving recorder can still generate audio and video files with a fixed duration in an environment where the actual frame rate is not fixed, which is not affected by system performance, thereby improving the user experience.

[0017] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0019] Figure 1 It is a schematic diagram of an implementation environment of an audio and video file generation system shown in an exemplary embodiment of the present application;

[0020] Figure 2 is a flow chart of a method for generating an audio and video file shown in an exemplary embodiment of the present application;

[0021] Figure 3 is a schematic diagram of storing video frames in a video buffer pool shown in an exemplary embodiment of the present application;

[0022] Figure 4is a schematic diagram of storing video frames in another video buffer pool according to an exemplary embodiment of the present application;

[0023] Figure 5 is a schematic diagram of video file information shown in an exemplary embodiment of the present application;

[0024] Figure 6 is a schematic diagram of a structure of video data shown in an exemplary embodiment of the present application;

[0025] Figure 7 is a schematic diagram of another structure of video data shown in an exemplary embodiment of the present application;

[0026] Figure 8 is a flowchart of a specific method for generating an audio and video file shown in an exemplary embodiment of the present application;

[0027] Fig. 9 is a block diagram of an audio and video file generation system shown in an exemplary embodiment of the present application;

[0028] Fig.10 It is a structural schematic diagram of a vehicle-mounted terminal provided in one embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will describe the implementation methods of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, not for limiting the scope of protection of the present application.

[0030] It should be noted that the illustrations provided in the following embodiments are only used to illustrate the basic concept of the present application in a schematic manner, and therefore the illustrations only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the form, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0031] In the following description, a large number of details are discussed to provide a more thorough explanation of the embodiments of the present application. However, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present application difficult to understand.

[0032] The dashcam first adds watermarks to the original data captured by the camera and the original data collected by the audio acquisition device, then encodes the video data and audio data respectively, and then synthesizes the audio and video to obtain the audio and video files to write into the digital media files. For example, in the car system of the Android operating system, after adding watermarks to the original data through OpenGL (OpenGraphics Library, open graphics library) technology, it is encoded through MediaCodec (multimedia codec), and then synthesized through MediaMuxer (audio and video encapsulation class), and the synthesized data is written into the mp4 file.

[0033] The inventor of the present application has found that in the process of generating audio and video files, the audio and video files are synthesized according to the number of frames. If you want to generate a file with a fixed duration, there may be a problem that the actual duration of the generated audio and video file is greater than the set duration. For example, in order to generate an audio and video file with a fixed duration of 30 seconds, if the frame rate is 25 frames per second, a total of 30*25=750 frames of data are required, and the audio and video file is synthesized according to 750 frames of data when it is generated. If the frame rate is 25 frames per second under ideal circumstances (the vehicle system runs stably), the frame rate of the video frame and the audio frame will be unstable due to the influence of the vehicle system's running stability. A fixed frame rate of 25 frames per second is an ideal situation. The actual frame rate obtained will fluctuate at a value less than 25 frames, for example, between 15 and 25 frames. That is, if the vehicle system is a little stuck, it may be around 20 frames per second. If it is stuck, it may be around 15 frames per second. Therefore, the number of frames per second will be reduced compared to the ideal situation. In this case, the audio and video files synthesized according to 750 frames of data will be longer than 30S, and the total duration of the audio and video files will not be fixed due to the uncertainty of the system's running stability.

[0034] If the length of the audio and video files generated by the dashcam is not fixed, there will be at least the following problems. For example, storage management becomes complicated. If the file size and length are inconsistent, it will become more difficult to manage the storage space. Retrieval and playback are inconvenient. When an accident or incident occurs, it is necessary to quickly find the audio and video files of the relevant time period. If the length is not fixed, it will become complicated to locate the clips at a specific time point. Data analysis becomes more difficult. For the automated analysis of dashcam data, the non-fixed length will make the automated data processing and analysis more complicated, and even require the algorithm to be adjusted for files of different lengths.

[0035] Therefore, see Figure 1 , Figure 1 FIG. 1 is a schematic diagram of an implementation environment of an audio and video file generation system shown in an exemplary embodiment of the present application. Figure 1As shown, the implementation environment includes a vehicle 110 and an audio and video file generation system 120, wherein the audio and video file generation system 120 is embedded in the vehicle 110, and is used to realize the generation of audio and video files of the driving recorder in the vehicle 110. The audio and video file generation system 120 includes but is not limited to a vehicle system, a vehicle-mounted computer, etc. Through a fixed audio and video duration requirement, an overlay storage period of the buffer pool is set, so that the video data and audio data in the buffer pool can be maintained at the set fixed duration. In this way, the frame buffer technology with a set overlay storage period is used to generate an audio and video file of a fixed duration, which can ensure that the driving recorder can still generate an audio and video file of a fixed duration in an environment where the actual frame rate is not fixed, and is not affected by system performance, thereby improving the user experience.

[0036] See also Figure 2 , Figure 2 is a flowchart of a method for generating an audio and video file as shown in an exemplary embodiment of the present application. The method can be applied to Figure 1 The implementation environment shown is a schematic diagram. It should be understood that the method may also be applicable to other exemplary implementation environments, and this embodiment does not limit the implementation environment to which the method is applicable.

[0037] In addition, the audio and video file generation method can be applied to vehicle systems under various operating systems, including but not limited to vehicle systems under the Android operating system.

[0038] like Figure 2 As shown, in an exemplary embodiment, the method for generating an audio and video file includes at least steps S210 to S230, which are described in detail as follows:

[0039] Step S210, obtaining driving record data, the driving record data includes video frames and audio frames, and the video frames and the audio frames are both configured with time stamps.

[0040] Among them, the video frames are obtained from the camera of the driving recorder, and the audio frames are obtained from the recording device corresponding to the driving recorder; the timestamps configured for the video frames and audio frames use the system relative time instead of obtaining the current system time, so as to avoid problems with the obtained time and the configured timestamps if the user actively modifies the system time.

[0041] In addition, when acquiring video frames and audio frames at the same time, ideally the timestamps configured for the video frames and audio frames are the same. However, since the video frames and audio frames come from different devices, even though the video frames and audio frames are acquired at the same time, there may be errors in the timestamps marked by the two.

[0042] Step S220, the video frames and audio frames are stored in the video cache pool and the audio cache pool respectively based on the timestamp and the first-in-first-out principle. The video cache pool and the audio cache pool are both configured with a coverage storage period, and the coverage storage period is set according to the preset audio and video duration.

[0043] In this embodiment, if the driving recorder is started, the data in the video buffer pool and the audio buffer pool are in a state of continuous updating, that is, the video frames and audio frames are continuously obtained, and the video frames and audio frames are stored in the video buffer pool and the audio buffer pool respectively according to the first-in-first-out principle based on the timestamp. Therefore, the duration of the data in the video buffer pool and the audio buffer pool is always the duration corresponding to the covering storage period, that is, the required audio and video duration is maintained. In this way, the purpose of generating audio and video files of fixed duration is achieved by setting the frame buffer technology of the covering storage period.

[0044] In a possible embodiment, after the video frame is obtained from the camera of the dash cam and the audio frame is obtained from the recording device corresponding to the dash cam, they will be cached in their respective corresponding total cache pools. In the total cache pool, the first-in-first-out overwriting storage principle is not involved, but the video frame or video frames captured after the dash cam is started are stored, and all frame data are saved. If the cleaning conditions of the data in the total cache pool are met, the data therein is cleaned up, and the cleaning conditions include but are not limited to the data memory reaching the preset value, the time reaching the cleaning time, and the user triggering the cleaning request.

[0045] As a possible embodiment, the video frame will be stored in its corresponding total buffer pool. In this case, the data in the total buffer pool will be cleared only when the clearing condition is met. At the same time, it will also be stored in the video buffer pool. In this case, it is stored in the video buffer pool based on the first-in-first-out principle based on the timestamp. The audio frame is the same and will not be described in detail. The data in the video buffer pool and the data in the audio buffer pool are used to generate an audio and video file of a fixed length in response to an audio and video file generation signal, and the data in the total buffer pool can also be taken out to generate an audio and video file according to specific needs. For example, data of other durations or certain time periods can be taken out to generate audio and video files.

[0046] In one embodiment, the setting method of the coverage storage period includes: determining the duration compensation factor according to the second-level rounding rule of the audio and video duration, the second-level rounding rule is rounding down, and the duration compensation factor is less than the second-level unit duration; determining the coverage storage period according to the audio and video duration and the duration compensation factor, the coverage storage period is the sum of the audio and video duration and the duration compensation factor, so that the actual duration of the generated audio and video file is the audio and video duration.

[0047] In this embodiment, considering that if the coverage storage period is set to the preset audio and video duration, due to system freezes, when the video frames and audio frames are stored in the video cache pool and the audio cache pool respectively based on the timestamp according to the first-in-first-out principle, the video data in the video cache pool and / or the audio data in the audio cache pool will be less than the preset audio and video duration. Therefore, a duration compensation factor is introduced. The duration compensation factor is a duration value. According to the audio and video duration and the duration value, the coverage storage period is determined. Under the rule of rounding down the audio and video duration to the second level, in this way, in the case of system instability, it is ensured that the actual duration of the generated audio and video file will not exceed the preset audio and video duration, nor be less than the preset audio and video duration, and the duration of each generated audio and video file is the preset audio and video duration.

[0048] Taking the generation of an audio and video file with a fixed duration of 30 seconds as an example, due to the system jamming problem, there will be a large difference in the timestamps between the two frames. At this time, after the previous frame that is greater than the coverage storage period is removed, the difference in the timestamps between the next frame and the last frame in the cache pool will be less than 30 seconds, that is, the duration of the video data in the video cache pool and the duration of the audio data in the audio cache pool will be less than 30 seconds. Therefore, according to the second-level rounding down rule for the audio and video duration, that is, if the duration of the data is in [30, 31), the duration of the generated audio and video files is 30 seconds, so the duration value of the duration compensation factor is greater than 0S and less than 1S (second-level unit duration). In this way, the coverage storage period is a period greater than 30S and less than 31S, which can effectively avoid the problem that the duration of video data and audio data is less than the preset audio and video duration due to system jamming.

[0049] In one embodiment, video frames and audio frames are stored in a video cache pool and an audio cache pool respectively based on timestamps according to a first-in-first-out principle, including: determining a first time difference between the video frame and the video start frame according to the timestamp of the video frame and a first historical timestamp of the video start frame in the video cache pool, and determining a second time difference between the audio frame and the audio start frame according to the timestamp of the audio frame and a second historical timestamp of the audio start frame in the audio cache pool; caching the video frame in the video cache pool, and moving the video start frame out of the video cache pool under the condition that the first time difference is greater than a covering storage period; caching the audio frame in the audio cache pool, and moving the audio start frame out of the audio cache pool under the condition that the second time difference is greater than a covering cache period.

[0050] In this embodiment, a new video frame is placed at the end of the video buffer pool and compared with the timestamp of the video start frame in the video buffer pool. If the time difference between the two is greater than the coverage storage period, the video start frame is removed from the video buffer pool. Similarly, a new audio frame is placed at the end of the audio buffer pool and compared with the timestamp of the audio start frame in the audio buffer pool. If the time difference between the two is greater than the coverage storage period, the audio start frame is removed from the video buffer pool. In this way, the new data frame is compared with the start frame, and the first-in-first-out of the data frame is implemented in the video buffer pool and the audio buffer pool based on the timestamp and the coverage storage period. Regardless of the actual frame rate of the system, the video data in the buffer pool and the audio data in the audio buffer pool can be kept at a fixed audio and video duration.

[0051] See also Figure 3 , Figure 3 FIG. 1 is a schematic diagram showing an exemplary embodiment of the present application, in which a video frame is stored in a video buffer pool. Figure 3 As shown, after a new video frame enters the video buffer pool, the time difference between the new video frame and the start frame is calculated, and the new video frame is stored at the end of the video buffer pool. If the calculated time difference is greater than the coverage storage period, the start frame is removed from the video buffer pool.

[0052] In one embodiment, video frames and audio frames are stored in a video cache pool and an audio cache pool respectively based on timestamps according to a first-in-first-out principle, including: calculating the third time difference between the timestamp of the video frame and the timestamp of each historical video frame in the video cache pool in sequence according to the order of timestamps, and calculating the fourth time difference between the timestamp of the audio frame and the timestamp of each historical audio frame in the audio cache pool in sequence; caching the video frame in the video cache pool, and if the third time difference of any historical video frame is greater than the overwriting storage period, any historical video frame is moved out of the video cache pool; caching the audio frame in the audio cache pool, and if the fourth time difference of any historical audio frame is greater than the overwriting storage period, any historical audio frame is moved out of the audio cache pool.

[0053] In this embodiment, taking into account the case where the system freezes severely, that is, the difference between the timestamp of the new data frame and the timestamp of the previous frame is large, at this time, the time difference between the timestamp of the new data frame and multiple data frames in the first segment of the cache pool will be greater than the coverage storage period. Therefore, the time difference of the new data frame and the data frame starting from the head end is calculated in turn, and the video frames with time differences greater than the coverage storage period are all moved out of the cache pool until the time difference is no greater than the coverage storage period. In this way, the new data frame is compared with each data frame from the head end in turn, and in the video cache pool and the audio cache pool, the first-in-first-out of the data frame is implemented based on the timestamp and the coverage storage period. Regardless of the actual frame rate of the system, the video data in the cache pool and the audio data in the audio cache pool can be kept at a fixed audio and video duration.

[0054] See also Figure 4 , Figure 4 FIG. 1 is a schematic diagram of another exemplary embodiment of the present application showing a video frame stored in a video buffer pool. Figure 4 As shown, after a new video frame enters the video buffer pool, the time difference between it and each video frame will be calculated in sequence starting from the head end, and the new video frame will be stored at the end of the video buffer pool, and the video frames with a time difference greater than the coverage storage period will be removed from the video buffer pool.

[0055] Step S230, in response to the received audio and video file generation signal, generate an audio and video file according to the video data in the video buffer pool and the audio data in the audio buffer pool.

[0056] Among them, the audio and video file generation signal is generated based on event triggering, and the events include but are not limited to sudden acceleration or deceleration (such as collision or sudden braking), user operation, and the preset generation mechanism meeting conditions (such as timed generation, periodic generation, etc.).

[0057] In this embodiment, if a signal for generating an audio or video file is received, the video data in the video cache pool and the audio data in the audio cache pool are read in response to the signal, and the audio or video file is synthesized. Based on the frame cache technology of the video cache pool and the audio cache pool, the generated audio or video file always maintains the preset audio or video duration.

[0058] In one embodiment, an audio and video file is generated based on video data in a video cache pool and audio data in an audio cache pool, including: obtaining timestamps of the first and last frames in the video data, recorded as a first timestamp and a second timestamp, respectively; removing audio frames in the audio data whose timestamps are earlier than the first timestamp, and / or removing audio frames in the audio data whose timestamps are later than the second timestamp, to obtain target audio data; and synchronously integrating the video data and the target audio data according to the timestamps of each video frame in the video data and the timestamps of each audio frame in the target audio data to obtain an audio and video file.

[0059] In this embodiment, considering that the video data and audio data are encoded and cached separately, the read video data and audio data may have the problem of mismatching the first and last frame timestamps, that is, the audio and video are not synchronized, so it is necessary to synchronously integrate the video data and audio data to ensure the quality of the audio and video files.

[0060] As a possible implementation example, the audio data is corrected based on the video data, that is, the audio frames whose timestamps in the audio data are earlier than the timestamps of the first frame in the video data are removed, and / or the audio frames whose timestamps in the audio data are later than the timestamps of the last frame in the video data are removed, and then the video data and the target audio data are synchronously integrated according to the timestamps of each video frame in the video data and the timestamps of each audio frame in the remaining target audio data, so as to obtain an audio and video file. In this way, the synthesized audio and video file will not have the problem of sound without picture or the sound and picture being out of sync. On the basis of ensuring that the audio and video file is maintained at a fixed length, the quality of the audio and video file is guaranteed, thereby improving the user experience.

[0061] In addition, in this embodiment, both the video data and the audio data maintain a fixed duration. Due to the error in data acquisition, the audio data and the video data cannot be aligned in time periods based on the video data, including: the timestamp of the first frame of the audio data is earlier than the timestamp of the first frame of the video data, the timestamp of the last frame of the audio data is later than the timestamp of the last frame of the video data, and the timestamp of the first frame of the audio data is both earlier than the timestamp of the first frame of the video data and the timestamp of the last frame of the audio data is also later than the timestamp of the last frame of the video data. Therefore, the audio data is corrected, including: removing the audio frames whose timestamps in the audio data are earlier than the timestamps of the first frame of the video data, removing the audio frames whose timestamps in the audio data are later than the timestamps of the last frame of the video data, and removing both the audio frames whose timestamps in the audio data are earlier than the timestamps of the first frame of the video data and the audio frames whose timestamps in the audio data are later than the timestamps of the last frame of the video data.

[0062] In actual audio and video file synthesis, the frame buffer technology can prevent the generated audio and video files from exceeding the fixed audio and video duration. However, after checking the detailed information of the generated audio and video files, the actual duration will be less than the audio and video duration. Figure 5 , Figure 5 FIG. 1 is a schematic diagram of video file information shown in an exemplary embodiment of the present application. Figure 5 As shown in the figure, taking the fixed audio and video duration of 30S as an example, if the total number of frames of the 30S video data is 570 frames before the audio and video files are synthesized, the actual number of frames may be only 550. When the actual number of frames is only 550, the duration of the synthesized audio and video file may be less than 30S.

[0063] The above situation occurs because before encoding the video frame, when creating the video frame encoder, the frame rate and key frame interval parameters need to be set. For example, the key frame interval is set to 1 second, that is, the video frame encoder will set a key frame every 1 second. When synthesizing audio and video files, if the video data is a non-key frame at the beginning, these non-key frame data need to be discarded until the key frame is found before starting to synthesize. Among them, the key frame is a completely self-contained video frame that does not rely on other frames for decoding. Each key frame contains all the information of the picture at that moment and can be decoded and displayed independently. The non-key frame uses the previous key frame or non-key frame as a reference and uses predictive coding to represent the difference between the current frame and the reference frame. For example, in the car system under the Android operating system, after investigating the audio and video encoding MediaCodec and the audio and video synthesis MediaMuxer, it was found that when synthesizing audio and video, if the first frame obtained is not a key frame, MediaMuxer will discard these data until the obtained frame is a key frame and start to synthesize and write data.

[0064] See also Figure 6 , Figure 6 FIG. 1 is a schematic diagram of a structure of video data shown in an exemplary embodiment of the present application. Figure 6 As shown, key frames are represented by I frames, and non-key frames are represented by P frames. If multiple frames in the header of the video data are all P frames, these P frames need to be discarded until an I frame is found and then synthesis begins.

[0065] Since the signal generated by audio and video files is random, it is impossible to ensure whether the first frame of the video data is a key frame. Therefore, if a series of non-key frames are discarded, the video data that originally had a fixed length will have an actual length after synthesis that is lower than the set fixed length.

[0066] In one embodiment, a video frame is configured with a type identifier, which is used to distinguish between key frames and non-key frames; before generating an audio and video file based on the video data in the video cache pool and the audio data in the audio cache pool, it also includes: if the first frame of the video data is a key frame, then entering the audio and video file generation operation; if the first frame of the video data is a non-key frame, then setting the first frame of the video data as a key frame, and entering the audio and video file generation operation.

[0067] As a possible implementation example, before generating an audio and video file, first determine whether the first frame of the video data is a key frame. If so, and there is no frame loss, directly enter the audio and video file generation operation. If not, avoid discarding non-key frames, set the first frame as a key frame, and then enter the audio and video file generation operation. In this way, there will be no frame loss when the audio and video files are synthesized, thus avoiding the problem that the actual duration of the audio and video files will be lower than the fixed duration.

[0068] See also Figure 7 , Figure 7 FIG. 1 is a schematic diagram of another structure of video data shown in an exemplary embodiment of the present application. Figure 7 As shown, in the video data, if the first frame of the video data is a P frame, the frame is set as an I frame.

[0069] In one embodiment, before the video frames and audio frames are stored in the video cache pool and the audio cache pool respectively based on the timestamp according to the first-in-first-out principle, it also includes: setting a frame type encoding strategy, the frame type encoding strategy is to set each video frame as a key frame when encoding the video frame.

[0070] As a possible implementation example, in order to avoid the problem that the initial non-key frames in the video data will be discarded when synthesizing audio and video files, each frame is set as a key frame when encoding the video frame, that is, the frame type encoding strategy of the video frame encoder is configured to set each video frame as a key frame when encoding the video frame. In this way, no frames will be lost when synthesizing the audio and video files, thereby avoiding the problem that the actual duration of the audio and video files will be lower than the fixed duration.

[0071] In a possible embodiment, the received audio and video file generation signal carries the event occurrence time and time requirements, that is, when the audio and video file is generated based on an event trigger, the event also sets the time requirements for the audio and video files. The time requirements can be understood as the required audio and video files are audio and video of a period of time before and after the event. At this time, based on the event occurrence time, continue to wait for the data in the cache pool to meet the time requirements, and then read the data from the video cache pool and the audio cache pool to synthesize the audio and video files.

[0072] As a possible implementation example, audio and video files with different requirements may be generated based on time requirements, thereby improving user experience.

[0073] For example, the fixed length of an audio or video file is 30 seconds. When a vehicle collides, 15 seconds of audio or video are required before and after the collision. After the received audio or video file generates a signal, the data in the video cache pool and the audio cache pool are updated for another 15 seconds, and then the data is read out to synthesize the audio or video file.

[0074] See also Figure 8 , Figure 8 FIG. 1 is a flowchart of a specific method for generating an audio and video file according to an exemplary embodiment of the present application. Figure 8 As shown, taking the car system under the Android operating system as an example, the specific audio and video file generation method is described in detail as follows: after the car system and the driving recorder are started, the driving recorder turns on the camera to obtain video frames, and turns on the recording to obtain audio frames; after obtaining the video frames and audio frames, watermarks are added through OpenGL; then the video frames are encoded through MediaCodec, including setting timestamps and type identifiers, and the audio frames are encoded through MediaCodec, including setting timestamps; then the encoded video frames are cached in VideoQueue (video cache pool), and the encoded audio frames are cached in Au In dioQueue (audio cache pool), video frames and audio frames are overwritten and stored in VideoQueue and AudioQueue respectively according to the overwriting storage cycle; if the audio and video file generation signal is detected, MediaMuxer obtains the video data in VideoQueue and the audio data in AudioQueue, and corrects the audio data based on the video data; then it determines whether the first frame of the video data is a key frame. If so, the audio and video file is generated directly according to the video data and audio data. If not, the first frame of the video data is first set as a key frame, and then the audio and video file is generated according to the video data and audio data.

[0075] In an exemplary embodiment, in the operation of modifying the audio data and the operation of determining whether the first frame of the video data is a key frame, the timing between the two is not limited.

[0076] In this way, by combining frame buffer technology, key frame setting strategy and synchronization integration strategy between video data and audio data, it is ensured that the driving recorder can record an audio and video file of fixed length in an environment where the actual frame rate is not fixed, and it is not affected by the performance of the system, thereby facilitating the storage management, retrieval and playback of audio and video files, and data analysis, etc., improving the user experience

[0077] The above-mentioned audio and video file generation method first obtains driving record data, including video frames and audio frames, both of which are configured with timestamps, and then stores the video frames and audio frames in the video cache pool and the audio cache pool respectively according to the first-in-first-out principle based on the timestamps, wherein the video cache pool and the audio cache pool are both configured with a coverage storage period, and the coverage storage period is set according to the preset audio and video duration. If an audio and video file generation signal is received, then in response to the signal, an audio and video file is generated according to the video data in the video cache pool and the audio data in the audio cache pool. According to the fixed audio and video duration requirement, the coverage storage period of the cache pool is set, so that the video data and audio data in the cache pool can be maintained at the set fixed duration. In this way, the frame cache technology with a coverage storage period is used to generate audio and video files of fixed duration, which can ensure that the driving recorder can still generate audio and video files of fixed duration in an environment where the actual frame rate is not fixed, which is not affected by system performance, thereby improving the user experience.

[0078] See also Fig. 9 , Fig. 9 is a block diagram of a command line based ECU access system shown in an exemplary embodiment of the present application. The system can be applied to Figure 1 The implementation environment shown is a schematic diagram. It should be understood that the system may also be applicable to other exemplary implementation environments, and this embodiment does not limit the implementation environment to which the system is applicable.

[0079] like Fig. 9 As shown, in an exemplary embodiment, the audio and video file generation system 900 includes at least an acquisition module 910, a cache module 920 and a generation module 930, which are described in detail as follows:

[0080] An acquisition module 910 is used to acquire driving record data, where the driving record data includes video frames and audio frames, and both the video frames and the audio frames are configured with timestamps;

[0081] A cache module 920 is used to store video frames and audio frames in a video cache pool and an audio cache pool respectively based on a timestamp and a first-in-first-out principle. Both the video cache pool and the audio cache pool are configured with a coverage storage period, and the coverage storage period is set according to a preset audio and video duration;

[0082] The generation module 930 is used to generate an audio and video file in response to a received audio and video file generation signal, and generate an audio and video file according to the video data in the video buffer pool and the audio data in the audio buffer pool.

[0083] It should be noted that the audio and video file generation system provided in the above embodiment and the audio and video file generation method provided in the above embodiment belong to the same concept, wherein the contents of the operations performed by each module have been described in detail in the method embodiment and will not be repeated here.

[0084] See also Fig.10 , Fig.10 It is a structural schematic diagram of a vehicle-mounted terminal provided in one embodiment of the present application. Fig.10 The structure diagram of the computer system of the vehicle terminal suitable for implementing the embodiment of the present application is shown. It should be noted that: Fig.10 The computer system 1000 of the vehicle-mounted terminal shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0085] like Fig.10 As shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage part 1008 to the random access memory (RAM) 1003, such as executing the method in the above embodiment. In the RAM 1003, various programs and data required for system operation are also stored. The CPU 1001, the ROM 1002 and the RAM 1003 are connected to each other through the bus 1004. The input / output (I / O) interface 1005 is also connected to the bus 1004.

[0086] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0087] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1009, and / or installed from a removable medium 1011. When the computer program is executed by a central processing unit (CPU) 1001, various functions defined in the system of the present application are executed.

[0088] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor of a computer, the computer executes the above-mentioned method for generating audio and video files. The computer-readable storage medium may be included in the vehicle-mounted terminal described in the above-mentioned embodiment, or may exist independently without being assembled into the vehicle-mounted terminal.

[0089] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal can take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0090] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0091] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not constitute limitations on the units themselves in some cases.

[0092] The above embodiments are merely illustrative of the principles and effects of the present application, and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A method for generating an audio and video file, characterized in that: The method comprises: Acquire driving record data, the driving record data including video frames and audio frames, and the video frames and the audio frames are both configured with timestamps; The video frame and the audio frame are stored in a video buffer pool and an audio buffer pool respectively according to a first-in-first-out principle based on the timestamp, and both the video buffer pool and the audio buffer pool are configured with a coverage storage period, and the coverage storage period is set according to a preset audio and video duration; In response to the received audio and video file generation signal, an audio and video file is generated according to the video data in the video buffer pool and the audio data in the audio buffer pool.

2. The method for generating an audio and video file according to claim 1, characterized in that: The method for setting the overlay storage period includes: Determine a duration compensation factor according to a second-level rounding rule for the duration of the audio and video, wherein the second-level rounding rule is rounding down, and the duration compensation factor is less than the second-level unit duration; The coverage storage period is determined according to the audio and video duration and the duration compensation factor, and the coverage storage period is the sum of the audio and video duration and the duration compensation factor, so that the actual duration of the generated audio and video file is the audio and video duration.

3. The method for generating an audio and video file according to claim 2, characterized in that: The storing the video frame and the audio frame into a video buffer pool and an audio buffer pool respectively according to a first-in-first-out principle based on the timestamp includes: Determine a first time difference between the video frame and the video start frame according to the timestamp of the video frame and a first historical timestamp of the video start frame in the video buffer pool, and determine a second time difference between the audio frame and the audio start frame according to the timestamp of the audio frame and a second historical timestamp of the audio start frame in the audio buffer pool; Cache the video frame into the video cache pool, and move the video start frame out of the video cache pool under the condition that the first time difference is greater than the coverage storage period; The audio frame is cached in the audio cache pool, and under the condition that the second time difference is greater than the overlay cache period, the audio start frame is moved out of the audio cache pool.

4. The method for generating an audio and video file according to claim 2, characterized in that: The storing the video frame and the audio frame into a video buffer pool and an audio buffer pool respectively according to a first-in-first-out principle based on the timestamp includes: According to the order of timestamps, sequentially calculate a third time difference between the timestamp of the video frame and the timestamp of each historical video frame in the video buffer pool, and sequentially calculate a fourth time difference between the timestamp of the audio frame and the timestamp of each historical audio frame in the audio buffer pool; Cache the video frame into the video cache pool, and if the third time difference of any of the historical video frames is greater than the overlay storage period, remove the any historical video frame from the video cache pool; The audio frame is cached in the audio cache pool, and if the fourth time difference of any of the historical audio frames is greater than the overlay storage period, the any historical audio frame is removed from the audio cache pool.

5. The method for generating an audio and video file according to claim 1, characterized in that: The step of generating an audio and video file according to the video data in the video buffer pool and the audio data in the audio buffer pool includes: Obtaining timestamps of the first and last frames in the video data, which are recorded as a first timestamp and a second timestamp respectively; removing audio frames whose timestamps are earlier than the first timestamp in the audio data, and / or removing audio frames whose timestamps are later than the second timestamp in the audio data, to obtain target audio data; The video data and the target audio data are synchronously integrated according to the timestamp of each video frame in the video data and the timestamp of each audio frame in the target audio data to obtain the audio and video file.

6. The method for generating an audio and video file according to any one of claims 1 to 5, characterized in that: The video frame is configured with a type identifier, where the type identifier is used to distinguish between a key frame and a non-key frame; Before generating the audio and video file according to the video data in the video buffer pool and the audio data in the audio buffer pool, the method further includes: If the first frame of the video data is the key frame, the generation operation of the audio and video file is started; If the first frame of the video data is the non-key frame, the first frame of the video data is set as the key frame, and the generation operation of the audio and video file is started.

7. The method for generating an audio and video file according to any one of claims 1 to 5, characterized in that: Before storing the video frame and the audio frame in the video buffer pool and the audio buffer pool respectively according to the first-in-first-out principle based on the timestamp, the method further includes: A frame type encoding strategy is set, wherein each video frame is set as a key frame during video frame encoding.

8. A system for generating audio and video files, characterized in that: The system comprises: An acquisition module, used for acquiring driving record data, wherein the driving record data includes video frames and audio frames, and the video frames and the audio frames are both configured with time stamps; A cache module, used for storing the video frame and the audio frame in a video cache pool and an audio cache pool respectively according to a first-in-first-out principle based on the timestamp, wherein both the video cache pool and the audio cache pool are configured with an overlay storage period, and the overlay storage period is set according to a preset audio and video duration; The generating module is used to generate a signal in response to the received audio and video file, and to generate the audio and video file according to the video data in the video buffer pool and the audio data in the audio buffer pool.

9. A vehicle-mounted terminal, characterized in that: include: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, enables the vehicle terminal to implement the audio and video file generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer is caused to execute the method for generating an audio and video file as described in any one of claims 1 to 7.

Images