Precise on-demand method, device and medium for massive video data of autonomous driving

Abstract

The embodiment of the present invention discloses a precise on-demand method, device and medium for massive video data of autonomous driving, and relates to the field of video data processing technology. The method includes: in response to an on-demand operation on a certain logical frame number of an on-board video, calling a proxy file of the on-board video, wherein the on-board video is composed of multiple video segments, and the proxy file stores the frame number and arrangement order of each video segment; according to the frame number and arrangement order of each video segment, determining the target video segment where the certain logical frame number is located, and the target physical frame number of the certain logical frame number in the target video segment; according to the physical frame number of each I frame in the target video segment, determining the I frame in the GOP where the target physical frame number is located; according to the IDR image in the I frame, synthesizing the video image corresponding to the target physical frame number and playing it. This embodiment can realize the rapid and accurate positioning of any frame of the on-board video.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of video data processing technology, and in particular to a method, device and medium for accurate on-demand viewing of massive video data for autonomous driving. Background Art

[0002] The autonomous driving data recording system has a massive amount of video data and files. In the prior art, when playing the autonomous driving video data, multiple raw stream video files are first combined into a large raw stream file, and then the video file is played or reprocessed.

[0003] This method requires decoding and encoding during the file synthesis process, which consumes a lot of time; different encodings will cause video data loss or distortion, resulting in serious distortion of video data for subsequent playback or reprocessing. In addition, most raw stream video data does not have timestamp information, and when on-demand, it can usually only start playing from a certain video segment or an I frame, and it is impossible to directly and accurately locate the picture of any frame. Summary of the invention

[0004] The embodiments of the present invention provide a method, device and medium for accurate on-demand viewing of massive video data for autonomous driving to solve the above-mentioned technical problems.

[0005] In a first aspect, an embodiment of the present invention provides a method for accurately watching massive video data on demand in an autonomous driving system, comprising:

[0006] In response to a video-on-demand operation on a certain logical frame number of an in-vehicle video, a proxy file of the in-vehicle video is called, wherein the in-vehicle video is composed of a plurality of video segments, and the proxy file stores the frame number and arrangement order of each video segment;

[0007] Determine, according to the number of frames and arrangement order of each video segment, a target video segment where the certain logical frame number is located, and a target physical frame number of the certain logical frame number in the target video segment;

[0008] Determine, according to the physical frame number of each I frame in the target video segment, the I frame in the GOP where the target physical frame number is located;

[0009] According to the IDR image in the I frame, the video image corresponding to the target physical frame number is synthesized and played.

[0010] In a second aspect, an embodiment of the present invention provides an electronic device, the electronic device comprising:

[0011] one or more processors;

[0012] a memory for storing one or more programs,

[0013] When the one or more programs are executed by the one or more processors, the one or more processors implement the precise on-demand method for massive video data for autonomous driving as described in any embodiment.

[0014] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the precise on-demand method for massive video data for autonomous driving as described in any embodiment.

[0015] In summary, the embodiments of the present invention provide a precise on-demand method for massive video data in autonomous driving, which records the file name and frame number of each video segment in the vehicle-mounted video through a proxy file, and realizes the precise positioning of any frame through the conversion of logical frame number and physical frame number, so that the user can start playback from any frame, thereby improving the granularity of video on demand. There is no need to synthesize multiple video segments into a large video file, which saves synthesis steps and time, and avoids data loss and distortion in the intermediate synthesis, realizing precise and fast video on demand. It is particularly suitable for vehicle-mounted environments and autonomous driving environments with fast moving speeds, large amounts of video data, and high real-time requirements. Especially in accident determination scenarios, it can provide frame-level picture positioning, greatly improving the supporting role of video data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 is a flow chart of a method for accurately watching massive video data on demand for autonomous driving provided by an embodiment of the present invention;

[0018] Figure 2 is a schematic diagram of an in-vehicle video playback interface provided by an embodiment of the present invention;

[0019] Figure 3 is a schematic diagram of an in-vehicle video and a video segment provided by an embodiment of the present invention;

[0020] Figure 4 It is a schematic diagram of a frame sequence of a raw stream video file provided by an embodiment of the present invention;

[0021] Figure 5 is a schematic diagram of an I frame search provided by an embodiment of the present invention;

[0022] Figure 6It is a schematic diagram of searching a video image corresponding to a target physical frame number according to an I frame provided by an embodiment of the present invention;

[0023] Figure 7 is a schematic diagram of a primary video stream and a secondary video stream provided by an embodiment of the present invention;

[0024] Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0026] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0027] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0028] An embodiment of the present invention provides a precise on-demand method for massive video data of autonomous driving, which is suitable for on-demand playback of raw streaming video data stored in an autonomous driving data recording system. With the help of a proxy file of the on-board video, this method can achieve precise positioning of any frame of the video, and is particularly suitable for scenarios with high requirements for video playback granularity, such as accident responsibility determination, wherein the accident responsibility determination here includes both the definition of responsibility between vehicles and the definition of responsibility between the driver and the autonomous driving system in the same autonomous driving vehicle. In order to illustrate this method, the proxy file of the on-board video that supports the implementation of this method is first introduced.

[0029] In this embodiment, the environment image of the autonomous driving vehicle is captured by the on-board camera. In order to ensure a certain fault tolerance mechanism, the captured video is segmented and stored in multiple video files. Each video segment has a fixed duration or a fixed number of frames. The duration and the number of frames can be converted to each other through the frame rate. Each video segment is stored separately in a video file. In this embodiment, these files are referred to as physical files.

[0030] When the user needs to replay the in-vehicle video, the playback interface usually displays a longer period of time, such as 8:00-12:00 on a certain day. However, the duration of each video segment shot by the camera is relatively short, such as a fixed duration of 10 minutes, so the in-vehicle video from 8:00 to 12:00 displayed on the user interface will include a large number of video segments. In order to avoid splicing a large number of video segments, this embodiment constructs a separate proxy file for the overall in-vehicle video displayed on the user interface. The file stores the file names and frame numbers (or durations) of the multiple video segments included in the overall in-vehicle video. The information of each video segment is arranged in sequence in the proxy file according to the playback order (that is, the order of recording) in the overall in-vehicle video. For the sake of ease of description, the overall in-vehicle video of a longer period of time displayed on the playback interface mentioned above is referred to as the in-vehicle video below.

[0031] In a specific implementation, a new file type may be defined for the proxy file. The main contents stored in the proxy file include:

[0032] #COOTYPE ---The start mark of the file, corresponding to the file type of the proxy file

[0033] #COO-TARGETDURATION: <s>---Indicates the maximum duration or number of frames of each video segment. The parameter s indicates the maximum duration or number of frames.

[0034] #COO-SEQUENCE: <number>----The serial number of the video segment. The parameter number is the serial number.

[0035] [video-segment-1.h264] --- The file name of the first video segment

[0036] #COOINF:10.000, --- Duration or number of frames of the first video segment (10 here means 10s, corresponding to duration)

[0037] [video-segment-2.h264] ---The file name of the second video segment

[0038] #COOINF:10.000, ---Duration or number of frames of the second video segment ...

[0039] [video-segment-n.h264] ---The file name of the last video segment

[0040] #COOINF:5.000, ---The duration or number of frames of the last video segment can be less than the maximum duration or number of frames. In this example, the duration of the last video segment is 5s, which corresponds to the case where the vehicle video cannot be an integer multiple of 10s.

[0041] #COO-ENDLIST --- The end of the file

[0042] Among them, COO-SEQUENCE: <number>Indicates the serial number of the first video segment in the playlist. Each video segment has a unique integer serial number. The serial number of each video segment increases by 1 in the order of appearance. The serial number is irrelevant to the file name.

[0043] Based on the above proxy files, Figure 1 is a flow chart of a method for accurately watching massive video data on demand in an autonomous driving system provided by an embodiment of the present invention. The method is executed by an electronic device, such as Figure 1 As shown, the specific steps include:

[0044] S110 . In response to a video-on-demand operation on a certain logical frame number of the in-vehicle video, a proxy file of the in-vehicle video is called.

[0045] In this embodiment, the frame number of the video frame is divided into a logical frame number and a physical frame number. The logical frame number refers to the position of a frame in the vehicle video (i.e., the arrangement order). From the above introduction to the proxy file, it can be seen that the vehicle video includes a large number of video segments, but these video segments are not actually spliced ​​into a large physical file. Therefore, the vehicle video is not an actual large physical file. The logical frame number is a logical position. The physical frame number refers to the position of a frame in the video segment. Since the physical file of the video segment actually exists, this position is a physical position.

[0046] Combination Figure 2 In the playback interface of the in-car video, the user can drag or click any position in the progress bar to play the video on demand, hoping to start playing from the video frame corresponding to that position. At this time, it is necessary to call the above proxy file of the in-car video, and then locate the physical position of the on-demand frame through the physical file name and frame number stored in the proxy file. For example, Figure 2 As shown, the playback interface can display the total number of frames of the vehicle video (750 frames), and the user can use the progress bar to request a certain frame (the 207th frame), and the frame number is the logical frame number.

[0047] Of course, the playback interface can also display the total duration of the in-vehicle video, and the user's click position can also be displayed as a certain moment. At this time, the requested moment can be converted into a logical frame number in the in-vehicle video according to the starting moment of the in-vehicle video and the video frame rate. Optionally, the duration of a unit video frame can be calculated according to the video frame rate; then the starting moment of the in-vehicle video is subtracted from the certain moment, and the difference is divided by the duration and rounded down to obtain the logical frame number of the certain moment in the in-vehicle video.

[0048] S120: Determine, according to the number of frames and arrangement order of each video segment, the target video segment where the certain logical frame number is located, and the target physical frame number of the certain logical frame number in the target video segment.

[0049] This step finds the physical frame number of the on-demand video frame (referred to as on-demand frame) in the actual physical file according to the logical frame number of the on-demand video frame in the vehicle video, so as to achieve accurate positioning of the frame.

[0050] In a specific implementation, firstly, according to the arrangement order of each video segment in the proxy file, the frame numbers of each video segment are sequentially accumulated to obtain the starting logical frame number and the ending logical frame number of each video segment. Figure 3 An example of a vehicle video is shown, which includes 8 video segments, each of which corresponds to a physical file. The format of each video file is unified, and the basic parameter information is shown in Table 1:

[0051] Table 1

[0052]

[0053] After the number of frames of each video segment is accumulated in sequence, the starting logical frame number, ending logical frame number, starting physical frame number and ending physical frame number of each video segment are shown in the following Table 2:

[0054] Table 2

[0055]

[0056] Based on the above correspondence, if the requested logical frame number is between the starting logical frame number and the ending logical frame number of a certain video segment, the certain video segment is determined as the video segment where the logical frame number is located. For example, when the user requests to play from the position where the logical frame number is 854, according to Table 2, the logical frame number is located in the logical frame number interval [600,900) of the video segment 3.h264, and 3.h264 is the video segment where the logical frame number 854 is located, which is called the target video segment in this embodiment.

[0057] Then, the starting logical frame number of the target video segment is subtracted from the requested logical frame number to obtain the physical frame number of the logical frame number in the target video segment. In this embodiment, the physical frame number is referred to as the target physical frame number. As in the above example, the requested logical frame number - the starting logical frame number of 3.h264 = 854 - 600 = 254 (frames), i.e., the 254th frame in 3.h264, which is the target physical frame number of the requested frame in 3.h264.

[0058] S130. Determine, according to the physical frame number of each I frame in the target video segment, the I frame in the GOP where the target physical frame number is located.

[0059] Specifically, the images in the raw stream video file are organized in units of GOP (Group of Picture). One GOP is a data stream of image encoding. The first frame in a GOP is called an I frame. The image stored in the I frame is an IDR (Instantaneous Decoder Refresh) image, which includes the complete image content. There are multiple P frames stored after the I frame. The P frame image does not include the complete image content, but only includes the difference information between itself and the I frame image in the same GOP. Therefore, the decoding of the P frame image requires the help of the I frame image. Figure 4 The frame sequence structure of a raw stream video file is shown as an example.

[0060] Based on the above structure, this embodiment pre-parses the physical files of each video segment, parses the physical frame number and position offset of each I frame in the file, and records them in the I frame list. Figure 4 For example, the corresponding I frame list is shown in Table 3:

[0061] Table 3

[0062]

[0063] After obtaining the target physical frame number through S120, this step can call the I frame list of the target video segment and locate the image frame corresponding to the target physical frame number according to the list. Optionally, if the target physical frame number is between the frame numbers of two I frames, the previous I frame of the two I frames is used as the I frame in the GOP where the target physical frame number is located. That is, the specific position of the previous I frame closest to the on-demand frame is found through the list, such as Figure 5 shown.

[0064] S140: Synthesize the video image corresponding to the target physical frame number according to the IDR image in the I frame and play it.

[0065] According to the position offset of the I frame, the IDR image in the I frame can be read. The frame number difference N between the target physical frame number and the I frame can be obtained by subtracting the physical frame number of the I frame from the target physical frame number.

[0066] If N=0, it indicates that the I frame is the video frame corresponding to the target physical frame number being requested, and the IDR image in the frame is played.

[0067] If N>0, then read the Nth P frame after the I frame, which is the video frame corresponding to the target physical frame number on demand, extract the image data in the frame, and synthesize the video image corresponding to the target physical frame number by the image data and the IDR image, such as Figure 6 shown.

[0068] Furthermore, considering that the on-board camera may be turned off or disconnected, the present embodiment can also install dual cameras on the autonomous driving vehicle to jointly shoot the surrounding environment. One of the two cameras is the main camera, and the other is the auxiliary camera. The two cameras shoot at the same time and generate video segments respectively. During playback, the video segment of the main camera (called the main video segment) is mainly played. Only when an unexpected situation occurs, such as power failure of the main camera, the video segment shot by the auxiliary camera (called the auxiliary video segment) will be played; once the main camera is restored to use, the switching point will be found in the fastest time to switch the playback content to the main video segment. At this time, the proxy file used to play the video needs to be automatically modified according to the online status of the main camera. In a specific implementation, the creation and modification of the proxy file may include the following steps:

[0069] Step 1, use the main camera and the auxiliary camera to shoot the driving environment respectively. After the main camera shoots M frames, the M frames are stored as a main video segment, and the file name and frame number of each main video segment are stored in the proxy file in order. At the same time, after the auxiliary camera shoots N frames, the N frames are stored as an auxiliary video segment. Among them, M>N, and m×M=n×N, where M, N, m, and n are all natural numbers. In other words, the fixed frame number of the main video segment is M, and the fixed frame number of the auxiliary video segment is N, and there is an integer relationship between the two with a certain period. Exemplarily, M=30, N=20, m=2, and n=3. It should be noted that the fixed frame number here refers to the video segments shot by the same camera, except for the last video segment, the frame number of the remaining video segments is fixed, and the frame number of the video segments between the two cameras can be different.

[0070] Combination Figure 7 For a long period of in-vehicle video, if the main camera and the auxiliary camera are working properly, the following images will be captured: Figure 7 As shown in FIG. 1 , multiple main video segments A1, A2 ..., A5 ..., and multiple auxiliary video segments B1, B2 ..., B7 ... are arranged in time sequence to form the following: Figure 7 The main video stream and auxiliary video stream are shown. By default, the proxy file of the vehicle video will record the file names and frame numbers of A1, A2..., A5... in sequence. When playing the entire vehicle video according to the proxy file, the video frames will be located from the main video segments A1, A2..., A5... and played.

[0071] Step 2: In response to the main camera being turned off or disconnected, extract the auxiliary video segment currently being shot, and find the situation where the start time of the auxiliary video segment currently being shot and one of the n-1 auxiliary video segments before it overlaps with the start time of the main video segment currently being shot and one of the m-1 main video segments before it. After finding, if the file name and frame number of the overlapping main video segment are already stored in the proxy file, then the file name and frame number of the main video segment and the main video segment after it are deleted from the proxy file, and the file name and frame number of the overlapping auxiliary video segment and the auxiliary video segment after it are stored in the proxy file instead. If the file name and frame number of the overlapping main video segment are not stored in the proxy file, then the file name and frame number of the overlapping auxiliary video segment and the auxiliary video segment after it are directly stored in the proxy file.

[0072] In this step, when the main camera is turned off or offline, a suitable switching point is found to switch the content in the proxy file to the video segment shot by the auxiliary camera. Figure 7 , if the main camera is turned off or disconnected when shooting the S1 frame (logical frame number), the auxiliary video segment currently being shot is B1, and the main video segment currently being shot is A1. At this time, the auxiliary video segment B1 currently being shot and the 2 (=3-1) auxiliary video segments before it constitute an auxiliary video set. Since there are no other auxiliary video segments before B1, there is only one auxiliary video segment B1 in the set; at the same time, the main video segment A1 currently being shot and the 1 (=2-1) main video segment before it constitute a main video set. Since there are no other main video segments before A1, there is only one main video segment A1 in the set. Then, find whether there is a main video segment in the main video set that coincides with the start time of a certain auxiliary video segment in the auxiliary video set. After finding that the start time of A1 coincides with that of B1, continue to check whether the file name and frame number of the overlapping main video segment A1 have been stored in the proxy file. Upon checking, if the file name and frame number of the main video segment A1 have not been stored in the proxy file, the file name and frame number of the overlapping auxiliary video segment B1 and the auxiliary video segments B2, B3, ... thereafter are stored in the proxy file.

[0073] Exemplary, combined Figure 7 , after the main camera shoots A1, the file name and frame number of A1 are stored in the proxy file. When the main camera continues to shoot to the S2th frame (logical frame number), the main camera is turned off or offline, then the auxiliary video segment currently being shot is B2, and the main video segment currently being shot is A2. At this time, the auxiliary video segment B2 currently being shot and the 2 (=3-1) auxiliary video segments before it constitute an auxiliary video set. Since there is only one auxiliary video segment B1 before B2, the auxiliary video segments in this set include B1 and B2. At the same time, the main video segment A2 currently being shot and the 1 (=2-1) main video segment before it constitute a main video set, and the main video segments in this set include A1 and A2. Then, find whether there is a main video segment in the main video set that coincides with the start time of a certain auxiliary video segment in the auxiliary video set. After finding that the start time of A1 coincides with that of B1, continue to check whether the file name and frame number of the overlapping main video segment A1 have been stored in the proxy file. After checking, the file name and frame number of the main video segment A1 have been stored in the proxy file, so the file name and frame number of A1 and the main video segments after it are deleted from the proxy file. In this example, the last video segment in the proxy file is A1, so the file name and frame number of A1 can be deleted from the proxy file, and then the file name and frame number of the overlapping auxiliary video segment B1 and the auxiliary video segments B2, B3... after it are stored in the proxy file.

[0074] Exemplary, combined Figure 7 , after the main camera shoots A1, the file name and frame number of A1 are stored in the proxy file. When the main camera continues to shoot to the S3 frame (logical frame number), the main camera is turned off or offline, then the auxiliary video segment currently being shot is B3, and the main video segment currently being shot is A2. At this time, the auxiliary video segment B3 currently being shot and the 2 (=3-1) auxiliary video segments B1 and B2 before it constitute an auxiliary video set, that is, the auxiliary video segments in the set are B1, B2, and B3. At the same time, the main video segment A2 currently being shot and the 1 (=2-1) main video segment A1 before it constitute a main video set, that is, the main video segments in the set are A1 and A2. Then, find whether there is a main video segment in the main video set that coincides with the start time of a certain auxiliary video segment in the auxiliary video set. After finding that the start time of A1 coincides with that of B1, continue to check whether the file name and frame number of the overlapping main video segment A1 have been stored in the proxy file. After checking, the file name and frame number of the main video segment A1 have been stored in the proxy file, so the file name and frame number of A1 and the main video segments after it are deleted from the proxy file. In this example, the last video segment in the proxy file is A1, so the file name and frame number of A1 can be deleted from the proxy file, and then the file name and frame number of the overlapping auxiliary video segment B1 and the auxiliary video segments B2, B3... after it are stored in the proxy file.

[0075] Exemplary, combined Figure 7 , after the main camera shoots A1 and A2, the file names and frame numbers of A1 and A2 are stored in the proxy file. When the main camera continues to shoot to the S4 frame (logical frame number), the main camera is turned off or offline, and the auxiliary video segment currently being shot is B4, and the main video segment currently being shot is A3. At this time, the auxiliary video segment B4 currently being shot and the 2 (=3-1) auxiliary video segments B2 and B3 before it constitute an auxiliary video set, that is, the auxiliary video segments in the set are B2, B3, and B4; the main video segment A3 currently being shot and the 1 (=2-1) main video segment A2 before it constitute a main video set, that is, the main video segments in the set are A2 and A3. Then, find whether there is a main video segment in the main video set that coincides with the start time of a certain auxiliary video segment in the auxiliary video set. After finding that the start time of A3 coincides with that of B4, continue to check whether the file name and frame number of the overlapping main video segment A3 have been stored in the proxy file. After checking, the file name and frame number of the main video segment A3 have not been stored in the proxy file, so the file names and frame numbers of the overlapping auxiliary video segment B4 and the subsequent auxiliary video segments B5, B6, ... are stored in the proxy file.

[0076] Step 4: In response to the restart of the main camera, when the total number of frames of the vehicle-mounted video meets an integer multiple of M, restart the generation of the main video segment; if the end time of a certain auxiliary video segment after the restart can be connected with the start time of a certain main video segment after the restart, after the certain auxiliary video segment, the file name and frame number of the main video segment and the main video segments after it are stored in the proxy file instead.

[0077] After the main camera is restarted, this step finds a suitable switching point and switches the content in the proxy file back to the video segment shot by the main camera. Figure 7 , when the main camera is turned off or offline when shooting the S4th frame (logical frame number), according to the analysis of the previous embodiment, the file name and frame number of B4 will be stored in the proxy file. If the main camera is turned on again or goes online when the auxiliary camera shoots the S5th frame (logical frame number), the main camera and the auxiliary camera will shoot subsequent videos at the same time, and when the total number of frames of the vehicle-mounted video meets the integer multiple S6 of M, the main video segments A4, A5... will be generated again. While generating, check whether there is an auxiliary video segment in the restarted auxiliary video segments B6, B7... that can be connected with the start time of a main video segment in the restarted main video segments A4, A5... before and after. Once a pair of video segments that can be connected is found, such as the end time of the auxiliary video segment B6 can be connected with the start time of the main video segment A5, then after the auxiliary video segment B6, the file name and frame number of the auxiliary video segment will no longer be stored in the proxy file, but the file name and frame number of the main video segment A5 and thereafter will be stored instead.

[0078] It can be seen that under the dual-camera shooting mechanism, when the main camera is turned off or disconnected, the video segment corresponding to the vehicle video can be switched in time by dynamically adjusting the video segment information recorded in the file when generating the proxy file, avoiding the loss of the video images during the main period. At the same time, through the fixed frame number and multiple relationship m×M=n×N between the main video segment and the auxiliary video segment, the main video segment and the auxiliary video segment can find the correct switching position and achieve seamless switching.

[0079] It should be noted that in the above method, when the main camera is turned on or online, the information stored in the proxy file is still mainly based on the main video segment, and the storage space resources for video storage also preferentially meet the needs of the main video segment. However, whether it is the main video segment or the secondary video segment, as long as their information is stored in the proxy file, the priority of these video segments will be increased, and these video segments will be marked in this embodiment. Correspondingly, when the in-vehicle storage space of the video segment reaches the upper limit, the video segments can be deleted in the following order: First, delete the earliest unmarked secondary video segment in chronological order; then, delete the earliest unmarked main video segment in chronological order; finally, delete the earliest marked video segment in chronological order. After the marked video segment is deleted, its file name and frame number in the proxy file will also be deleted, and all the storage space released by the deleted video segments will be used to store new video segments, and the new video segments will be added to the proxy file in the same way. In this way, the in-vehicle video is updated and covered in a cycle.

[0080] It is worth mentioning that the quantities of M and N can both be set through the in-vehicle system. In this embodiment, since the secondary video segment will be deleted first, setting the number of frames N of the secondary video segment to be less than M can refine the granularity of video space allocation and make the space allocation more flexible; at the same time, if individual frames are lost due to video segmentation in some in-vehicle systems, the difference between M and N can make the main video stream and the secondary video stream form a certain complementary effect, that is, the frames lost during the segmentation of the main video segment exist in the secondary video segment, and positioning can be continued through the secondary video segment. In addition, a relatively small N also has lower requirements for the cache, memory, etc. of the secondary camera, so a camera with better configuration can be used as the main camera, which is more in line with practical applications.

[0081] In summary, the embodiment of the present invention provides a method for accurate on-demand playback of a large amount of video data for autonomous driving. By recording the file name and frame number of each video segment behind the in-vehicle video through a proxy file, and realizing the accurate positioning of any frame through the conversion between the logical frame number and the physical frame number, the user can start playing from any frame, improving the granularity of video on-demand playback. The whole process does not require synthesizing multiple video segments into a large video file, saving the synthesis steps and time, and also avoiding data loss and distortion during the intermediate synthesis, realizing accurate and fast video on-demand playback, which is particularly suitable for in-vehicle environments and autonomous driving environments with high moving speed, large amount of video data, and high real-time requirements. Especially in the accident liability determination scenario, it can provide frame-level picture positioning, greatly improving the supporting role of video data.

[0082] In addition, considering unexpected situations such as camera shutdown or disconnection, this embodiment proposes a dual-camera shooting mechanism, and through the specific frame number relationship between the main video segment and the auxiliary video segment, the content in the proxy file can be seamlessly switched between the main camera and the auxiliary camera to avoid the loss of video data during important periods. At the same time, the coverage priority of various video segments is provided to ensure the update coverage of the vehicle video.

[0083] Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention is shown in FIG. Figure 8 As shown, the device includes a processor 60, a memory 61, an input device 62 and an output device 63; the number of processors 60 in the device can be one or more. Figure 8 A processor 60 is taken as an example; the processor 60, the memory 61, the input device 62 and the output device 63 in the device can be connected by a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0084] The memory 61 is a computer-readable storage medium that can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the precise on-demand method for massive video data of autonomous driving in the embodiment of the present invention. The processor 60 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 61, that is, realizing the precise on-demand method for massive video data of autonomous driving.

[0085] The memory 61 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 61 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 61 may further include a memory remotely arranged relative to the processor 60, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0086] The input device 62 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 63 may include a display device such as a display screen.

[0087] An embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the precise on-demand method for massive video data for autonomous driving of any embodiment.

[0088] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0089] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0090] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0091] Computer program code for performing the operation of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.< / number> < / number> < / s>

Claims

1. A precise on-demand method for massive video data of autonomous driving, characterized in that: include: In response to a video-on-demand operation on a certain logical frame number of an in-vehicle video, a proxy file of the in-vehicle video is called, wherein the logical frame number refers to the position of a certain frame in the in-vehicle video, the in-vehicle video is composed of a plurality of video segments, the plurality of video segments are not actually spliced ​​into a large physical file, and the proxy file stores the frame number and arrangement order of each video segment; Determine, according to the number of frames and arrangement order of each video segment, a target video segment where the certain logical frame number is located, and a target physical frame number of the certain logical frame number in the target video segment; wherein the physical frame number refers to the position of a certain frame in the video segment; Determine, according to the physical frame number of each I frame in the target video segment, the I frame in the GOP where the target physical frame number is located; According to the IDR image in the I frame, synthesize the video image corresponding to the target physical frame number and play it; Wherein, before the proxy file of the vehicle-mounted video is called in response to the on-demand operation of a certain logical frame number of the vehicle-mounted video, the method further includes: Use the main camera and auxiliary camera on the vehicle to simultaneously shoot the vehicle environment; After the main camera takes M frames, the M frames are stored as a main video segment, and the file name and frame number of each main video segment are stored in sequence in the proxy file of the vehicle-mounted video; at the same time, after the auxiliary camera takes N frames, the N frames are stored as an auxiliary video segment, where M>N, and , where M, N, m, and n are all natural numbers; In response to the main camera being turned off or offline, extracting the auxiliary video segment currently being shot, and finding the situation where the start time of the auxiliary video segment currently being shot and one of the n-1 auxiliary video segments before it overlap with the start time of the main video segment currently being shot and one of the m-1 main video segments before it; deleting the file name and frame number of the overlapping main video segment and the main video segment after it from the proxy file, and storing the file name and frame number of the overlapping auxiliary video segment and the auxiliary video segment after it in the proxy file instead; In response to the restart of the main camera, when the total number of frames of the in-vehicle video satisfies an integer multiple of M, the main video segment is restarted to be generated; and the end time of a certain auxiliary video segment after the restart is found to be connected with the start time of a certain main video segment after the restart; after the certain auxiliary video segment, the file name and frame number of the certain main video segment and the main video segments after it are stored in the proxy file instead.

2. The method according to claim 1, characterized in that: The step of calling the proxy file of the vehicle-mounted video in response to the on-demand operation of a certain logical frame number of the vehicle-mounted video includes: In response to a video-on-demand operation for a certain moment of the in-vehicle video, the certain moment is converted into a logical frame number in the in-vehicle video according to a start time and a video frame rate of the in-vehicle video.

3. The method according to claim 2, characterized in that The step of converting the certain moment into a logical frame number in the vehicle-mounted video according to the starting moment and the video frame rate of the vehicle-mounted video includes: According to the video frame rate, calculate the duration of a unit video frame; The starting time of the vehicle-mounted video is subtracted from the certain time, and the difference is divided by the duration and rounded down to obtain a logical frame number of the certain time in the vehicle-mounted video.

4. The method according to claim 1, characterized in that The step of determining, according to the number of frames and arrangement order of each video segment, a target video segment where the certain logical frame number is located, and a target physical frame number of the certain logical frame number in the target video segment comprises: According to the arrangement order of each video segment, the frame numbers of each video segment are accumulated in sequence to obtain the starting logical frame number and the ending logical frame number of each video segment; If the certain logical frame number is between the starting logical frame number and the ending logical frame number of a certain video segment, determining that the certain video segment is the target video segment where the certain logical frame number is located; The starting logical frame number of the target video segment is subtracted from the certain logical frame number to obtain the target physical frame number of the certain logical frame number in the target video segment.

5. The method according to claim 1, characterized in that The step of determining the I frame in the GOP where the target physical frame number is located according to the physical frame number of each I frame in the target video segment includes: Calling the I frame list of the target video segment, wherein the I frame list stores the physical frame number and position offset of each I frame in the target video segment; If the target physical frame number is between the physical frame numbers of two I frames, the previous I frame of the two I frames is determined as the I frame in the GOP where the target physical frame number is located.

6. The method according to claim 5, characterized in that The step of synthesizing and playing the video image corresponding to the target physical frame number according to the IDR image in the I frame includes: Reading an IDR image in the I frame according to the position offset of the I frame; Subtract the physical frame number of the I frame from the target physical frame number to obtain a frame number difference N1; The image data of the N1th P frame after the I frame is read, and the video image corresponding to the target physical frame number is synthesized by the image data and the IDR image and played.

7. The method according to claim 1, characterized in that Also includes: Annotating each video segment stored in the proxy file; When the vehicle storage space for a video segment reaches the upper limit, the video segments are deleted in the following order: The earliest unlabeled auxiliary video segments, the earliest unlabeled main video segments, and the earliest labeled video segments.

8. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the precise on-demand method for massive video data for autonomous driving as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, which, when executed by a processor, implements the precise on-demand method for massive video data for autonomous driving as described in any one of claims 1-7.

Images