Driving recorder video detection method and device based on ffmpeg and image recognition

Abstract

The present invention discloses a driving recorder video detection method and device based on ffmpeg and image recognition, the method comprising the following steps: obtaining a preset resolution image of the first frame of the first second of a video file, comparing it with a thumbnail of the video file, and determining whether the video file is abnormal according to the degree of difference; obtaining an original image of one frame of the video file per second based on ffmpeg, cutting out a time watermark area, extracting a time watermark value in the time watermark area based on image recognition, and judging whether the video file is continuous and normally segmented according to the time watermark value; comparing the actual duration of the video file with the preset duration, and judging whether the video file is abnormal according to the degree of error. The present invention can automatically detect a large number of video files in a relatively short time, locate the video files that may have problems and the specific time point when the problems occur, save test manpower, improve test efficiency and test accuracy, and shorten test time.

Description

Technical Field

[0001] The present invention relates to the technical field of driving recorder video recording test, and in particular to a driving recorder video recording detection method and device based on ffmpeg and image recognition. Background Art

[0002] As the dash cam market continues to be hot, product quality is becoming increasingly important. Video recording, as the core function of a dash cam, has strict requirements different from other products. During the testing process, it faces problems such as large number of video files, many test items, and time-consuming manual testing.

[0003] A large number of video files will be generated during the operation of the dash cam. For example, if a 16G TF card has a coverage threshold of 90%, and a 100M 1-minute video file is used as an example, about 144 files can be generated, totaling 2.5 hours of video. If the ordinary manual test method is used, it will take 2.5 hours just to play all the video files from beginning to end. If other checks are performed in accordance with the above-mentioned video requirements, the time required will be much longer than 2.5 hours. If the number of TF cards is doubled, the time required is obviously unacceptable. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes a driving recorder video detection method based on ffmpeg and image recognition, which can automatically detect a large number of video files in a short time, improve test efficiency, and shorten test time.

[0005] The present invention also proposes a driving recorder video detection device based on ffmpeg and image recognition having the above-mentioned driving recorder video detection method based on ffmpeg and image recognition.

[0006] The present invention also proposes a computer-readable storage medium having the above-mentioned driving recorder video detection method based on ffmpeg and image recognition.

[0007] According to the first aspect of the present invention, a dash cam video detection method based on ffmpeg and image recognition includes the following steps: obtaining a preset resolution image of the first frame of the first second of a video file, comparing it with a thumbnail of the video file, and determining whether the video file is abnormal according to the degree of difference; obtaining an original image of one frame of the video file per second based on ffmpeg, cropping a time watermark area, extracting a time watermark value in the time watermark area based on image recognition, and judging whether the video file is continuous and normally divided according to the time watermark value; comparing the actual duration of the video file with the preset duration, and judging whether the video file is abnormal according to the degree of error.

[0008] The dash cam video detection method based on ffmpeg and image recognition according to the embodiment of the present invention has at least the following beneficial effects: by performing thumbnail comparison, file continuity check and video duration detection, a large number of video files can be automatically detected in a short period of time, and the videos that may have problems and the specific time points when the problems occur can be located, thus saving testing manpower, improving testing efficiency and testing accuracy, and shortening testing time.

[0009] According to some embodiments of the present invention, a method for extracting a time watermark value in the time watermark region based on image recognition is: performing binarization denoising and domain denoising on the time watermark region and then performing image recognition.

[0010] According to some embodiments of the present invention, the time value listed in the video file name is extracted and compared with the time watermark value of the first frame of the first second of the video file. If they are equal, it is confirmed that the video file is segmented normally.

[0011] According to some embodiments of the present invention, judging whether the video files are continuous according to the time watermark value includes: if, in two temporally adjacent video files, the time watermark value corresponding to the last second of the previous video file is continuous with the time watermark value corresponding to the first second of the latter video file, then the two video files are confirmed to be continuous; if the time watermark values ​​obtained in two adjacent seconds of a single video file are continuous, then the single video file is confirmed to be continuous.

[0012] According to some embodiments of the present invention, it also includes: if a single video file is discontinuous, based on a preset threshold range, based on the time watermark value when the discontinuity occurs being within the corresponding preset time threshold, searching for freeze frames based on ffmpeg; if no freeze frames are found, it is confirmed that frame loss has occurred, and the video file and the corresponding time watermark value when the discontinuity occurs are recorded; if a freeze frame is found, it is confirmed that the screen is frozen, and the video file and the corresponding time watermark value when the discontinuity occurs are recorded.

[0013] According to some embodiments of the present invention, the method further includes: calling the blackdetect filter provided by ffmpeg to check the single video file to obtain the start time, duration and end time of the black screen frame.

[0014] According to some embodiments of the present invention, the method further includes: calling the silencedetect interface provided by ffmpeg to check the single video file to obtain the start time, duration and end time of the silent frame.

[0015] According to some embodiments of the present invention, it also includes: calling ffmpeg to decode a single video file, parsing the frame information of the audio frame, obtaining the pts_size value of each audio frame, and obtaining the corresponding volume average value; comparing the difference between the pts_size value of each audio frame and the volume average value, if the difference is not within the corresponding threshold, confirming that a recording mutation abnormality has occurred, and recording the video file and the time when the recording mutation occurred.

[0016] According to the second aspect of the present invention, a driving recorder video detection device based on ffmpeg and image recognition includes: a thumbnail comparison module, which is used to obtain a preset resolution image of the first frame of the first second of the video file, compare it with the thumbnail of the video file, and determine whether the video file is abnormal according to the degree of difference; a continuity detection module, which is used to obtain the original image of one frame of the video file per second based on ffmpeg, crop out the time watermark area, extract the time watermark value in the time watermark area based on image recognition, and judge whether the video file is continuous and normally divided according to the time watermark value; a duration detection module, which is used to compare the actual duration of the video file with the preset video duration, and judge whether the video file is abnormal according to the degree of error.

[0017] According to the embodiment of the present invention, the driving recorder video detection device based on ffmpeg and image recognition has at least the following beneficial effects: by performing thumbnail comparison, file continuity check and video duration detection, a large number of video files can be automatically detected in a short time, and the videos that may have problems and the specific time points when the problems occur can be located, thus saving testing manpower, improving testing efficiency and testing accuracy, and shortening testing time.

[0018] The computer-readable storage medium according to the third aspect of the present invention has a computer program stored thereon, and when the computer program is executed by a processor, the method according to the first aspect of the present invention is implemented.

[0019] The computer-readable storage medium according to the embodiment of the present invention has at least the same beneficial effects as the method of the embodiment of the first aspect of the present invention.

[0020] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0022] Figure 1A schematic diagram of a method according to an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of the main detection process in the method of an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of a detection process for a single video file in a method according to an embodiment of the present invention;

[0025] Figure 4 A schematic diagram of a process for detecting abnormal recordings in a method according to an embodiment of the present invention;

[0026] Figure 5 The present invention is a schematic block diagram of a device according to an embodiment of the present invention.

[0027] Reference numerals:

[0028] A driving recorder video detection device 100 , a thumbnail comparison module 110 , a continuity detection module 120 , and a duration detection module 130 . DETAILED DESCRIPTION

[0029] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0030] In the description of the present invention, the meaning of "several" is one or more, the meaning of "more" is two or more, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of the first and the second, it is only for the purpose of distinguishing the technical features, and it cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features. In the description of the present invention, the step numbers are only used for the convenience of description or citation. The size of the serial number of each step does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0031] Reference Figure 1The method of the embodiment of the present invention includes: obtaining a preset resolution image of the first frame of the first second of a video file, comparing it with the thumbnail of the video file, and determining whether the video file is abnormal according to the degree of difference; obtaining an original image of one frame of the video file per second based on ffmpeg, cutting out a time watermark area, extracting a time watermark value in the time watermark area based on image recognition, and judging whether the video file is continuous and normally divided according to the time watermark value; comparing the actual duration of the video file with the preset duration of the video, and judging whether the video file is abnormal according to the degree of error.

[0032] In an embodiment of the present invention, the following steps are included: checking the video thumbnail of the video file, checking whether the time corresponding to the first frame is the same as the time included in the video file name, checking whether the video duration is normal, checking whether the watermark time value of a single file is continuous, checking whether there is a black screen / frozen screen in a single video file, and checking whether the recording problem in the video file is normal.

[0033] Check the video thumbnail of the video file to determine whether it is displayed normally, including:

[0034] 1) Determine whether the thumbnail is damaged; check whether all video thumbnail files are damaged by opening the file. If it cannot be opened normally, the file is damaged and the abnormal file is recorded;

[0035] 2) Whether the thumbnail is the same as the first frame of the video; take out a picture of the specified resolution in the first second of a frame from the video, compare it with the video thumbnail, and determine whether the degree of difference is within the threshold range. If it exceeds the threshold range, the file is confirmed to be abnormal and the abnormal file is recorded.

[0036] Check whether the time corresponding to the first frame is the same as the time included in the video file name. Get the time value in the video file name and the start time of the video file (that is, the time watermark value of a frame of image in the first second), and compare them. If the time is the same, for example, the same second, the file is considered normal, otherwise the file is abnormal. This test can detect some errors that occur when the device is segmenting the video.

[0037] Since the duration of each video recorded by the dashcam is fixed, it is necessary to check whether the duration is normal. That is, obtain the actual duration of each video file (accurate to milliseconds) and compare it with the given duration. If the error is within the threshold range, it is considered normal. If it exceeds the error, it is considered abnormal and a failed file is recorded.

[0038] Check whether the watermark time value of a single file is continuous to prevent problems such as frame loss in the recording file. The time watermark value of a normally encoded video is continuous per second. Take the time watermark value of one frame of image per second from the video and compare whether it is continuous. If it is not continuous, there may be frame loss. If it is continuous, it may be normal, or there may be a frozen screen or a black screen. In both cases, the next step of judgment is required.

[0039] In the embodiment of the present invention, it is also checked whether the watermark time values ​​of two adjacent files are continuous. Check whether the time watermark values ​​of the last second of the previous video file and the first second of the next video file are continuous in chronological order to determine whether the two video segments are continuous and whether there is a phenomenon of missing seconds. This test is important for devices that record videos in segments, such as driving recorders and IPCs, and can help testers quickly and easily locate discontinuous video files.

[0040] Check whether there is a black screen or frozen screen in a single video file, including:

[0041] 1) Screen freeze detection: directly detect whether there is a screen freeze frame in the video file. If it exists, record the problem file, the time when the problem occurred, etc.

[0042] 2) Black screen detection: directly detect whether there is a black screen frame in the video file. If it exists, record the problem file, the time when the problem occurred, etc.

[0043] Check whether the recording problem in the video file is normal, including:

[0044] 1) Silence detection: directly detect whether there are silence frames in the video file. If there are, record the problem file, the time when the problem occurred, etc.

[0045] 2) Volume abnormality detection. Generally, the volume level is within a certain range and can be regarded as the average value. If the difference between the volume of a certain frame and the average value exceeds a certain threshold, it is confirmed that there is an abnormal volume mutation. The abnormal problem file and the time when the problem occurred are recorded.

[0046] The method of the embodiment of the present invention refers to Figure 2 At the beginning of the test, the test file path is imported (i.e. the path of the given video file to be tested), the thumbnail file corresponding to the video file is taken out and opened, and the image of the first frame of the first second of the video file is obtained at the specified resolution to perform a video thumbnail check.

[0047] Specifically: 1) Use ffmpeg to decode the video file, and extract the first frame image of the first second of the video file with the same resolution as the video thumbnail; 2) Compare the video thumbnail with the frame image extracted in the previous step. If the error is within the threshold, the two are considered to be the same; if the error is outside the threshold, the two are considered to be different, and the thumbnail of the file is recorded as abnormal.

[0048] Then, one frame of the original image is taken per second, and the time watermark area is cut out. After image preprocessing, the time watermark value is identified, and file segmentation and file continuity detection are performed based on the time watermark value. For file segmentation inspection, the specific process is as follows:

[0049] 1) Use ffmpeg to decode the video file and extract the first frame of the first second of the video at the original resolution;

[0050] 2) Based on the image taken in the previous step, cut out the area where the time watermark is located;

[0051] 3) Perform corresponding image processing (binarization denoising and neighborhood denoising) on ​​the time watermark area image obtained in step 2 to facilitate image recognition of specific time values;

[0052] 4) Extract the time value contained in the video file name;

[0053] 5) Compare the values ​​of step 3 and step 4. If they are equal, the file segmentation is considered normal. If they are not equal, the file segmentation is considered abnormal.

[0054] The file continuity detection includes: whether the watermark time value of a single video file is continuous, and whether the watermark time values ​​of two temporally adjacent videos are continuous.

[0055] Check whether the watermark time of a single video file is continuous, including obtaining the original image from the video file based on ffmpeg at a frequency of one frame per second according to the video duration, cutting the time watermark area and identifying the watermark time value, and comparing whether the watermark time values ​​taken from two adjacent times are continuous. The detailed judgment of a single video file will be described later.

[0056] For two adjacent video files in chronological order, specifically, including:

[0057] 1) Use ffmpeg to decode the video file, take one frame of image every 1 second at the original resolution according to the video duration;

[0058] 2) Based on the image taken in the previous step, cut out the area where the time watermark is located;

[0059] 3) Perform corresponding image processing (binarization denoising and neighborhood denoising) on ​​the time watermark area image obtained in step 2 to facilitate image recognition of specific time values;

[0060] 4) Compare the last 1 second time watermark value of the previous video with the first second time watermark value of the next video to check whether they are continuous. If so, the two video files can be considered continuous. If not, they are considered discontinuous.

[0061] The above detection of a single file to determine whether the time watermark is continuous, if the watermark time value is not continuous, there may be frame loss, if it is continuous, it may be normal, or there may be a frozen frame, both of which need to be judged in the next step.

[0062] The process of checking the abnormality of the video recording of a single file in the embodiment of the present invention is as follows: Figure 3 ,include:

[0063] 1) Perform black screen detection on the file: call the blackdetect filter that comes with ffmpeg, and you can customize the detection parameters. The returned result will include whether it contains black screen frames. If there are black screen frames, it will include the start time, duration, end time and other information of the black screen frames.

[0064] 2) Video recording freeze detection

[0065] Use ffmpeg to decode the video file, take one frame of image every 1 second at the original resolution according to the video duration;

[0066] Based on the image taken in the previous step, cut out the area where the time watermark is located;

[0067] Perform corresponding image processing (binarization denoising and neighborhood denoising) on ​​the time watermark area image obtained in the previous step to facilitate image recognition of specific time values;

[0068] Compare the time watermark value of each second of the video to see if it is continuous. If it is not continuous, there may be frame loss. If it is continuous, it may be normal, or there may be frozen frames. In both cases, the next step of judgment is required;

[0069] Case 1: If the time watermark is not continuous, continue to check the freeze frame:

[0070] If a frozen frame is detected within a certain time threshold range before and after the time point where the time watermark is discontinuous, it means that the video is stuck and the time watermark is discontinuous. The file name and the time point where the time watermark is discontinuous are recorded for manual review.

[0071] If no frozen frames are detected within a certain time threshold range before and after the time point of the time watermark discontinuity, it means that the video has discarded frozen frames and the picture jumps directly. The file name and the time point of the time watermark discontinuity are recorded for manual re-inspection;

[0072] If freeze frames are detected at other time points, it means that the video is stuck and the time watermark is continuous. The file name and freeze frame time point are recorded for manual review.

[0073] Case 2: Continuing from the previous item, if the time watermark is continuous, continue to check for frozen frames. If a frozen frame is detected, record the file name and the time point of the frozen frame for manual recheck.

[0074] In an embodiment of the present invention, the method further includes: detecting the volume of the video file. Figure 4 , including: importing the directory of video files and performing silence check. Silence check, specifically, calls the silencedetect interface that comes with ffmpeg, and the detection parameters can be customized. The returned result will include whether there is a silent frame. If there is a silent frame, it will include the start time, duration, end time and other information of the silent frame. If no silent file is detected, the recording is normal, and the volume abnormality detection continues. The volume abnormality check specifically includes: calling ffmpeg to decode the video file, parse the detailed information of each audio frame, and obtain the pts_size value; the pts_size value of all audio frames, and obtain its average value. If the difference between a certain audio frame and the average value exceeds the preset threshold, it is marked as a recording mutation abnormality, and the file name and time point of the abnormal audio frame are recorded for manual review.

[0075] The time watermark value recognition described above is all performed through the character library trained by Tesseract 4.0LSTM. The training process: sample image acquisition, image annotation, and character library training can all be automatically completed through script tools, and the training can be completed in a relatively short time. The correct recognition rate of the current trained character library is about 90%.

[0076] The device 100 in the embodiment of the present invention refers to Figure 5 , including: a thumbnail comparison module 110, used to obtain a preset resolution image of the first frame of the first second of a video file, compare it with the thumbnail of the video file, and determine whether the video file is abnormal according to the degree of difference; a continuity detection module 120, used to obtain an original image of one frame of the video file per second based on ffmpeg, cut out a time watermark area, extract the time watermark value in the time watermark area based on image recognition, and determine whether the video file is continuous and normally divided according to the time watermark value; a duration detection module 130, used to compare the actual duration of the video file with the preset video duration, and determine whether the video file is abnormal according to the degree of error.

[0077] Although specific embodiments are described herein, those of ordinary skill in the art will recognize that many other modifications or alternative embodiments are also within the scope of the present disclosure. For example, any of the functions and / or processing capabilities described in conjunction with a particular device or component may be performed by any other device or component. In addition, although various exemplary implementations and architectures have been described according to embodiments of the present disclosure, those of ordinary skill in the art will recognize that many other modifications to the exemplary implementations and architectures described herein are also within the scope of the present disclosure.

[0078] Some aspects of the present disclosure are described above with reference to the block diagram and flow chart of the system, method, system and / or computer program product according to the exemplary embodiments.It should be understood that the combination of one or more blocks in the block diagram and the flow chart and the blocks in the block diagram and the flow chart can be realized by executing computer executable program instructions respectively.Equally, according to some embodiments, some blocks in the block diagram and the flow chart may not need to be executed in the order shown, or may not need to be executed in full.In addition, additional components and / or operations beyond those components and / or operations shown in the blocks in the block diagram and the flow chart may be present in certain embodiments.

[0079] Therefore, the blocks in the block diagrams and flow charts support the combination of devices for performing the specified functions, the combination of elements or steps for performing the specified functions, and the program instruction devices for performing the specified functions. It should also be understood that each block in the block diagrams and flow charts and the combination of blocks in the block diagrams and flow charts can be implemented by a special-purpose hardware computer system or a combination of special-purpose hardware and computer instructions that performs a specific function, element or step.

[0080] The program modules, applications, etc. described herein may include one or more software components, including, for example, software objects, methods, data structures, etc. Each such software component may include computer-executable instructions that, in response to execution, cause at least a portion of the functionality described herein (e.g., one or more operations of the illustrative methods described herein) to be performed.

[0081] Software components can be encoded with any of various programming languages. An exemplary programming language can be a low-level programming language, such as an assembly language associated with a specific hardware architecture and / or an operating system platform. The software component comprising assembly language instructions may need to be converted to executable machine code by an assembler before being executed by hardware architecture and / or platform. Another exemplary programming language can be a higher-level programming language, which can be transplanted across multiple architectures. The software component comprising a higher-level programming language may need to be converted to an intermediate representation by an interpreter or a compiler before execution. Other examples of programming languages ​​include but are not limited to macro languages, shells or command languages, job control languages, scripting languages, database queries or search languages ​​or report writing languages. In one or more exemplary embodiments, the software component comprising the instruction of one of the above-mentioned programming language examples can be directly executed by an operating system or other software components, without first being converted into another form.

[0082] Software components may be stored as files or other data storage structures. Software components of similar type or related functions may be stored together, such as in a specific directory, folder, or library. Software components may be static (e.g., preset or fixed) or dynamic (e.g., created or modified at execution time).

[0083] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in the relevant technical field without departing from the purpose of the present invention.

Claims

1. A driving recorder video detection method based on ffmpeg and image recognition, characterized in that: The following steps are involved: Obtain a preset resolution image of the first frame of the first second of the video file, compare it with the thumbnail of the video file, and determine whether the video file is abnormal based on the degree of difference; Based on ffmpeg, one frame of the original image of the video file is obtained per second, a time watermark area is cut out, a time watermark value in the time watermark area is extracted based on image recognition, and whether the video file is continuous and normally segmented is determined according to the time watermark value; If a single video file is discontinuous, based on a preset threshold range, based on the time watermark value when the discontinuity occurs being within the corresponding preset time threshold, searching for frozen frames based on ffmpeg; If no frozen frame is found, it is confirmed that frame loss occurs, and the video file and the time watermark value when the corresponding discontinuity occurs are recorded; If a frozen frame is found, confirm that the image is frozen, and record the video file and the time watermark value when the corresponding discontinuity occurs; According to the actual duration of the video file, the preset duration is compared, and whether the video file is abnormal is determined according to the degree of error; Call the silencedetect interface of ffmpeg to check the single video file and obtain the start time, duration and end time of the silent frame; Call ffmpeg to decode a single video file, parse the frame information of the audio frame, obtain the pts_size value of each audio frame, and obtain the corresponding volume average value; Compare the difference between the pts_size value of each audio frame and the volume average value. If the difference is not within the corresponding threshold, it is confirmed that a recording mutation abnormality occurs, and the video file and the time when the recording mutation occurs are recorded.

2. The driving recorder video detection method based on ffmpeg and image recognition according to claim 1 is characterized in that: The method for extracting the time watermark value in the time watermark area based on image recognition is: after performing binarization denoising and domain denoising on the time watermark area, image recognition is performed.

3. The driving recorder video detection method based on ffmpeg and image recognition according to claim 1 is characterized in that: The time value at the name of the video file is extracted and compared with the time watermark value of the first frame of the first second of the video file. If they are equal, it is confirmed that the video file is segmented normally.

4. The driving recorder video detection method based on ffmpeg and image recognition according to claim 1 is characterized in that: Judging whether the video file continuously includes: If, in two video files adjacent in time, the time watermark value corresponding to the last second of the previous video file is continuous with the time watermark value corresponding to the first second of the next video file, then the two video files are confirmed to be continuous; If the time watermark values ​​obtained in two adjacent seconds of a single video file are continuous, it is confirmed that the single video file is continuous.

5. The driving recorder video detection method based on ffmpeg and image recognition according to claim 1 is characterized in that: Also includes: Call the blackdetect filter of ffmpeg to check the single video file and obtain the start time, duration and end time of the black screen frame.

6. A driving recorder video detection device based on ffmpeg and image recognition, used to execute any method in claims 1 to 5, characterized in that: include: A thumbnail comparison module is used to obtain a preset resolution image of the first frame of the first second of the video file, compare it with the thumbnail of the video file, and determine whether the video file is abnormal according to the degree of difference; A continuity detection module, used to obtain an original image of the video file of one frame per second based on ffmpeg, cut out a time watermark area, extract a time watermark value in the time watermark area based on image recognition, and determine whether the video file is continuous and normally segmented according to the time watermark value; The duration detection module is used to compare the actual duration of the video file with the preset video duration, and determine whether the video file is abnormal according to the degree of error.

7. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 5 when executed by a processor.

Images