Multi-channel video analysis method, analysis equipment and storage medium

Abstract

The embodiment of the invention relates to the technical field of video analysis, and discloses a multi-channel video analysis method, which comprises the following steps: acquiring a first coding stream of a plurality of original video sequences; acquiring a plurality of corresponding timestamp sequences based on the plurality of first coding streams; generating a video frame index matrix according to the plurality of timestamp sequences; and analyzing the video frame index matrix to generate an analysis report. According to the method, the timestamp sequence with high precision is obtained based on the first code in the original video sequence, the video frame index matrix is generated based on the obtained timestamp sequences, the analysis report is automatically generated according to the generated video frame index matrix, and automation of multi-channel video analysis is achieved.

Description

Technical Field

[0001] This application relates to the field of video analysis technology, and in particular to a method, analysis device and storage medium for analyzing multi-channel video. Background Technology

[0002] With the rapid development of microelectronics technology and computer vision, multi-camera array systems are widely used in cutting-edge fields such as autonomous driving, 3D reconstruction, robot navigation, industrial quality inspection, and virtual reality. In these applications, the system needs to synchronously acquire video data from multiple spatial perspectives to obtain complete environmental information. Therefore, there are high requirements for the integrity of each video data stream and the precise synchronization of timestamps between the video data streams.

[0003] Currently, existing technologies have limitations in ensuring the reliability and synchronization accuracy of multi-camera data acquisition, such as a single fault detection dimension and low time synchronization accuracy. At the same time, commonly used fault detection methods also suffer from difficulties in locating synchronization anomalies and a lack of automated quantification.

[0004] Application content The embodiments of this application mainly provide a method for analyzing multi-channel video, which uses more accurate timestamps to automatically generate analysis reports, improving the accuracy of analysis while achieving automation.

[0005] To solve the above-mentioned technical problems, the embodiments of this application adopt the following technical solutions: Firstly, embodiments of this application provide a method for analyzing multiple video streams, including: Obtain the first encoded stream of multiple raw video sequences; Multiple timestamp sequences are obtained based on multiple first encoded streams; Generate a video frame index matrix based on multiple timestamp sequences; The video frame index matrix is ​​parsed to generate an analysis report.

[0006] In some embodiments, generating a video frame index matrix based on multiple timestamp sequences includes: From multiple timestamp sequences, one timestamp sequence is selected as the first timestamp sequence, and the remaining timestamp sequences are used as the second timestamp sequence. Each second timestamp sequence is time-aligned with the first timestamp sequence to generate a video frame index matrix.

[0007] In some embodiments, each second timestamp sequence is time-aligned with a first timestamp sequence to generate a video frame index matrix, including: Based on the first timestamp sequence and preset rules, determine the second target timestamp in the second timestamp sequence that is aligned with the first timestamp; The video frame index corresponding to the second target timestamp is used as the matrix unit corresponding to the first timestamp to obtain the initial index matrix; The video frame index matrix is ​​obtained by filling the empty matrix cells in the initial index matrix with the lost frame identifier.

[0008] In some embodiments, the preset rule includes: taking the timestamps in the second timestamp sequence whose absolute difference from the first timestamp is less than a preset threshold as the second target timestamps.

[0009] In some embodiments, parsing the video frame index matrix to generate an analysis report includes: The missing frame identifiers in the video frame index matrix are statistically analyzed, and an analysis report is generated.

[0010] In some embodiments, before acquiring a first encoded stream of multiple raw video sequences, the method further includes: Perform functional testing on multiple video capture devices; If the functional test results of multiple video acquisition devices are normal, then multiple raw video sequences acquired by the multiple video acquisition devices are obtained.

[0011] In some embodiments, functional testing of multiple video capture devices includes: Test the network connectivity of multiple video capture devices using inspection commands and / or testing tools.

[0012] In some embodiments, functional testing of multiple video capture devices includes: Keyframes are extracted from multiple original video sequences according to a preset frequency, and the video acquisition function of multiple video acquisition devices is tested based on a preset functional model and keyframes.

[0013] In some embodiments, obtaining multiple timestamp sequences based on multiple first encoded streams includes: The first encoded stream is decomposed into multiple first units, and the first unit containing the first data information is used as the second unit; The second unit is parsed according to a preset protocol to obtain the timestamp sequence corresponding to the first encoded stream.

[0014] Secondly, embodiments of this application also provide a video analysis device, including: At least one processor, and A memory communicatively connected to at least one processor, wherein, The memory stores instructions that can be executed by at least one processor, such that the at least one processor is able to perform a method as described in any of the first aspects.

[0015] Thirdly, embodiments of this application also provide a video analysis system, including: At least two video capture devices; Network equipment, and video analytics equipment as provided in the second aspect.

[0016] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer-executable instructions for causing a computer device to perform the method as described in any of the first aspects.

[0017] The beneficial effects of this application's embodiments are as follows: Unlike existing technologies, this application provides a method for analyzing multiple video streams, including: acquiring first encoded streams of multiple original video sequences; acquiring multiple timestamp sequences corresponding to the multiple first encoded streams; generating a video frame index matrix based on the multiple timestamp sequences; and parsing the video frame index matrix to generate an analysis report. This method acquires high-precision timestamp sequences based on the first encoded streams of the original video sequences, generates a video frame index matrix based on the acquired multiple timestamp sequences, and automatically generates an analysis report based on the generated video frame index matrix, thus automating the analysis of multiple video streams. Attached Figure Description

[0018] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0019] Figure 1 This is a flowchart illustrating a multi-channel video analysis method provided in an embodiment of this application; Figure 2 This is a schematic diagram of a video frame index matrix provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a video analysis system provided in an embodiment of this application; Figure 4 This is a schematic diagram illustrating the specific process of a multi-channel video analysis method provided in an embodiment of this application. Detailed Implementation

[0020] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0022] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. In addition, the terms "first," "second," and "third" used herein do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0023] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0024] Furthermore, the technical features involved in the various embodiments of this application described below can be combined with each other as long as they do not conflict with each other.

[0025] With the rapid development of microelectronics and computer vision, multi-camera array systems have been widely applied in cutting-edge fields such as autonomous driving, 3D reconstruction, robot navigation, industrial quality inspection, and virtual reality. In these applications, the system needs to simultaneously acquire video data from multiple spatial perspectives to obtain complete environmental information. Simultaneously, it needs to ensure the integrity of the data stream and the precise synchronization of timestamps between the video streams to confirm the accuracy of subsequent algorithms, such as sensor fusion, stereo matching, and motion estimation.

[0026] Currently, typical multi-camera array systems usually connect each camera to one or more data acquisition servers via a network (such as Ethernet). The cameras push video data to the server in real time for recording using streaming media protocols (such as RTSP). To ensure data quality, it is necessary to monitor the stability and synchronization of the data stream.

[0027] Existing technologies for fault detection are limited in scope and cannot cover anomalies at the content layer. Furthermore, most systems rely on the system clock of the acquisition server as the timestamp for data frames, or on the timestamp recorded by the video container itself. Both of these timestamps are prone to errors and cannot guarantee accurate synchronization.

[0028] In addition, when frame loss or synchronization drift occurs during acquisition, operators need to manually compare multiple video files one by one to locate the problem, resulting in low fault detection efficiency.

[0029] In view of this, firstly, embodiments of this application provide a method S100 for analyzing multiple video streams, such as... Figure 1 As shown, the method includes: S10: Obtain the first encoded stream of multiple raw video sequences.

[0030] In a multi-camera array system, after multiple video capture devices acquire video data, they package the captured video sequences into data packets according to a specific format, such as MP4 or MKV. These packaged data packets are then uploaded to a server for storage or other processing. In this process, the video sequences captured by the video capture devices before packaging are the original video sequences. Correspondingly, the video sequences captured by multiple video capture devices constitute multiple original video sequences.

[0031] After the video acquisition device completes photoelectric conversion and performs minimal preprocessing (denoising, white balance, gain control) or compression by a video encoder such as an H.265 encoder, the first stream sent to the system memory is the first encoded stream, also known as the raw video encoded stream (Elementary Stream). This first encoded stream is not pre-packaged data, but rather a pure, unpackaged stream of raw pixels. Based on this, by acquiring multiple raw video sequences from multiple video acquisition devices, the first encoded stream corresponding to each raw video sequence can be obtained, thereby avoiding errors that may be introduced by video container packaging and avoiding errors caused by network transmission after packaging.

[0032] S20: Obtain multiple timestamp sequences based on multiple first encoded streams.

[0033] As can be seen from the foregoing, during the transmission of video data, the first encoded stream of the original video sequence itself contains a corresponding timestamp. At the same time, the video container used to store the packaged video data also has a corresponding timestamp, and the server receiving the relevant video data also has a corresponding timestamp.

[0034] Because the timestamps of video containers are affected by the encoder and packer buffer inside the video capture device, accurate synchronization cannot be guaranteed; while the timestamps of servers introduce uncertain delays and jitters on the network transmission path, resulting in timestamp errors that may reach tens or even hundreds of milliseconds, thus making it impossible to synchronize the timestamps accurately.

[0035] Therefore, in this embodiment, the timestamps in the first encoded stream are used as the timestamp sequence of the corresponding original video; that is, multiple timestamp sequences are obtained through multiple first encoded streams. It can be understood that this timestamp sequence is constructed sequentially from the timestamps of all video frames contained in the corresponding original video sequence. By using the timestamps in the first encoded stream of the original video sequence, which has higher precision, as the timestamp sequence of the corresponding original video sequence, the accuracy of the obtained timestamp sequence can be improved, and inaccuracies in the obtained timestamp sequence due to other influences can be avoided.

[0036] In some embodiments, the aforementioned step S20 specifically includes: S21: Decompose the first encoded stream into multiple first units and take the first unit containing the first data information as the second unit; parse the second unit according to the preset protocol to obtain the timestamp sequence corresponding to the first encoded stream.

[0037] As mentioned above, the first encoded stream is the raw encoded stream of the original video sequence that has not undergone container processing. Correspondingly, the first encoded stream is decomposed into network abstraction layer units, i.e., the first unit. The first unit is a self-contained binary data block, mainly composed of a start code, NALU header bytes, and RBSP payload (Raw ByteSequence Payload). The same video frame can be decomposed into multiple first units, and different first units carry different data information. For example, some first units store parameters, while others store images.

[0038] Based on this, after decomposing into multiple first units, the content stored in the first units is judged, and the first unit containing the first data information is used as the second unit. Here, the first data information is supplementary enhancement information; that is, the first unit containing the supplementary enhancement information is used as the second unit. The second unit is parsed according to a preset protocol to obtain the timestamp corresponding to the second unit. This timestamp is actually the timestamp corresponding to the video frame from which the second unit is decomposed. Repeating the above process for multiple first encoded streams yields a timestamp sequence corresponding to multiple first encoded streams, and there is a one-to-one correspondence between the obtained timestamp sequence and the multiple first encoded streams.

[0039] For example, in this embodiment, taking the H.265 (HEVC) bitstream as the first encoded stream, firstly, the acquired h.265 raw video file is demultiplexed into a series of Network Abstraction Layer Units (NALUnits); secondly, all NALUnits are traversed to identify and filter out units containing Supplemental Enhancement Information (SEI); then, according to a predefined protocol (e.g., a specific UUID identifier), the payload of the SEI unit is parsed to extract the nanosecond or millisecond-level high-precision timestamps embedded by the camera hardware or driver; finally, this process is repeated for all video streams to generate an independent timestamp sequence file for each video acquisition device.

[0040] By using the timestamps in the first encoded stream of the original video sequence, which has higher precision, as the timestamp sequence of the corresponding original video sequence, the accuracy of the obtained timestamp sequence can be improved, and the inaccuracy of the obtained timestamp sequence due to other influences can be avoided.

[0041] S30: Generate a video frame index matrix based on the timestamp sequence; parse the video frame index matrix to generate an analysis report.

[0042] As described above, the timestamp sequence contains the timestamps corresponding to all video frames in the original video sequence. Therefore, by analyzing multiple timestamp sequences in a list, abnormal or missing video frames can be identified. In this embodiment, a matrix is ​​used to analyze the timestamp sequences in a list. For example, the first column of the matrix is ​​the timestamp sequence, and the columns after the first column are the video frame indices corresponding to the timestamps in the first column from different acquisition devices. The matrix filled with these video frame indices is the video frame index matrix.

[0043] like Figure 2As shown, when the timestamp of the main camera in the first column is 1700000000001, the video frame index of video acquisition device 1 as the main camera (camera 1 frame index) is 1, the video frame index of video acquisition device 2 (camera 2 frame index) is 1, the video frame index of video acquisition device 3 (camera 3 frame index) is 1, and the video frame index of video acquisition device 4 (camera 4 frame index) is 1; when the timestamp of the first column is 1700000000018, the video frame index of video acquisition device 1 is 2, the video frame index of video acquisition device 2 is 2, and the video frame index of video acquisition device 3 is 2.

[0044] As can be seen, the aforementioned video frame index matrix includes the video frame indexes corresponding to the timestamps in the first column from each video acquisition device. By analyzing the content of the video frame index matrix, abnormal situations in multiple original video sequences acquired by each video acquisition device can be captured, and corresponding analysis reports can be generated.

[0045] In this embodiment, by analyzing the video frame index matrix filled with video frame indexes, an analysis report is automatically generated. Operators no longer need to manually compare multiple video files one by one and locate problems by visual observation, thereby improving the efficiency of multi-channel video analysis.

[0046] In some embodiments, step S30 above, "generating a video frame index matrix based on the timestamp sequence," specifically includes: S31: Determine one timestamp sequence from multiple timestamp sequences as the first timestamp sequence, and use the remaining timestamp sequences as the second timestamp sequences; align each second timestamp sequence with the first timestamp sequence in time to generate a video frame index matrix.

[0047] Understandably, in multi-channel video analysis (or more broadly, multimedia synchronization) scenarios, since the multiple raw video sequences acquired come from different video capture devices or different network paths, the timestamps of these raw video sequences are not perfectly aligned. Therefore, after obtaining the timestamp sequences corresponding to each raw video sequence through multiple first encoded streams, a time-series alignment operation of the timestamp sequences is required.

[0048] First, from multiple timestamp sequences, one timestamp sequence is determined as the master clock, i.e., the first timestamp sequence; the remaining timestamp sequences are designated as the second timestamp sequences. The second timestamp sequences are then aligned using the first timestamp sequence. This alignment between the second and first timestamp sequences is a temporal alignment. The result of this temporal alignment is used as matrix elements to fill the video frame index matrix. The temporal alignment result between the first and second timestamp sequences characterizes the state of the video frames in the original video sequences corresponding to the second timestamp sequence, including normal and abnormal states. Based on this, by analyzing the obtained video frame index matrix, the state of data frames in multiple original video sequences can be confirmed, and a corresponding analysis report can be automatically generated based on the analysis results.

[0049] Based on a defined first timestamp sequence, the second timestamp sequence is time-aligned, and the alignment result is used as matrix elements to fill the video frame index matrix. This can accurately capture abnormal states of video frames in the original video sequence, improving the accuracy of the analysis report obtained from the analysis based on the aforementioned video frame index matrix.

[0050] In some embodiments, the aforementioned step S31, "aligning each second timestamp sequence with the first timestamp sequence in time to generate a video frame index matrix," specifically includes: S321: Based on the first timestamp sequence and preset rules, determine the second target timestamp in the second timestamp sequence that is aligned with the first timestamp; use the video frame index corresponding to the second target timestamp as the matrix unit corresponding to the first timestamp to obtain the initial index matrix; fill the empty matrix units in the initial index matrix with the lost frame identifier to obtain the video frame index matrix.

[0051] Understandably, since the multiple original video frame sequences acquired come from different video acquisition devices, there are certain differences between the timestamp sequences contained in each original video frame sequence, and they are not completely synchronized. By selecting a first timestamp sequence from the multiple timestamp sequences obtained based on the first bitstream of the original video sequences as the master clock, and according to preset rules, a second target timestamp in the second target timestamp sequence that is aligned with the first timestamp can be determined; that is, the first timestamp and the second target timestamp satisfy the preset rules.

[0052] As can be seen from the foregoing, there is a correspondence between timestamps and video frames in the original video sequence. Therefore, by determining the second target timestamp, the video frame index in the original video sequence corresponding to that timestamp can be obtained. Using that video frame index as a matrix unit corresponding to the first timestamp, an initial index matrix can be obtained. This initial index matrix consists of the first timestamp and the video frame index in each original video sequence corresponding to the second target timestamp.

[0053] For example, such as Figure 2 As shown, the first timestamp sequence is used as the master clock, which is the main camera timestamp sequence. When the first timestamp is 1700000000001, the video frame index in the original video sequence captured by video acquisition device 1 (main camera 1) corresponding to the first timestamp is 1. The video frame index in the original video sequence captured by video acquisition device 2 (camera 2) corresponding to the second target timestamp is 1. The video frame index in the original video sequence captured by video acquisition device 3 (camera 3) corresponding to the second target timestamp is 1. The video frame index in the original video sequence captured by video acquisition device 4 (camera 4) corresponding to the second target timestamp is 1. When the first timestamp is 1700000000018, the video frame index in the original video sequence captured by video acquisition device 1 corresponding to the first timestamp is 2. The video frame index in the original video sequence captured by video acquisition device 2 corresponding to the second target timestamp is 2. The video frame index in the original video sequence captured by video acquisition device 3 corresponding to the second target timestamp is 2.

[0054] Understandably, due to the influence of various factors on the error between the first timestamp sequence and the second target timestamp sequence, there may be timestamps in the second target timestamp sequence that do not correspond to the first timestamps. In other words, not all first timestamps have corresponding second target timestamps in the second target timestamp sequence that meet the preset conditions. That is, there are empty matrix cells in the initial index matrix. These empty matrix cells represent that there are no second target timestamps in each original video sequence that meet the preset conditions for the corresponding first timestamps. Based on this, the empty matrix cells in the initial index matrix are filled with lost frame identifiers to obtain a video frame index matrix. This video frame index matrix includes both the video frame indices corresponding to the second target timestamps that meet the preset conditions in each original video sequence and the lost frame identifiers that do not meet the preset conditions.

[0055] For example, such as Figure 2 As shown, when the first timestamp is 1700000000018, the video frame index in the original video sequence captured by video acquisition device 1 corresponding to the first timestamp is 2, the video frame index in the original video sequence captured by video acquisition device 2 corresponding to the second target timestamp is 2, the video frame index in the original video sequence captured by video acquisition device 3 corresponding to the second target timestamp is 2, and there is no second target timestamp in video acquisition device 4 that meets the preset conditions with the first timestamp. Therefore, the data stored in the matrix unit corresponding to the first timestamp in video acquisition device 4 is the lost frame identifier e'.

[0056] By writing the video frame index of the corresponding original video sequence into the matrix unit that meets the preset conditions, and writing the lost frame identifier into the matrix unit that does not meet the preset conditions, a video frame index matrix is ​​obtained. This enables a structured representation of the time-series alignment results of the first and second timestamp sequences, accurately capturing the positions of video frames in abnormal states in each original video sequence, and improving the accuracy of video analysis. At the same time, by using a matrix to represent the time-series alignment results, the analysis device can automatically generate an analysis report based on the video frame index matrix, thus improving the efficiency of video analysis.

[0057] In some embodiments, the preset rule includes: using timestamps in the second timestamp sequence whose absolute difference from the first timestamp is less than a preset threshold as second target timestamps. For example, iterating through each first timestamp T_master in the first timestamp sequence, for each first timestamp T_master, in all other second timestamp sequences, if there exists a timestamp T_slave that satisfies the absolute difference |T_master-T_slave|<ΔT, where ΔT is a preset threshold, i.e., the synchronization tolerance window of the first timestamp sequence, then that timestamp T_slave is used as the second target timestamp.

[0058] It is understandable that the aforementioned preset rule actually determines the second target timestamp as the timestamp within the first timestamp tolerance window in each second timestamp sequence. Based on the preset rule, if there is an error between the second target timestamp determined by this method and the first timestamp, but the error is small and within the first timestamp tolerance window, it is considered a minor error caused by normal hardware differences. The corresponding first timestamp is then used as the new timestamp of the video frame corresponding to the second target timestamp for timing alignment. If there is no second target timestamp within the first timestamp tolerance window in the second timestamp sequence, it indicates that a large timestamp error has occurred in the second timestamp sequence due to external factors. Therefore, the position of the corresponding video frame in other original video sequences corresponding to the first timestamp is marked as a lost frame, indicating that at that timestamp, the original video sequence with the lost frame mark has experienced an abnormal state of frame loss.

[0059] For example, such as Figure 2 As shown in the figure, when the first timestamp is 1700000000018, the corresponding video frame index in the original video sequence captured by video acquisition device 4 is the lost frame identifier e', indicating that the original video sequence lost a frame at time 1700000000018; similarly, when the first timestamp is 1700000000052, the corresponding video frame index in the original video sequences captured by video acquisition device 3 and video acquisition device 4 is the lost frame identifier e', indicating that both of the above original video sequences lost a frame at time 1700000000052.

[0060] Based on the timing alignment of the second timestamp sequence according to the determined first timestamp sequence, a time tolerance window is adopted as the preset rule for timing alignment. This enables accurate timing alignment between the first and second timestamp sequences while tolerating crystal oscillator drift, coding delay difference, network jitter, or system robustness.

[0061] In some embodiments, step S30 above, "parse the video frame index matrix to generate an analysis report," specifically includes: S33: Statistically analyze the missing frame identifiers in the video frame index matrix and generate an analysis report.

[0062] As can be seen from the foregoing, the video frame index matrix contains the video frame indices of each original video sequence corresponding to the first timestamp after the first timestamp sequence and the second timestamp sequence are time-aligned, as well as the lost frame identifiers of each original video sequence corresponding to the first timestamp. The matrix unit where the lost frame identifier is located represents the specific original video sequence in which the frame was lost and the timestamp in which the frame was lost.

[0063] Based on this, by statistically analyzing the lost frame identifiers in the video frame index matrix, the frame loss situation in each original video sequence can be obtained. A corresponding analysis report can be generated based on the obtained frame loss information, including the total number of lost frames, the specific location of each lost frame, and the maximum length of consecutive lost frames. By statistically analyzing the lost frame identifiers in the video frame index matrix and automatically generating analysis reports based on the results, the efficiency of video analysis is effectively improved.

[0064] In some embodiments, prior to the aforementioned step S10, the method S100 further includes: S50: Perform functional tests on multiple video acquisition devices; if the functional test results of multiple video acquisition devices are normal, acquire multiple original video sequences acquired by multiple video acquisition devices.

[0065] Common video capture devices include network cameras (IPCs), professional broadcast / film cameras, and panoramic / industrial / vehicle cameras. Because external factors can affect the functionality of video capture devices, it is necessary to perform functional tests on them before video capture to ensure they are functioning correctly and capable of capturing normal raw video sequences. That is, before acquiring multiple raw video sequences, multiple video capture devices should first be functionally tested. If all video capture devices function normally, then multiple raw video sequences can be acquired using multiple video capture devices.

[0066] Before acquiring the original video sequence, the functionality of the video acquisition device should be tested to ensure that the device is functioning correctly. This can prevent abnormalities in the original video sequence caused by malfunctions in the acquisition device, thereby improving the accuracy of video analysis.

[0067] In some embodiments, the aforementioned step S50 specifically includes: S51: Test the network connectivity of multiple video capture devices using inspection commands and / or testing tools.

[0068] Understandably, after capturing video, the video capture device sends the captured raw video sequences to the analysis device, which then analyzes the received multiple raw video sequences using the aforementioned multi-video analysis method. During this process, the video capture device largely relies on the network for video data transmission. Therefore, before the video capture device begins capturing video, its network connectivity must be checked to ensure it functions correctly and can transmit the captured raw video sequences to the analysis device.

[0069] In some embodiments, the network connectivity of multiple video capture devices is tested using a check command. For example, first, the IP addresses and ports of the multiple video capture devices are identified, and then the network connectivity of the multiple video capture devices is tested using the ping command. If continuous replays are received after entering the ping command, it indicates that the network connection of the video capture device is normal; if after entering the ping command, only partial replays are received followed by other return content, or other return content is received directly, it indicates that the network connection of the video capture device is abnormal.

[0070] In some embodiments, a testing tool is used to test the network connectivity of multiple video capture devices. It is understood that the aforementioned command performs a functional test on the connection layer of the video capture devices, while the testing tool used in this embodiment performs a functional test on the service layer. For example, a probe request is initiated to the video stream address (such as an RTSP address) of each video capture device using a tool such as ffprobe, and the returned metadata is parsed. If information such as video encoding and resolution can be successfully parsed, it proves that the video service is normal; if a timeout or error occurs, it is determined that the service is abnormal.

[0071] In some embodiments, a retry mechanism is also provided during the aforementioned detection of multiple video detection devices. During the functional tests, if a video acquisition device exhibits an anomaly at the network or service layer, a preset number of retries will be performed to retest the video acquisition device. For example, when an analysis device initiates a probe request to the RTSP address of a video acquisition device using a detection tool, if a timeout or connection error occurs, the analysis device will not immediately determine that the camera is "service abnormal," but will trigger a "short-term delayed retry" logic. For instance, the analysis device will wait 1 second and then initiate the same probe request again until the number of retries reaches a preset number, such as 3, in which case the retry mechanism will attempt a total of 3 times. Only after 3 consecutive attempts (including the first attempt) fail will the system finally mark the video acquisition device as "service abnormal" and record it in the log.

[0072] The retry mechanism described above can effectively filter out false positives caused by momentary packet loss or brief router overload, thus improving the accuracy of the final analysis report.

[0073] S52: Extract key frames from multiple original video sequences according to a preset frequency, and test the video acquisition function of multiple video acquisition devices according to a preset functional model and key frames.

[0074] In some embodiments, the analysis device extracts keyframes from multiple original video sequences acquired during the video acquisition process of the video acquisition device at a preset frequency, and inputs the obtained keyframes into a preset functional model to detect content anomalies such as black screens, solid color screens, and frozen frames (highly similar to the previous keyframe). In some embodiments, the preset functional model is a lightweight image analysis model.

[0075] In some embodiments, the aforementioned lightweight model mainly refers to a classic image processing algorithm with extremely low computational overhead combined with a small, dedicated CNN. For example, when performing analysis, the aforementioned model first analyzes the distribution of the grayscale histogram and calculates the information entropy to quickly detect black screens, solid color screens, and overexposure in the original video sequence.

[0076] Then, by calculating the edge variance of the image, it is determined whether there are any abnormalities such as blurring or defocusing in the original video sequence. If the calculated edge variance is extremely low, it means that the image has been blurred or defocused.

[0077] Finally, by calculating frame difference, a lightweight model can be used to detect frozen frames. During the calculation of frame difference, only the previous frame needs to be buffered, and the absolute pixel difference between the current frame and the previous frame is calculated; if the absolute difference is close to 0, it is determined to be a frozen frame.

[0078] Furthermore, a very small CNN convolutional neural network model (such as MobileNetV3) can be trained to identify complex anomalies that are difficult to quantify accurately by classic algorithms, such as mosaic, noise, and partial imaging errors.

[0079] Secondly, embodiments of this application also provide a video analysis device, including: At least one processor, and A memory communicatively connected to at least one processor, wherein, The memory stores instructions that can be executed by at least one processor, such that the instructions are executed by at least one processor to enable the at least one processor to perform the method of any of the first aspects.

[0080] Thirdly, embodiments of this application also provide a video analysis system, including the video analysis device provided in the second aspect, at least two video acquisition devices, and a network device. For example... Figure 3 As shown, the video analytics device specifically includes: a health inspection module, a timestamp extraction module, a time-series alignment module, and a quantitative analysis module.

[0081] The health inspection module is used to perform pre-emptive health checks on video acquisition devices and to perform functional tests on multiple video acquisition devices. Specifically, the tests include: testing the network connectivity of multiple video acquisition devices through inspection commands and / or testing tools; and extracting keyframes from multiple original video sequences according to a preset frequency, and testing the video acquisition function of multiple video acquisition devices according to a preset functional model and keyframes.

[0082] The timestamp extraction module is used to extract multiple timestamp sequences by parsing the first encoded stream of each original video sequence, that is, to generate timestamp sequences corresponding to each original video sequence. For example, the first encoded stream is decomposed into multiple first units and the first unit containing the first data information is used as the second unit; the second unit is parsed according to a preset protocol to obtain the timestamp sequence corresponding to the first encoded stream.

[0083] Based on this, the timing alignment module arbitrarily selects one timestamp sequence as the first timestamp sequence and uses the remaining timestamp sequences as the second timestamp sequences. Then, based on the selected first timestamp sequence, the timing alignment module performs timing alignment on each of the second timestamp sequences, ultimately generating an alignment index matrix, which is the video frame index matrix. For example, based on the first timestamp sequence, the timing alignment module uses timestamps in the second timestamp sequence whose absolute difference from the first timestamp is less than a preset threshold as second target timestamps; it uses the video frame index corresponding to the second target timestamp as the matrix unit corresponding to the first timestamp to obtain the initial index matrix; and it fills the empty matrix units in the initial index matrix with lost frame identifiers to obtain the video frame index matrix.

[0084] The quantitative analysis module is used to parse the aforementioned video frame index matrix, statistically analyze the missing frame identifiers in the video frame index matrix, and generate a corresponding analysis report, which includes anomalies appearing in the video frame index matrix.

[0085] At least two video acquisition devices are communicatively connected to the video analysis device via a network device for data transmission during video analysis, such as sending the acquired raw video sequence to the video analysis device. In some embodiments, the network device is a network switch.

[0086] like Figure 4 As shown, when the starting node initiates a data acquisition task, it first checks for any anomalies in the pre-emptive health check. If no anomalies are found, the data acquisition task continues after the pre-emptive health check is completed. If an anomaly is found, the data acquisition task is terminated and an alarm is triggered. The health inspection module initiates ping commands to all video acquisition devices at the acquisition end via the network to confirm network connectivity and uses tools such as ffprobe to probe whether the video stream service of each video acquisition device is normal. If any video acquisition device has network connectivity anomalies and / or video stream service anomalies, the health inspection module retryes the above process according to a preset number of retries. If the above check still fails after the preset number of retries, the task terminates; if all video acquisition devices pass the check within the preset number of retries, the process continues.

[0087] After the aforementioned pre-health check is passed, each video acquisition device sends the acquired raw video sequence to the analysis device via the network device. The analysis system then enters the data recording and timestamp extraction stage. The timestamp extraction module extracts the timestamp sequence corresponding to each raw video sequence. Subsequently, the process enters the time alignment-generating matrix stage. In this stage, the time alignment module performs time alignment on the remaining second timestamp sequences based on the determined first timestamp sequence and through a preset threshold (tolerance window). The result of the time alignment is represented in the form of a video frame index matrix.

[0088] like Figure 2 As shown, this diagram is a simplified example of an alignment index matrix. Each row of the matrix represents a time point of the main camera, i.e., the first timestamp. Each column represents a camera, the second column represents the video acquisition device corresponding to the first timestamp sequence (main camera 1), and the remaining columns represent the video acquisition devices corresponding to each second timestamp sequence (cameras 2 / 3 / 4). The cell records the index number of the video frame corresponding to that camera at the first timestamp. When a camera cannot find a matching video frame at a certain first timestamp, its corresponding cell will be marked as a lost frame identifier, such as e'.

[0089] After the video frame index matrix is ​​generated, the process proceeds to quantitative analysis, anomaly statistics, and structured report generation. During this process, the quantitative analysis module automatically reads and analyzes the aforementioned video frame index matrix and generates the corresponding analysis report. For example, using... Figure 2 The index matrix shown is used as a video frame index matrix for analysis. The quantitative analysis module can analyze that: video acquisition device 2 dropped a frame at the first timestamp of 1700000000052; video acquisition device 4 dropped a frame at the first timestamp of 1700000000018; and video acquisition device 3 dropped a frame at the first timestamps of 1700000000035 and 1700000000052, respectively. That is, the video acquisition devices experienced two consecutive frame dropping events.

[0090] By scanning the entire matrix programmatically, the quantitative analysis module can automatically and accurately calculate key indicators such as the total number of dropped frames for each camera, the specific location of the dropped frames, and the maximum length of consecutive dropped frames. After the analysis is completed, the analysis device automatically generates a structured report, organizing all quantitative analysis results into a clear text or JSON format report.

[0091] Fourthly, embodiments of this application also provide a computer-readable storage medium, characterized in that the computer-readable storage medium stores computer-executable instructions for causing a computer device to perform the method provided in any of the first aspects.

[0092] In summary, this embodiment of the application bypasses intermediate interference factors such as network latency jitter, system clock errors, and encoding / decoding buffers by obtaining the corresponding timestamp sequence from the first encoded stream of the original video sequence, resulting in a high-precision timestamp sequence and thus achieving stable and reliable millisecond-level multi-channel synchronization. Simultaneously, it transforms the complex synchronization problem into a structured alignment index matrix, and based on this, enables automated and precise quantitative analysis of abnormal events such as frame drops and drift, improving the efficiency of video analysis.

[0093] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0094] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software and a general-purpose hardware platform, or of course, using hardware. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of this application as described above, which are not provided in detail for the sake of brevity; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for analyzing multi-channel video, characterized in that, include: Obtain the first encoded stream of multiple raw video sequences; Multiple timestamp sequences are obtained based on multiple first encoded streams; A video frame index matrix is ​​generated based on the multiple timestamp sequences; The video frame index matrix is ​​parsed to generate an analysis report.

2. The method according to claim 1, characterized in that, The step of generating a video frame index matrix based on the plurality of timestamp sequences includes: One timestamp sequence is determined from the plurality of timestamp sequences as the first timestamp sequence, and the remaining timestamp sequences are used as the second timestamp sequence; Each of the second timestamp sequences is time-aligned with the first timestamp sequence to generate the video frame index matrix.

3. The method according to claim 2, characterized in that, The step of aligning each of the second timestamp sequences with the first timestamp sequence in time to generate the video frame index matrix includes: Based on the first timestamp sequence and preset rules, determine the second target timestamp in the second timestamp sequence that is aligned with the first timestamp; The video frame index corresponding to the second target timestamp is used as the matrix unit corresponding to the first timestamp to obtain the initial index matrix; The video frame index matrix is ​​obtained by filling empty matrix cells in the initial index matrix with lost frame identifiers.

4. The method according to claim 3, characterized in that, The preset rule includes: taking the timestamps in the second timestamp sequence whose absolute difference from the first timestamp is less than a preset threshold as the second target timestamps.

5. The method according to claim 3, characterized in that, The step of parsing the video frame index matrix to generate an analysis report includes: The lost frame identifiers in the video frame index matrix are statistically analyzed to generate the analysis report.

6. The method according to claim 1, characterized in that, Prior to acquiring the first encoded stream of multiple raw video sequences, the method further includes: Perform functional testing on multiple video capture devices; If the functional test results of the multiple video acquisition devices are normal, then multiple original video sequences acquired by the multiple video acquisition devices are obtained.

7. The method according to claim 6, characterized in that, The functional testing of multiple video acquisition devices includes: The network connectivity of the multiple video capture devices is tested using inspection commands and / or testing tools.

8. The method according to claim 6, characterized in that, The functional testing of multiple video acquisition devices includes: Keyframes are extracted from the multiple original video sequences according to a preset frequency, and the video acquisition functions of the multiple video acquisition devices are tested according to a preset functional model and the keyframes.

9. The method according to any one of claims 1-8, characterized in that, The step of obtaining multiple timestamp sequences based on multiple first encoded streams includes: The first encoded stream is decomposed into multiple first units, and the first unit containing the first data information is used as the second unit; The second unit is parsed according to a preset protocol to obtain the timestamp sequence corresponding to the first encoded stream.

10. A video analysis device, characterized in that, include: At least one processor, and The memory communicatively connected to the at least one processor, wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-9.

11. A video analysis system, characterized in that, include: At least two video capture devices; Network devices, and the video analytics device as described in claim 10.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer device to perform the method as described in any one of claims 1-9.

Images