Video decoding method, video quality diagnosis method, system, device and medium

Abstract

The invention provides a video decoding method, a video quality diagnosis method, a system, equipment and a medium, and the video decoding method comprises the steps: receiving a first video of a plurality of channels, determining decoders corresponding to the channels, and setting the maximum number of frame buffers of the decoders as a preset frame buffer number; sequentially processing the first video according to the frame caching demand quantity of the first video, if the frame caching demand quantity of the first video is greater than a preset frame caching quantity, not decoding the first video, and recording channel information and the frame caching demand quantity corresponding to the first video; if the frame cache demand number of the first video is smaller than or equal to the preset frame cache number, taking the first video as a to-be-processed video, and decoding the to-be-processed video by adopting a decoder corresponding to the channel; and when a preset number resetting condition is satisfied, resetting the maximum frame buffer number of the decoder according to the recorded maximum frame buffer demand number, acquiring a second video as a to-be-processed video according to the recorded channel information, and decoding the to-be-processed video by adopting the decoder corresponding to the channel. Through adoption of the video decoding method and device, occupation of hardware resources in a video decoding process is effectively reduced.

Description

Background Art

[0002] With the construction of safe cities, front-end cameras (hereinafter referred to as IPCs) are everywhere in the streets and alleys of the city, guarding the harmony and tranquility of the city. Abnormalities often occur due to network environment, hardware failure or improper human operation, including camera screen distortion, black screen, offline, blue screen, occlusion, lens blur and other failures. It is a huge and cumbersome task to rely solely on human observation, and it is necessary to detect the operating status of each IPC in a long-term and stable manner. Therefore, a complete intelligent analysis solution is needed to realize the operation and maintenance of massive IPCs.

[0003] At present, the mainstream video quality diagnosis scheme is that the analysis system pulls the bitstream from the streaming media, decodes it and sends it to the algorithm for analysis. The algorithm gives the detection result according to the picture status. If an abnormal situation occurs, the business end notifies the user to handle it in time. There are two main specific implementation plans. The first plan is to continuously detect the access bitstream; the second plan is to perform polling access detection on the device. Each plan has obvious advantages and disadvantages.

[0004] The first solution analyzes the accessed bitstream in real time and reports any anomalies in a timely manner, so that users can discover and handle them in a timely manner. However, it consumes huge hardware resources. In the following text, 10,000 devices and Nvidia T4 cards are used as examples. A single card can concurrently process 35 channels of 25fps 1080P bitstream decoding. Therefore, 285 graphics cards are required to access all devices. Graphics cards are expensive. Obviously, this solution does not have universal practical value. The probability of IPC failure is small, so most of the detection is in invalid calculation. The limitations of this solution are quite obvious.

[0005] The second solution adopts a polling mechanism, that is, a certain number of streams are pulled from the streaming media in batches, and the next batch of devices are replaced after analyzing for a period of time. This solution can greatly reduce the hardware cost. Taking 10,000 devices as an example, if the stream of one device is continuously analyzed for 4 minutes, a T4 card server can poll all devices once a day. It can basically meet user needs, but there are also obvious problems. Because each device can only be polled once a day, it is possible that an abnormality cannot be reported in time, and the problem discovery cycle is long. At the same time, when the 35-channel decoder is created, it consumes a lot of video memory. The graphics card has a decoding core and a computing core. The decoding core can realize the hard decoding function of the video, and the computing core is often used for deep learning calculations. With this second solution, the usage rate of the video memory, decoding core and computing core is relatively high, and the hardware requirements are also high. Because the number of concurrent connections of a single graphics card is limited, the video decoding efficiency and video quality diagnosis efficiency cannot be improved, and the number of device polling cannot be increased.

[0006] Therefore, in the existing video quality diagnosis method, video decoding occupies a lot of hardware resources, has a high hardware cost, and is not efficient. Therefore, it is necessary to improve the existing video decoding method and provide a new video decoding method that can reduce the occupation of hardware resources and solve the problem in the prior art that the number of concurrent connections is limited and the video decoding efficiency and video diagnosis efficiency cannot be improved. Summary of the invention

[0007] In response to the problems in the prior art, the purpose of this application is to provide a video decoding method, video quality diagnosis method, system, device and medium, which can reduce the occupation of hardware resources during video decoding, have low hardware cost, simple and efficient deployment, and improve video diagnosis efficiency.

[0008] The present invention provides a video decoding method, comprising the following steps:

[0009] Receiving first videos of multiple channels, determining decoders corresponding to the channels, and setting a maximum number of frame buffers of the decoders to a preset number of frame buffers;

[0010] The first video is processed in sequence according to the number of frame buffer requirements of the first video, wherein if the number of frame buffer requirements of the first video is greater than the preset number of frame buffers, the first video is not decoded, and the channel information and the number of frame buffer requirements corresponding to the first video are recorded; if the number of frame buffer requirements of the first video is less than or equal to the preset number of frame buffers, the first video is used as a video to be processed, and the decoder corresponding to the channel is used to decode the video to be processed;

[0011] When the preset quantity reset condition is met, the maximum number of frame buffers of the decoder is reset according to the recorded maximum frame buffer requirement number, the second video is obtained as the video to be processed according to the recorded channel information, and the decoder corresponding to the channel is used to decode the video to be processed.

[0012] The present application adopts the video decoding method, obtains the first video of each channel, sets the maximum number of frame buffers of the decoder to a preset value, first decodes the first video with a smaller number of frame buffer requirements, and records the channel information and frame buffer requirements of the first video with a larger number of frame buffer requirements that cannot be decoded at present, and then resets the maximum number of frame buffers of the decoder to the maximum value previously recorded, obtains the second video of the remaining channel and successfully decodes it using the decoder. Therefore, the present application enables a decoder to be applicable to videos of more channels, so that multiplexing technology of multiple channels multiplexing a decoder can be realized, reducing the number of requirements for decoders, and no longer being restricted by the maximum number of decoders created by a single graphics card. In addition, when the maximum number of frame buffers of the decoder is initially set, the present application does not directly set it according to the maximum value, avoiding the decoder from occupying a large amount of video memory space from the beginning, avoiding excessive waste of hardware resources, and effectively reducing the video memory occupancy rate, the usage rate of decoding cores and computing cores. The video decoding method reduces the requirements and the number of requirements for hardware, thereby effectively reducing the hardware cost, and will not affect the decoding speed due to hardware limitations, thereby improving the decoding efficiency.

[0013] In some embodiments, determining the decoder corresponding to the channel comprises the following steps:

[0014] Acquire video parameters of the channel and determine the channel category of the channel, wherein the video parameters include video resolution and / or video bitstream format;

[0015] According to the preset correspondence between the channel category and the decoder, the decoder corresponding to the channel is determined.

[0016] In some embodiments, decoding the video to be processed using a decoder corresponding to the channel includes: inputting a key frame of the video to be processed into a corresponding decoder, and obtaining decoded data of the key frame output by the decoder; or

[0017] The video to be processed is decoded using a decoder corresponding to the channel, including: inputting video frames in the video to be processed of each channel into the corresponding decoder, wherein one or more complete image group sequences in the video to be processed of each channel are continuously input into the decoder, and then one or more complete image group sequences in the video to be processed of another channel are input.

[0018] In some embodiments, the preset quantity reset condition includes the end of the current polling round;

[0019] After obtaining the second video as the video to be processed according to the recorded channel information and decoding the video to be processed using the decoder corresponding to the channel, the following steps are also included:

[0020] The maximum number of frame buffers is reset according to the preset number of frame buffers for decoding in the next polling round.

[0021] In some embodiments, decoding the video to be processed using a decoder corresponding to the channel includes the following steps:

[0022] Determining channel information of the video to be processed;

[0023] Generate an update timestamp based on the channel information and the real timestamp of the video to be processed;

[0024] Input the video to be processed and the update timestamp into the decoder corresponding to the channel, and obtain the decoded video data and the update timestamp output by the decoder;

[0025] Determine the channel information corresponding to the decoded video data according to the update timestamp output by the decoder;

[0026] The decoded video data is assembled according to each channel to obtain decoded data corresponding to each channel.

[0027] In some embodiments, processing the first video in sequence according to the number of frame buffer requirements of the first video includes the following steps:

[0028] Adding the first video to a data queue of a decoder corresponding to the channel, wherein an arrangement priority of the first video in the data queue is set according to fault information of a device corresponding to the channel;

[0029] According to the arrangement order of the first videos in the data queue, the first videos are selected in sequence for processing.

[0030] The present application also provides a video quality diagnosis method, comprising the following steps:

[0031] Establish channels connected to multiple devices respectively and determine the corresponding relationship between devices and channels;

[0032] Polling to obtain videos of multiple channels, and using the video decoding method in each round to decode the videos of the multiple channels to obtain decoded video data;

[0033] Perform quality diagnosis on the decoded video data to determine whether a corresponding device fails.

[0034] By adopting the video quality diagnosis method, the decoded video data is obtained through the above-mentioned video decoding method, and then quality diagnosis is performed based on the decoded video data. Since the hardware cost requirements and quantity requirements are reduced in the video decoding process and the decoding efficiency is higher, the video quality diagnosis method can also reduce the hardware cost requirements and quantity requirements, achieve faster video quality diagnosis, and ensure that the video quality of more devices can be polled in the same time, meeting the use requirements of continuous monitoring of multiple devices.

[0035] The embodiment of the present application further provides a video quality diagnosis system, which is applied to the video decoding method, and the system includes:

[0036] The streaming media server is used to establish channels connected to multiple devices respectively, determine the corresponding relationship between devices and channels, and poll to obtain videos from multiple channels;

[0037] An intelligent analysis server, configured to decode the videos of the multiple channels to obtain decoded video data, perform quality diagnosis on the decoded video data, and determine whether a corresponding device fails;

[0038] The management server is used to send quality diagnosis tasks to the intelligent analysis server.

[0039] By adopting the video quality diagnosis system provided by the present application, the functions of each functional module are implemented by implementing the steps of the video quality diagnosis method, thereby obtaining the beneficial effects of the above-mentioned video decoding method.

[0040] The embodiment of the present application also provides a video quality diagnosis device, including:

[0041] processor;

[0042] a memory storing executable instructions of the processor;

[0043] The processor is configured to perform the steps of the video quality diagnosis method by executing the executable instructions.

[0044] By adopting the video quality diagnosis device provided in the present application, the processor executes the video quality diagnosis method when executing the executable instructions, thereby obtaining the beneficial effects of the above-mentioned video decoding method.

[0045] The embodiment of the present application also provides a computer-readable storage medium for storing a program, and when the program is executed by a processor, the steps of the video quality diagnosis method are implemented.

[0046] By adopting the computer-readable storage medium provided by the present application, the program stored therein implements the steps of the video quality diagnosis method when being executed, thereby obtaining the beneficial effects of the above-mentioned video decoding method. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings.

[0048] Figure 1 is a flowchart of a video decoding method according to an embodiment of the present application;

[0049] Figure 2 is a flowchart of video quality diagnosis according to an embodiment of the present application;

[0050] Figure 3 is a schematic diagram of a video quality diagnosis system according to an embodiment of the present application;

[0051] Figure 4 is a processing topology diagram of a multi-channel multiplexing decoder according to an embodiment of the present application;

[0052] Figure 5 It is a schematic diagram of key frame data fusion after multi-channel code stream decoding according to an embodiment of the present application;

[0053] Figure 6 is a structural diagram of a video quality diagnosis device according to an embodiment of the present application;

[0054] Figure 7 It is a schematic diagram of the structure of a computer storage medium according to an embodiment of the present application. DETAILED DESCRIPTION

[0055] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be comprehensive and complete and the concepts of the example embodiments will be fully conveyed to those skilled in the art. The same reference numerals in the figures represent the same or similar structures, and thus their repeated description will be omitted. Although "first" or "second" etc. are used in this specification to represent certain features, they are only used to represent the function and are not used as a limitation on the number and importance of specific features.

[0056] like Figure 1 As shown, in one embodiment, the present application provides a video decoding method, comprising the following steps:

[0057] S210: receiving first videos of multiple channels, determining decoders corresponding to the channels, and setting the maximum number of frame buffers of the decoders to a preset number of frame buffers;

[0058] Here, each channel corresponds to a video acquisition device, such as a front-end camera of a monitoring system (but the present application is not limited thereto, and may also be other application scenarios and other types of devices); the first video of each channel includes video stream data acquired in real time by the video acquisition device corresponding to each channel;

[0059] The maximum number of frame buffers of the decoder corresponds to the number of surfaces required for buffering when decoding an image. The preset number of frame buffers is a preset value, such as an empirical value set by a staff member based on experience (for example, 10, but the present application is not limited thereto), which can meet the decoding requirements of most images. When the number of surfaces required for buffering when decoding an image in a video input to the decoder (the number of frame buffer requirements) is greater than the maximum number of frame buffers of the decoder, the decoder cannot perform decoding normally. The specific value of the preset number of frame buffers here can be set and adjusted according to actual conditions, and is not limited here. For example, the number of frame buffer requirements corresponding to videos of multiple channels within a period of time is counted, and the frame buffer requirement value of videos that can cover 80% (or other ranges, such as 85%, 90%, etc.) is selected (that is, the number of frame buffer requirements of 80% of the videos is less than or equal to this value) as the preset number of frame buffers.

[0060] S220: Processing the first video in sequence according to the number of frame buffer requirements of the first video, wherein if the number of frame buffer requirements of the first video is greater than the preset number of frame buffers, not decoding the first video, and recording the channel information and the number of frame buffer requirements corresponding to the first video; if the number of frame buffer requirements of the first video is less than or equal to the preset number of frame buffers, taking the first video as a to-be-processed video, and using the decoder corresponding to the channel to decode the to-be-processed video;

[0061] Different videos will affect the resource usage when creating a decoder due to factors such as resolution, encoding complexity, bit rate, and video content. Therefore, even videos with the same resolution and encoding format will have inconsistent decoding resource consumption when creating a decoder. Here, the decoder resource usage is uniformly referred to as the maximum number of frame buffers of the decoder, and the video's demand for resources is uniformly referred to as the frame buffer requirement number.

[0062] Therefore, when the number of frame buffer requirements of the first video is less than or equal to the maximum number of frame buffers set by the current decoder (currently set to the preset number of frame buffers), the decoder can decode the video normally, and for the first video that the decoder cannot decode (the number of frame buffer requirements is greater than the maximum number of frame buffers of the current decoder), it is not decoded at present, but the channel information and the number of frame buffer requirements corresponding to the first video are recorded, and the channel information includes, for example, information that can identify the specific channel, such as a channel number or a device number;

[0063] S230: When a preset quantity reset condition is met, the maximum number of frame buffers of the decoder is reset according to the recorded maximum number of frame buffer requirements, a second video is obtained as a to-be-processed video according to the recorded channel information, and the to-be-processed video is decoded using a decoder corresponding to the channel;

[0064] The maximum frame buffer requirement quantity recorded here is the maximum value of all frame buffer requirement quantities that were not successfully decoded recorded in step S200. After resetting the maximum frame buffer quantity of the decoder based on the maximum value, the video of the channel that could not be decoded before can also be successfully decoded using the decoder.

[0065] In the existing video decoding method, when creating a decoder, a decoder is created for each channel, and the maximum number of frame buffers of the decoder is set according to the number of frame buffers required for the video of each channel, so that the decoder is only applicable to the video decoding of the channel. When it is applicable to multi-channel video decoding scenarios, multiple decoders need to be created. When the number of decoders supported by the graphics card reaches the maximum value, the videos of other channels cannot be decoded quickly, or the hardware cost needs to be increased to meet the usage requirements. Therefore, the existing video decoding method requires a high hardware cost, and the video decoding efficiency is not high.

[0066] The present application adopts the video decoding method, obtains the first video of each channel by adopting step S210, sets the maximum number of frame buffers of the decoder to a preset value, and first decodes the first video with a smaller number of frame buffer requirements by step S220, and records the channel information and frame buffer requirements of the first video with a larger number of frame buffer requirements that cannot be decoded at present, and then resets the maximum number of frame buffers of the decoder to the maximum value previously recorded by step S230, obtains the second video of the remaining channels and successfully decodes it by adopting the decoder. Therefore, the present application enables a decoder to be applicable to videos of more channels, so that a multiplexing technology of multiple channels multiplexing a decoder can be realized, reducing the number of decoders required, and no longer being limited by the maximum number of decoders created by a single graphics card. In addition, when the maximum number of frame buffers of the decoder is initially set in step S210, the present application does not directly set it according to the maximum value, which avoids the decoder from occupying a large amount of video memory space from the beginning, avoids excessive waste of hardware resources, and effectively reduces the video memory occupancy rate, the usage rate of decoding cores and computing cores. The video decoding method reduces the requirements and quantity of hardware, thereby effectively reducing the hardware cost, and will not affect the decoding speed due to hardware limitations, thereby improving the decoding efficiency.

[0067] The video decoding method is particularly suitable for scenes where the real-time requirements for video analysis are not high. After recording the channel information corresponding to the first video that cannot be decoded at present in step S220, when the preset quantity reset condition is met, after resetting the maximum number of frame buffers of the decoder, the second video of the channel is obtained again according to the channel information, and then the second video is decoded. For example, the decoded video obtained by the video decoding method can be used for subsequent video quality diagnosis, such as diagnosing the video quality of the front-end camera of the monitoring system to determine whether the front-end camera has a fault. This scenario does not require particularly high real-time and synchronization, as long as the fault can be found within a certain time range. For the same channel, other video streams between the first video and the second video are discarded and not processed. Since the fault is continuous when the screen fails, such as a black screen, it is difficult to return to normal without manual intervention, and strong real-time performance is not required. It only needs to be able to find the fault and report it during the next polling detection. The present application can improve the timeliness of equipment fault discovery by increasing the polling cycle, but it does not necessarily require completely real-time analysis. The video decoding method of the present application, when applied to a video diagnosis scenario, can well meet the needs of polling monitoring of equipment, and compared with the prior art, the polling cycle is greatly shortened, and equipment failures can be discovered more quickly.

[0068] However, the present application is not limited to the application in the video quality diagnosis scenario. In other implementations, the decoded video obtained by the video decoding method can also be used for subsequent other types of video analysis, such as target recognition of images, etc., all of which fall within the protection scope of the present application. The following takes the application of the video decoding method to video quality diagnosis as an example to specifically describe its implementation method, but it is not intended to limit the protection scope of the present application.

[0069] like Figure 2 As shown, the embodiment of the present application also provides a video quality diagnosis method, comprising the following steps:

[0070] S100: establishing channels connected to multiple devices respectively, and determining the corresponding relationship between the devices and the channels;

[0071] The device here refers to the video acquisition device being monitored, such as the front-end camera of the monitoring system;

[0072] S200: polling to obtain videos of multiple channels, and using the video decoding method in each round to decode the videos of multiple channels to obtain decoded video data;

[0073] The decoded video may be different according to the comparison result between the required number of frame buffers of the first video and the preset number of frame buffers, and may be the first video obtained in step S210 or the second video obtained in step S220;

[0074] S300: Perform quality diagnosis on the decoded video data to determine whether the corresponding device has a fault. The quality diagnosis of the decoded video data here can be performed using existing diagnostic methods, which are not limited here. For example, by analyzing the pixel values ​​of the image frame, it is determined whether the device has a flowery screen, a black screen, offline, blocked, blurred lens, and other faults, thereby determining whether a device fault has occurred. This analysis can be implemented by automatic detection and calculation using some script programs, or by using a machine learning model, thereby achieving automation of the entire process from video acquisition, video decoding to video quality diagnosis, greatly improving the efficiency of device fault detection.

[0075] By adopting the video quality diagnosis method, the decoded video data is obtained through the above-mentioned video decoding method, and then quality diagnosis is performed based on the decoded video data. Since the hardware cost requirements and quantity requirements are reduced in the video decoding process and the decoding efficiency is higher, the video quality diagnosis method can also reduce the hardware cost requirements and quantity requirements, achieve faster video quality diagnosis, and ensure that the video quality of more devices can be polled in the same time, meeting the use requirements of continuous monitoring of multiple devices.

[0076] like Figure 3 As shown, the embodiment of the present application also provides a video quality diagnosis system, which is applied to a video decoding method, and the system includes:

[0077] The streaming media server M100 is used to establish channels connected to multiple devices, determine the corresponding relationship between devices and channels, and poll to obtain videos from multiple channels; the streaming media server is used to push video code streams and transmit audio and video to the client through a streaming protocol (RTSP, RTMP, etc.);

[0078] The intelligent analysis server M200 is used to decode videos of multiple channels, obtain decoded video data, perform quality diagnosis on the decoded video data, and determine whether the corresponding device has a fault; wherein the intelligent analysis server M200 uses the above-mentioned video decoding method to decode the received video, and determines whether the device has a fault based on the results of the video quality diagnosis.

[0079] The management server M300 is used to send quality diagnosis tasks to the intelligent analysis server.

[0080] The management server is used to uniformly manage each intelligent analysis server M200 through signaling keep-alive, task distribution, result aggregation, etc. The intelligent analysis server M200 uniformly registers with the management server M300 as a node, and receives the quality diagnosis task quality issued by the management server M300. The instructions include operations such as adding and deleting quality diagnosis tasks. The task parameters of the quality diagnosis task include at least the bitstream address (channel address). After the quality diagnosis task is issued to the intelligent analysis server M200, the intelligent analysis server M200 obtains the video to be processed from the streaming media server M100 according to the bitstream address in the task parameters, and decodes and diagnoses the quality of the video bitstream, identifies whether the picture is abnormal, and thus determines whether the equipment has a fault. Through the management of the intelligent analysis server M200 nodes by the management server M300, the system is flexible and scalable, and can meet the needs of different stages by adding and deleting intelligent analysis service instances.

[0081] By adopting the video quality diagnosis system provided by the present application, the functions of each functional module are implemented by implementing the steps of the video quality diagnosis method, thereby obtaining the beneficial effects of the above-mentioned video decoding method.

[0082] In this embodiment, step S100 includes the following steps:

[0083] First, a decapsulation thread pool is created according to the maximum access capacity of the video quality diagnosis system. When the video streams Stream1, Stream2, ..., StreamN of multiple devices are connected to the system, an idle thread is first obtained from the decapsulation thread pool. After successful acquisition, the device is initialized, including establishing a connection channel with the device, determining the correspondence between the device and the channel, parsing the streaming media protocol of the device, parsing the sequence parameter set SPS and the image parameter set PPS of the device, etc. After successful initialization, the video resolution information and bitstream format are obtained; if initialization fails, the decapsulation thread is returned and a bitstream connection abnormality alarm is reported.

[0084] In this embodiment, the step S210: determining the decoder corresponding to the channel includes the following steps:

[0085] Obtain video parameters of the channel and determine the channel category of the channel. The video parameters include video resolution and / or video bitstream format;

[0086] According to the preset correspondence between the channel category and the decoder, the decoder corresponding to the channel is determined.

[0087] For example, the correspondence between the channel and the decoder is determined according to the video resolution and the video bitstream format. Each decoder is only for the video with the specified resolution and the specified bitstream format. For example, for the video bitstream with 1080p resolution h264 format, it corresponds to a decoder, and each decoder here refers to a separate decoding thread. If the channels of multiple devices belong to the same category, only one decoding thread needs to be established. If there are multiple different channel categories, multiple decoding threads are established accordingly, and independent decoders are created for each decoding thread. Taking the NVIDIA decoder as an example, when creating it, it is necessary to set several key parameters such as the bitstream format CodecType, the decoding output type OutputFormat, and the number of decoding surfaces ulNumDecodeSurface. The first two parameters of the same type of bitstream are consistent. ulNumDecodeSurface represents the maximum number of decoded frames used to cache the decoded frames inside the decoder, that is, the maximum number of frame buffers of the decoder, which is mainly used for the frame buffer management inside the decoder, affecting the efficiency of the decoding pipeline. If the value is set too small, it may cause decoding failure, or the decoder pipeline is blocked and affects the efficiency. If it is set too large, it will cause too much video memory to be occupied. The type of decoder here is only an example, and this application can be applied to various different decoders.

[0088] In the step S220, before decoding the video, it is first determined whether the number of frame buffer requirements of the first video is greater than the preset number of frame buffers. Specifically, the video code stream will have large differences during decoding due to the influence of factors such as compression efficiency, bit rate, frame rate, and complexity of video content. After the code stream is decapsulated to obtain sps, pps, and video frame data, the decoder will comprehensively calculate the number of frame buffer requirements required to meet the decoding of the channel. Different code streams often calculate different numbers of frame buffer requirements. After each code stream is input into the decoder, the decoder first finds the code stream header of the code stream, then traverses a certain number of bytes backward and parses them, thereby obtaining the number of frame buffer requirements for the code stream and outputting it. After obtaining the number of frame buffer requirements of the code stream through the intelligent analysis server, it is compared with the preset number of frame buffers to determine whether the decoder can decode the code stream. If it can be decoded (the required number of frame buffers is less than or equal to the preset number of frame buffers), the code stream will continue to be input into the decoder for decoding. If it cannot be decoded (the required number of frame buffers is greater than the preset number of frame buffers), the code stream will not be input into the decoder for decoding, and the corresponding channel information and the required number of frame buffers will be recorded.

[0089] In this embodiment, a polling method is used to perform video quality diagnosis on multiple devices. The preset quantity reset condition includes the end of the current polling round, that is, the videos whose frame buffer requirement quantity is less than or equal to the maximum number of frame buffers in the current polling round have been decoded. In the step S230, after obtaining the second video as the video to be processed according to the recorded channel information and decoding the video to be processed using the decoder corresponding to the channel, it also includes: resetting the maximum number of frame buffers according to the preset number of frame buffers for decoding in the next polling round. It is defined here that at the beginning of a polling round, the video code stream obtained for each channel is the first video, and the frame buffer requirement number of the first video is detected in sequence and compared with the preset frame buffer number. If the frame buffer requirement number of the first video is less than or equal to the preset frame buffer number, the first video is directly decoded. If the frame buffer requirement number of the first video is greater than the preset frame buffer number, the first video is not decoded. After the first videos in the subsequent polling rounds have all been detected and compared with the frame buffer requirement number, and all first videos with frame buffer requirements less than or equal to the preset frame buffer number are decoded, the video of the channel that has not been decoded is obtained according to the recorded channel information (the channel information corresponding to the first video that has not been decoded before), which is defined as the second video. In the present application, the video to be processed refers to the video that is determined to be decoded by the decoder and input into the decoder for actual decoding. Depending on the situation, it may be the first video or the second video.

[0090] When the video codec adopts the H264 standard, it has a high compression ratio while taking into account high-quality images, and is suitable for network transmission. The video code stream is composed of a series of continuous images. An image is also called a frame. The H264 code stream is divided into I frames, P frames, and B frames according to the encoding type. I frames are also called key frames or intra-frame coded frames. When decoding, only the data of the current frame needs to be processed to complete image restoration, while P frames and B frames need to rely on the previous frames when decoding to complete image restoration. When encoding, the time interval of the key frames can be controlled by setting GOP parameters (GOP length, I frame interval). The size of the I frame interval has an important influence on the quality of the video, compression efficiency, and the amount of calculation during decoding. For example, in the field of monitoring, a key frame is usually set to appear every 3 seconds or 5 seconds, but the present application is not limited to this. In this embodiment, considering the timeliness of video decoding when multiplexing a decoder, in order to improve the efficiency of video decoding, only the key frames in each video are decoded. Therefore, in the steps S220 and S230, the decoder corresponding to the channel is used to decode the video to be processed, including: inputting the key frame of the video to be processed into the decoder corresponding to the video, and obtaining the decoded data of the key frame output by the decoder. That is, after the decapsulation thread obtains the video code stream data, only the video key frame is retained, and the audio frame, video non-key frame, etc. are discarded, and the key frame data of different channels are stored in the queues P1, P2, ..., Pn of the video frames to be processed. The key frame data mainly includes the code stream data packet data, the data packet size size, the timestamp pts and the flag information of the data packet. By only decoding the key frame data, the efficiency of decoding the video data of each channel is improved. When multiple channels reuse a decoder, the decoder can decode the video data of more channels in the same time, improve the polling speed of the device, and help to find the fault of the device faster. In addition, when the present application is applied to the video quality diagnosis scenario, it only needs to analyze and diagnose the key frame to meet the need to determine whether the device is faulty. The present application reduces the input of non-key frames, greatly reduces the amount of invalid calculations, and reduces power consumption.

[0091] In another alternative embodiment, in the steps S220 and S230, the decoder corresponding to the channel is used to decode the video to be processed, including: inputting the video frames in the video to be processed of each channel into the corresponding decoder, wherein one or more complete image group sequences in the video to be processed of each channel are continuously input into the decoder, and then one or more complete image group sequences in the video to be processed of another channel are input. That is, in this embodiment, not only the key frames are decoded, but all the key frames and non-key frames in the image group are decoded, and complete decoded video data can be obtained, which is suitable for application scenarios that require complete video information analysis. In order to ensure that the non-key frames in each image group can be successfully decoded, when input into the decoder, the video to be processed of each channel is input as one or more complete image group sequences. After the input is completed, the video to be processed of other channels is allowed to enter the decoder, so as to ensure that when decoding the video to be processed of one channel, video frames of other channels will not be inserted in the middle of an image group to affect the normal decoding of non-key frames. For example, there are three channels T1, T2, and T3. After a complete image group sequence [IP P PP…] of channel T1 is input into the decoder, a complete image group sequence [IP PPP…] of channel T2 is input into the decoder, and then, a complete image group sequence [IP PPP…] of channel T3 is input into the decoder.

[0092] Since multiple channels of video are multiplexed with the same decoder, and the decoder itself uses asynchronous processing logic, when multiple stream data is input to the decoder, the decoder cannot determine which channel a frame of decoded data belongs to when outputting it, and it is difficult to merge the decoded video frames according to the channels. Therefore, in this embodiment, the timestamp is processed, and the shift and logical combination algorithm in the bit operation is used to integrate the channel information into the timestamp pts, so as to realize the transmission of the channel information before and after decoding.

[0093] Specifically, in the steps S220 and S230, the decoder corresponding to the channel is used to decode the video to be processed, including the following steps:

[0094] Determine the channel information of the video to be processed;

[0095] Generate an updated timestamp based on the channel information and the real timestamp of the video to be processed;

[0096] The timestamp is, for example, of uint64_t type. The processing logic for generating the update timestamp new_pts based on the real timestamp old_pts is as follows:

[0097] new_pts=((uint64_t)chan_id<<32)+(old_pts&0xFFFFFFFF)

[0098] Among them, chan_id is the channel information, for example, the channel number identifier, each code stream corresponds to a unique channel number, which is an integer type;

[0099] The video to be processed and the update timestamp are input into the decoder corresponding to the channel, and the decoded video data and the update timestamp output by the decoder are obtained; after the update timestamp is input into the decoder, it is transparently transmitted out through the decoder;

[0100] Among them, take decoding only the key frame as an example, Figure 4 and Figure 5 As shown, a key frame data merging thread is created, and according to the device code stream priority, the key frame data is obtained from the queues P1, P2, ..., Pn of each channel in a phased polling manner according to the first-in-first-out principle, and the data is merged into the corresponding M1, M2, ..., Mm queues; each key frame in the fused queue is atomic and can be decoded independently without referring to other data. Therefore, no matter how the data in the fused queue is arranged, it can be decoded normally, that is, it is decoupled from the specific channel number data when multiplexing the decoder;

[0101] Determine the channel information corresponding to the decoded video data according to the update timestamp output by the decoder;

[0102] The parsing threads T1, T2, ..., Tm corresponding to each decoder obtain data from the corresponding fusion queues M1, M2, ..., Mm respectively, and obtain the YUV data and its transparently transmitted new_pts timestamp after decoding. According to the inverse operation of the above processing logic, the real channel number and timestamp can be obtained, thereby realizing the separation of the decoded data of each channel; the processing logic of the inverse operation is as follows:

[0103] chan_id=(new_pts>>32)&0xFFFFFFFF;

[0104] old_pts=new_pts&0xFFFFFFFF;

[0105] Assemble the decoded video data according to each channel to obtain the decoded data corresponding to each channel;

[0106] like Figure 4 As shown, the decoded video data (yuv data) obtained by decoding is assembled according to information such as the channel number and the real time stamp, and stored in the algorithm analysis queue A for video quality diagnosis in the subsequent step S300.

[0107] In this embodiment, in step S200, in each polling round, the first video is processed in turn according to the number of frame buffer requirements of the first video, including: adding the first video to the data queue of the decoder corresponding to the channel, wherein the arrangement priority of the first video in the data queue is set according to the fault information of the device corresponding to the channel; and selecting the first video for processing in turn according to the arrangement order of the first video in the data queue.

[0108] When all devices are initially connected, the priority of all channels is set to low priority to ensure that the weight of each video code stream remains consistent when being processed. When obtaining video frame data (only key frame data, or obtaining video frame data in units of one or more image group sequences) from the queues P1, P2, ..., Pn of each channel according to the first-in-first-out principle, and merging the video frame data into the corresponding M1, M2, ..., Mm queues, the fusion is performed according to the priority of each channel. When the priority of all channels is the same, the video frame data of each channel is arranged in chronological order. When sorting, for an embodiment in which only key frames are decoded, the key frames of each channel are sorted as sorting elements respectively. For an embodiment in which all video frames are decoded, one or more image group sequences of each channel are used as a whole sorting element. When it is determined through video quality diagnosis that a device fails, the arrangement priority of the channel corresponding to the device is increased, for example, it is set to medium priority. When merging into the M queue, the video frame data in the queue of the channel with high priority is first obtained. After the device with medium priority returns to normal, it is determined that the device is working normally according to the video quality result, and the channel priority of the device is reset to low priority.

[0109] In the step S300, the decoded video data is diagnosed for quality and it is determined whether the corresponding device has a fault. If a device is identified to have a fault, the device is notified to the user side via an http message to remind the user to handle the fault in time. If there is no fault, no processing is required.

[0110] After the processing of each concurrent code stream reaches the set time, the decapsulation stops, and each thread is returned to the thread pool, the thread state is changed to idle, and the video quality diagnosis of the current polling round ends. When the time set for the next polling round arrives, through the above steps S100 to S300, the access device performs a new round of video decoding and quality diagnosis analysis, and the cycle repeats.

[0111] In summary, the present application only needs to create a small number of decoders to handle hundreds of concurrent access of code streams. For multiple videos of the same category, a decoder can be reused, which greatly reduces the occupation of the decoder to create video memory. And the invention can automatically update and improve the compatibility of the decoders in the current system, and complete the decoding compatibility of all access code streams. By adopting the video decoding method of the present application, under ideal network bandwidth conditions, 500 1080P resolution h264 format code streams only need to create a single decoder. According to 1 key frame per second, a single decoder can achieve real-time decoding. Compared with the prior art solution, the performance of a single graphics card is improved by 14 times. Taking the polling detection of 10,000 devices as an example, only a single graphics card is needed, an intelligent analysis service is deployed, and polling is performed 20 times in batches to complete all detections. The hardware cost is low and the deployment is simple and efficient. Therefore, the polling frequency of each device is high, and the user can be notified in a timely manner when a failure occurs. The intelligent analysis server can save a lot of video memory, lower the threshold for graphics card configuration, and thus reduce hardware costs; for example, the minimum configuration in the prior art requires 16G video memory, but after adopting the method of this application, 8G video memory can meet the requirements, which can reduce hardware procurement costs when selecting graphics cards.

[0112] The following is a specific example to illustrate the technical effect that can be obtained by using the video quality diagnosis method of the present application.

[0113] Taking the T4 graphics card as an example, based on a 10G network bandwidth environment, the streaming media supports 1,000 concurrent channels. A single graphics card is used to access 500 1080P streams, and the key frame interval of each stream is 5 seconds. The following takes decoding of only key frames as an example.

[0114] At 0:00, connect the first batch of 500 devices (including device a), take real-time flow analysis for 5 minutes, and report any fault alarms. After 5 minutes, stop the current 500 devices and replace them with the next batch of 500 devices; poll in sequence.

[0115] Therefore, within one day, device a performs quality diagnosis and analysis for about 5 minutes every 2 hours; if device a fails at 5 o'clock, the failure will be reported at 6 o'clock, and the maximum time error does not exceed 2 hours. Therefore, this video quality diagnosis method can reduce the requirements for hardware costs and quantity requirements, achieve faster video quality diagnosis, ensure that the video quality of more devices can be polled in the same time, and meet the use requirements of continuous monitoring of multiple devices.

[0116] An embodiment of the present application also provides a video quality diagnosis device, comprising a processor; a memory, in which executable instructions of the processor are stored; wherein the processor is configured to perform the steps of the video quality diagnosis method by executing the executable instructions.

[0117] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0118] Refer to the following Figure 6 The electronic device 600 according to this embodiment of the present application is described. Figure 6 The electronic device 600 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0119] like Figure 6 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0120] The storage unit stores a program code, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present application described in the above-mentioned video decoding method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 2 Follow the steps shown in .

[0121] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0122] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0123] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0124] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0125] By adopting the video quality diagnosis device provided in the present application, the processor executes the video quality diagnosis method when executing the executable instructions, thereby obtaining the beneficial effects of the above-mentioned video decoding method.

[0126] The embodiment of the present application also provides a computer-readable storage medium for storing a program, and when the program is executed by a processor, the steps of the video quality diagnosis method are implemented. In some possible implementations, various aspects of the present application can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to perform the steps of various exemplary implementations of the present application described in the above-mentioned video decoding method section of this specification.

[0127] refer to Figure 7 As shown, a program product 800 for implementing the above method according to an embodiment of the present application is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0128] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0129] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0130] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or cluster. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0131] By adopting the computer-readable storage medium provided by the present application, the program stored therein implements the steps of the video quality diagnosis method when being executed, thereby obtaining the beneficial effects of the above-mentioned video decoding method.

[0132] The above content is a further detailed description of the present application in combination with specific preferred implementation methods, and it cannot be determined that the specific implementation of the present application is limited to these descriptions. For ordinary technicians in the technical field to which the present application belongs, several simple deductions or substitutions can be made without departing from the concept of the present application, which should be deemed to fall within the scope of protection of the present application.

Claims

1. A video decoding method, characterized in that: The steps include: Receiving first videos of multiple channels, determining decoders corresponding to the channels, and setting a maximum number of frame buffers of the decoders to a preset number of frame buffers; The first video is processed in sequence according to the frame buffer requirement quantity of the first video, wherein if the frame buffer requirement quantity of the first video is greater than the preset frame buffer quantity, the first video is not decoded, and the channel information and the frame buffer requirement quantity corresponding to the first video are recorded; if the frame buffer requirement quantity of the first video is less than or equal to the preset frame buffer quantity, the first video is used as a video to be processed, and the decoder corresponding to the channel is used to decode the video to be processed; When the preset quantity reset condition is met, the maximum number of frame buffers of the decoder is reset according to the recorded maximum frame buffer requirement number, the second video is obtained as the video to be processed according to the recorded channel information, and the decoder corresponding to the channel is used to decode the video to be processed.

2. The video decoding method according to claim 1, characterized in that: Determining the decoder corresponding to the channel comprises the following steps: Acquire video parameters of the channel and determine the channel category of the channel, wherein the video parameters include video resolution and / or video bitstream format; According to the preset correspondence between the channel category and the decoder, the decoder corresponding to the channel is determined.

3. The video decoding method according to claim 1, characterized in that: Decoding the video to be processed using the decoder corresponding to the channel includes: inputting the key frame of the video to be processed into the corresponding decoder, and obtaining decoded data of the key frame output by the decoder; or, The video to be processed is decoded using a decoder corresponding to the channel, including: inputting video frames in the video to be processed of each channel into the corresponding decoder, wherein one or more complete image group sequences in the video to be processed of each channel are continuously input into the decoder, and then one or more complete image group sequences in the video to be processed of another channel are input.

4. The video decoding method according to claim 1, characterized in that: The preset quantity reset condition includes the end of the current polling round; After obtaining the second video as the video to be processed according to the recorded channel information and decoding the video to be processed using the decoder corresponding to the channel, the following steps are also included: The maximum number of frame buffers is reset according to the preset number of frame buffers for decoding in the next polling round.

5. The video decoding method according to claim 1, characterized in that: Decoding the video to be processed using a decoder corresponding to the channel includes the following steps: Determining channel information of the video to be processed; Generate an update timestamp based on the channel information and the real timestamp of the video to be processed; Input the video to be processed and the update timestamp into the decoder corresponding to the channel, and obtain the decoded video data and the update timestamp output by the decoder; Determine the channel information corresponding to the decoded video data according to the update timestamp output by the decoder; The decoded video data is assembled according to each channel to obtain decoded data corresponding to each channel.

6. The video decoding method according to claim 1, characterized in that: The first video is processed sequentially according to the number of frame buffer requirements of the first video, comprising the following steps: Adding the first video to a data queue of a decoder corresponding to the channel, wherein an arrangement priority of the first video in the data queue is set according to fault information of a device corresponding to the channel; According to the arrangement order of the first videos in the data queue, the first videos are selected in sequence for processing.

7. A video quality diagnosis method, characterized in that: The steps include: Establish channels connected to multiple devices respectively and determine the corresponding relationship between devices and channels; Polling to obtain videos of multiple channels, and in each round, using the video decoding method according to any one of claims 1 to 6 to decode the videos of the multiple channels to obtain decoded video data; Perform quality diagnosis on the decoded video data to determine whether a corresponding device fails.

8. A video quality diagnosis system, characterized in that: Applied to the video decoding method according to claim 7, the system comprises: The streaming media server is used to establish channels connected to multiple devices respectively, determine the corresponding relationship between devices and channels, and poll to obtain videos from multiple channels; An intelligent analysis server, configured to decode the videos of the multiple channels to obtain decoded video data, perform quality diagnosis on the decoded video data, and determine whether a corresponding device fails; The management server is used to send quality diagnosis tasks to the intelligent analysis server.

9. A video quality diagnostic device, characterized in that: include: processor; a memory storing executable instructions of the processor; The processor is configured to perform the steps of the video quality diagnosis method of claim 7 by executing the executable instructions.

10. A computer-readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps of the video quality diagnosis method according to claim 7 are implemented.

Images