Video stream processing method and device, computer equipment and readable storage medium

Abstract

The invention relates to a video stream processing method and device, computer equipment and a readable storage medium. The method comprises the following steps: under the condition that a streaming media server establishes communication connection with a camera, analyzing a video packet in a video stream transmitted by the camera to obtain video stream information; determining an abnormal identifier for the video stream according to the video stream information, and determining a processing measure corresponding to the video stream according to the abnormal identifier; and executing the processing measures to obtain a processed video stream, and analyzing the processed video stream to play the processed video stream. By adopting the method, the abnormity can be analyzed and the corresponding processing measure can be determined in the process of analyzing the video stream, so that the video can be successfully played after the processing measure is executed, the condition of video stream analysis failure is avoided, and the reliability of video transmission is improved.

Description

Technical Field

[0001] The present application relates to the field of video surveillance technology, and in particular to a video streaming method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] In the field of video surveillance, cameras capture video images in real time, and streaming media servers transmit the images collected by cameras in real time, which can achieve efficient and reliable video transmission and meet the video transmission needs in various application scenarios.

[0003] In traditional technology, as the types and numbers of cameras increase, devices produced by different manufacturers have differences in implementing video transmission standards, which may lead to various problems when parsing the data stream transmitted by the camera, resulting in failure of video stream parsing, affecting the reliability of video transmission, and not conducive to improving the reliability of video transmission. Summary of the invention

[0004] Based on this, it is necessary to provide a video stream processing method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve the reliability of video transmission in response to the above technical problems.

[0005] In a first aspect, the present application provides a video stream processing method, which is applied to a streaming media server, comprising:

[0006] When the streaming media server establishes a communication connection with the camera, parsing video packets in the video stream transmitted by the camera to obtain video stream information;

[0007] Determine an abnormality mark for the video stream according to the video stream information, and determine a processing measure corresponding to the video stream according to the abnormality mark;

[0008] The processing measures are executed to obtain a processed video stream, and the processed video stream is parsed to play the processed video stream.

[0009] In one embodiment, determining an abnormality flag for the video stream according to the video stream information includes:

[0010] Determining first video type information according to program mapping stream information in the video stream information;

[0011] In the case where the first video type information is inconsistent with the video type of the video data in the video packet, a first exception flag is generated for the video stream; the first exception flag indicates that the video type does not match.

[0012] In one embodiment, determining an abnormality flag for the video stream according to the video stream information includes:

[0013] Determine a video timestamp sequence according to the video element stream packets in the video stream information;

[0014] A second abnormality flag for the video stream is generated according to the difference and / or size relationship between the timestamps in the video timestamp sequence; the second abnormality flag indicates that the video timestamp is abnormal.

[0015] In one embodiment, determining an abnormality flag for the video stream according to the video stream information includes:

[0016] Determine a video timestamp sequence according to the video element stream packets in the video stream information;

[0017] Determine an audio timestamp sequence according to the audio element stream packets in the video stream information;

[0018] A third abnormality flag for the video stream is generated according to the matching relationship between the video timestamp sequence and the audio timestamp sequence; the third abnormality flag indicates that the video timestamp does not match the audio timestamp.

[0019] In one embodiment, determining an abnormality flag for the video stream according to the video stream information includes:

[0020] Determine Slice encoding according to the video element stream packet in the video stream information;

[0021] When the number of Slice codes is greater than a preset threshold, a fourth abnormality flag is generated for the video stream; the fourth abnormality flag indicates that Slice codes are abnormal.

[0022] In one embodiment, determining an abnormality flag for the video stream according to the video stream information includes:

[0023] Determine data packet size information according to packet header information in the video stream information;

[0024] When the data packet size information is larger than a preset data packet threshold size, a fifth abnormality flag is generated for the video stream; the fifth abnormality flag indicates that the data packet is too large.

[0025] In a second aspect, the present application further provides a video stream processing device, which is applied to a streaming media server, comprising:

[0026] A video stream parsing module, used for parsing video packets in the video stream transmitted by the camera to obtain video stream information when the streaming media server establishes a communication connection with the camera;

[0027] An abnormality analysis module, used to determine an abnormality mark for the video stream according to the video stream information, and determine a processing measure corresponding to the video stream according to the abnormality mark;

[0028] The video processing module is used to execute the processing measures to obtain a processed video stream, and to parse the processed video stream to play the processed video stream.

[0029] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the above method are implemented.

[0030] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0031] In a fifth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0032] The above-mentioned video stream processing method, device, computer equipment, computer-readable storage medium and computer program product parse the video packets in the video stream transmitted by the camera to obtain video stream information when a streaming media server establishes a communication connection with the camera, thereby accurately parsing the video packets transmitted by the camera to obtain accurate video stream information; determining an abnormal identification for the video stream according to the video stream information, and determining the corresponding processing measures for the video stream according to the abnormal identification, thereby analyzing the video stream information to determine whether there is an abnormality and the abnormal identification, and accurately determining the corresponding processing measures using the abnormal identification; executing the processing measures to obtain a processed video stream, and parsing the processed video stream to play the processed video stream, so that by executing the processing measures, the video stream can be played normally, and in the process of parsing the video stream, the abnormality can be analyzed and the corresponding processing measures can be determined, so that the video can be successfully played after the processing measures are executed, and the failure of video stream parsing can be avoided, thereby improving the reliability of video transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 A diagram of an application environment of a video stream processing method in an embodiment;

[0035] Figure 2 A schematic diagram of a process of a video stream processing method in an embodiment;

[0036] Figure 3 is a schematic diagram of a video key frame video packet structure in one embodiment;

[0037] Figure 4 is a schematic diagram of a program mapping stream in one embodiment;

[0038] Figure 5 is a structural block diagram of a video stream processing device in one embodiment;

[0039] Figure 6 The figure is a diagram of the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0041] The video stream processing method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the camera 102 communicates with the streaming media server 104 through the network. The data storage system can store the data that the streaming media server 104 needs to process. The data storage system can be integrated on the streaming media server 104, or it can be placed on the cloud or other network servers. When the streaming media server and the camera establish a communication connection, the streaming media server 104 parses the video packets in the video stream transmitted by the camera to obtain the video stream information; the streaming media server 104 determines the abnormal identification for the video stream according to the video stream information, and determines the corresponding processing measures for the video stream according to the abnormal identification; the streaming media server 104 executes the processing measures to obtain the processed video stream, and parses the processed video stream to play the processed video stream. The streaming media server 104 can be an independent physical server, or it can be a server cluster or distributed system composed of multiple physical servers, or it can be a cloud server that provides cloud computing services.

[0042] In an exemplary embodiment, Figure 2 As shown, a video stream processing method is provided, and the method is applied to a streaming media server as an example for description, including the following steps S202 to S206. Among them:

[0043] Step S202: When the streaming media server establishes a communication connection with the camera, the video packets in the video stream transmitted by the camera are parsed to obtain video stream information.

[0044] The streaming media server may refer to a server used to store, manage and transmit multimedia content such as audio and video.

[0045] The camera may include but is not limited to a video acquisition device / image acquisition device used for video surveillance.

[0046] The video stream transmitted by the camera may refer to the image data captured by the camera, which is encoded and transmitted to the receiving end through the network.

[0047] The video packet may refer to a series of small data units into which the video data is divided during the video stream transmission process. Each video packet may contain a portion of the encoded video frame or audio data, as well as related metadata information.

[0048] Among them, video stream information may refer to information that characterizes the structure of video packets in the video stream and the content of video packets. In practical applications, video stream information may be used to decode and display videos. Video stream information may include but is not limited to program mapping stream information, video element stream packets, audio element stream packets, Slice encoding, data packet size and other information.

[0049] As an example, when a streaming media server establishes a communication connection with a camera, the streaming media server can receive the video stream transmitted by the camera and perform management operations such as decoding and forwarding on the video stream. In order to avoid the differences in the camera's implementation of video transmission standards affecting the normal display of the video, the streaming media server can first parse the video packets in the video stream transmitted by the camera, analyze the program mapping stream, video element stream packets, audio element stream packets, Slice encoding, data packet size and other information, and determine the video stream information.

[0050] Step S204: determining an abnormality mark for the video stream according to the video stream information, and determining a processing measure corresponding to the video stream according to the abnormality mark.

[0051] The anomaly identifier may refer to information representing the type of anomaly in a video stream / video packet. In practical applications, the anomaly identifier may be used as a label.

[0052] The processing measure may refer to an operation that needs to be performed for the video stream anomaly represented by the anomaly identifier. In practical applications, the anomaly identifier may have at least one corresponding processing measure.

[0053] As an example, the streaming media server can analyze the video stream information in combination with preset information and / or data in the video stream to determine whether there is an abnormality in the video stream and determine a corresponding abnormality identifier. The abnormality identifier can characterize the corresponding video stream abnormality. The streaming media server can generate processing measures for the video stream based on the video stream abnormality represented by the abnormality identifier.

[0054] Step S206, executing the processing measures to obtain the processed video stream, and parsing the processed video stream to play the processed video stream.

[0055] The processed video stream may refer to a video stream obtained after executing a processing measure on the video stream.

[0056] As an example, after determining the processing measures corresponding to the abnormal identification, the streaming media server can execute the processing measures corresponding to the abnormal identification. In actual applications, the processing measures corresponding to each abnormal identification can be pre-set. Executing the processing measures corresponding to the abnormal identification can cause the video stream to change, and can also cause other systems in the streaming media server (such as the video decoding system) to change. If executing the processing measures corresponding to the abnormal identification can cause the video stream to change, the streaming media server can use the changed video stream as the processed video stream, and can play the processed video stream after parsing the processed video stream and decoding, rendering and other operations of the parsing result; if executing the processing measures corresponding to the abnormal identification causes other systems in the streaming media server (such as the video decoding system) to change, the streaming media can use the source video stream as the processed video stream, and can play the processed video stream after parsing the processed video stream and decoding, rendering and other operations of the video decoding system.

[0057] In the above-mentioned video stream processing method, by establishing a communication connection between the streaming media server and the camera, the video packets in the video stream transmitted by the camera are parsed to obtain the video stream information, thereby accurately parsing the video packets transmitted by the camera to obtain accurate video stream information; determining the abnormal identification for the video stream according to the video stream information, and determining the corresponding processing measures for the video stream according to the abnormal identification, thereby analyzing the video stream information to determine whether there is an abnormality and the abnormal identification, and accurately determining the corresponding processing measures using the abnormal identification; executing the processing measures to obtain the processed video stream, and parsing the processed video stream to play the processed video stream, thereby executing the processing measures to play the video stream normally, and being able to analyze the abnormality and determine the corresponding processing measures in the process of parsing the video stream, so that the video can be successfully played after the processing measures are executed, avoiding the situation where the video stream parsing fails, thereby improving the reliability of video transmission.

[0058] In an exemplary embodiment, determining an exception identifier for a video stream based on video stream information includes: determining first video type information based on program mapping stream information in the video stream information; generating a first exception identifier for the video stream when the first video type information is inconsistent with the video type of video data in a video packet; the first exception identifier indicates a video type mismatch.

[0059] The program map stream information may refer to information used to describe and organize multiple programs and related data in digital television broadcasting. In practical applications, the program map stream information may be expressed as a PSM (Program Map Table).

[0060] The first video type information may refer to the information of the video type recorded / described in the program mapping stream information.

[0061] The video type of the video data in the video packet may refer to the type of the video data actually carried in the video packet. In practical applications, the video type of the video data in the video packet may be used as the type of the video data in the payload (Payload).

[0062] The first abnormality indicator may refer to information indicating that the video type does not match.

[0063] As an example, the streaming media server can determine the first video type information set by the program mapping stream information PSM according to the program mapping stream information PSM in the video stream information. The streaming media server can also obtain the video type of the payload in the video packet as the video type of the video data in the video packet, and determine whether the first video type information and the video type of the video data in the video packet are consistent by comparing the first video type information and the video type of the video data in the video packet. If the first video type information and the video type of the video data in the video packet are consistent, the streaming media server can determine that there is no abnormality in the video type of the video stream. In the case where the first video type information is inconsistent with the video type of the video data in the video packet, the streaming media server can generate a first abnormality identifier for the video stream, thereby determining that the video stream has an abnormality that the video type does not match through the first abnormality identifier, so as to perform corresponding processing measures. In practical applications, the processing measures corresponding to the first abnormality identifier may include adjusting the encoder for the video stream, for example: adjusting the encoder from the encoder corresponding to the first video type information to the encoder corresponding to the video type of the video data in the video packet.

[0064] In this embodiment, the first video type information is determined based on the program mapping stream information in the video stream information; when the first video type information is inconsistent with the video type of the video data in the video packet, a first exception identifier is generated for the video stream; the first exception identifier represents the video type mismatch, and can accurately analyze whether there is an abnormality in the video type in the video stream, avoid video parsing failure caused by video type abnormality, and thus improve the reliability of video transmission.

[0065] In some embodiments, determining an exception identifier for a video stream based on video stream information includes: determining a video timestamp sequence based on video element stream packets in the video stream information; generating a second exception identifier for the video stream based on the difference and / or size relationship between timestamps in the video timestamp sequence; and the second exception identifier represents a video timestamp anomaly.

[0066] The video element stream packet may refer to a basic data unit in a video coding stream. In practical applications, the video element stream packet may include a coded video.

[0067] The video timestamp sequence may refer to a sequence in which the timestamps corresponding to the videos in the video stream are arranged in a preset order.

[0068] The second abnormality indicator may refer to information indicating an abnormality in the video timestamp.

[0069] As an example, the streaming media server can first obtain and arrange each video timestamp in the video stream according to the video element stream packet in the video stream information to obtain a video timestamp sequence, and then the streaming media server can analyze and calculate the difference and / or size relationship between each video timestamp in the video timestamp sequence, and generate a second abnormality mark for the video stream, so as to represent the abnormal situation of the video timestamp abnormality in the video stream through the second abnormality mark. In practical applications, if the difference between any two adjacent timestamps in the video timestamp sequence is less than or equal to a preset threshold, the streaming media server can determine that there is no abnormality in the video timestamp, and if the difference between any two adjacent timestamps in the video timestamp sequence is greater than a preset threshold, the streaming media server can determine that there is an abnormality in the video timestamp. If the time represented by the timestamp T2 after a timestamp T1 in the video timestamp sequence is less than the time represented by the timestamp T1, the streaming media server can determine that there is an abnormality in the video timestamp, and if the time represented by each timestamp in the video timestamp sequence is greater than the time represented by the timestamp after the timestamp, the streaming media server can determine that there is no abnormality in the video timestamp. In a specific implementation, the processing measures corresponding to the second abnormality mark may include correcting the video timestamp.

[0070] In this embodiment, a video timestamp sequence is determined based on video element stream packets in the video stream information; a second exception identifier for the video stream is generated based on the difference and / or size relationship between each timestamp in the video timestamp sequence; the second exception identifier represents the video timestamp anomaly, and can accurately analyze whether there is an anomaly in the video timestamp in the video stream, avoid video parsing failure caused by the video timestamp anomaly, and thus improve the reliability of video transmission.

[0071] In some embodiments, determining an exception identifier for a video stream based on video stream information includes: determining a video timestamp sequence based on video element stream packets in the video stream information; determining an audio timestamp sequence based on audio element stream packets in the video stream information; generating a third exception identifier for the video stream based on a matching relationship between the video timestamp sequence and the audio timestamp sequence; the third exception identifier indicates that the video timestamp does not match the audio timestamp.

[0072] The audio element stream packet may refer to a basic data unit in an audio coding stream. In practical applications, the audio element stream packet may include encoded audio data.

[0073] The audio timestamp sequence may refer to a sequence in which timestamps corresponding to the audio in the video stream are arranged in a preset order.

[0074] The third abnormality indicator may refer to information indicating that the video timestamp and the audio timestamp do not match.

[0075] As an example, the streaming media server can first obtain and arrange the video timestamps in the video stream according to the video element stream packet in the video stream information to obtain a video timestamp sequence. The streaming media server can also first obtain and arrange the audio timestamps in the video stream according to the audio element stream packet in the video stream information to obtain an audio timestamp sequence. After that, the streaming media server can generate a third abnormality identifier for the video stream according to the matching relationship between the video timestamp sequence and the audio timestamp sequence, so as to characterize the abnormal situation that the video timestamp and the audio timestamp do not match in the video stream through the third abnormality identifier. In practical applications, if the video timestamp sequence and the audio timestamp sequence do not match, the streaming media server can determine that the video stream has an abnormal situation that the video timestamp and the audio timestamp do not match. If the video timestamp sequence and the audio timestamp sequence match, the streaming media server can determine that the video stream has a video timestamp and an audio timestamp that can match normally, and there is no abnormality. In a specific implementation, the processing measures corresponding to the third abnormality identifier may include adjusting the video timestamp and the audio timestamp so that the video timestamp and the audio timestamp are aligned.

[0076] In this embodiment, a video timestamp sequence is determined according to a video element stream packet in the video stream information; an audio timestamp sequence is determined according to an audio element stream packet in the video stream information; a third exception identifier is generated for the video stream according to a matching relationship between the video timestamp sequence and the audio timestamp sequence; the third exception identifier represents a mismatch between the video timestamp and the audio timestamp, and can accurately analyze whether the video timestamp and the audio timestamp in the video stream match, thereby avoiding video parsing failure caused by a mismatch between the video timestamp and the audio timestamp, thereby improving the reliability of video transmission.

[0077] In some embodiments, determining an exception identifier for a video stream based on video stream information includes: determining Slice encoding based on video element stream packets in the video stream information; generating a fourth exception identifier for the video stream when the number of Slice encodings is greater than a preset threshold; the fourth exception identifier represents a Slice encoding exception.

[0078] Slice coding may refer to information used to divide a video frame into smaller, independent coding units.

[0079] The preset threshold may refer to information used to determine whether there is an abnormality in the number of Slice codes.

[0080] The fourth abnormality flag may refer to information indicating an abnormality in Slice encoding.

[0081] As an example, the streaming media server can determine the Slice coding and detect and count the number of Slice coding based on the video element stream packets in the video stream information. When the number of Slice coding is greater than a preset threshold, the streaming media server can determine that the video stream has an abnormal Slice coding abnormality, and then the streaming media server can generate a fourth abnormality identifier for the video stream, thereby characterizing the abnormal Slice coding abnormality of the video stream through the fourth abnormality identifier. In a specific implementation, the processing measures corresponding to the fourth abnormality identifier may include merging Slice coding units.

[0082] In this embodiment, Slice coding is determined based on the video element stream packet in the video stream information; when the number of Slice coding is greater than a preset threshold, a fourth exception flag is generated for the video stream; the fourth exception flag represents the Slice coding exception, and can accurately analyze whether there is an exception in the Slice coding in the video stream, avoid the failure of video parsing caused by the Slice coding exception, and thus improve the reliability of video transmission.

[0083] In some embodiments, determining an exception flag for a video stream based on video stream information includes: determining data packet size information based on packet header information in the video stream information; generating a fifth exception flag for the video stream when the data packet size information is greater than a preset data packet threshold; the fifth exception flag indicates that the data packet is too large.

[0084] The packet header information may refer to information used to identify and describe each data packet / video packet. In actual applications, the packet header information may include a PS (Program Stream) packet header.

[0085] The data packet size information may refer to information representing the total number of bytes contained in the video packet.

[0086] The preset data packet threshold may refer to pre-set information used to determine whether a data packet / video packet is too large.

[0087] The fifth abnormality indicator may refer to information indicating that the data packet is too large.

[0088] As an example, the streaming media server can determine the data packet size information based on the packet header information in the video stream information, and then the streaming media server can compare the data packet size information with the preset data packet threshold to determine the size relationship between the data packet size information and the preset data packet threshold. In the case where the data packet size information is greater than the preset data packet threshold, the streaming media server can generate a fifth abnormality flag for the video stream, thereby characterizing the video stream with an abnormality of excessive data packets through the fifth abnormality flag. In a specific implementation, the processing measures corresponding to the fifth abnormality flag may include splitting data packets / video packets and setting relevant thresholds of the transmission interface so that the data packet size does not exceed the set threshold.

[0089] In this embodiment, the data packet size information is determined based on the packet header information in the video stream information; when the data packet size information is greater than a preset data packet threshold, a fifth exception flag is generated for the video stream; the fifth exception flag indicates that the data packet is too large, and can accurately analyze whether the data packet in the video stream is too large, avoiding video parsing failure caused by the data packet being too large, thereby improving the reliability of video transmission.

[0090] In some embodiments, after the streaming media server receives the video stream transmitted by the camera, the streaming media server can parse the video stream and determine the video key frame video packet structure, such as Figure 3 As shown, a schematic diagram of a video key frame video packet structure is provided. The video key frame video packet structure may include a PA packet header, a system header System Header, a mapping stream PSM, a video element stream packet PESV and an audio element stream packet PESA.

[0091] When parsing the video stream, the streaming media server can detect and determine whether there is an abnormality in the video stream. The abnormal video stream can be regarded as an abnormal stream. When parsing some abnormal streams in PS, the video type set in the program map stream (PSM) is H265, but the payload data in the PES Body is actually H264. This inconsistency will cause abnormal parsing of the video stream, such as Figure 4 As shown in the figure, a schematic diagram of a program mapping stream is provided. In addition, other common abnormalities include abnormal original video timestamp, multi-slice encoding, asynchronization of audio and video timestamps, and RTP packets that are too large. These abnormalities will affect the normal operation of the streaming media server, resulting in reduced stability and reliability of the video surveillance system.

[0092] In order to solve the above problems, a streaming media service based on the video abnormal stream adaptation framework is proposed. The framework aims to identify and process various abnormal streams. By marking the abnormal streams with specific ErrorType tags, the streaming media server can initialize the audio and video encoders according to the abnormal types, so as to gradually adapt and parse these abnormal streams. This can not only ensure the normal parsing of the most common video streams, but also continuously expand the streaming media server's adaptation capabilities for various abnormal streams, and improve its compatibility and adaptability to various cameras on the market. An abnormal stream identification module is introduced into the streaming media server, which can detect and identify various types of abnormal video streams, so as to realize abnormal stream identification and labeling. For the detected abnormal video streams, a specific ErrorType tag is marked. For example, when the video type set in PSM is H265, but the actual video type in PES Body is H264, the tag is ErrorType="PsmError". Based on the detected ErrorType tag, the streaming media server will select and initialize the appropriate encoder according to the tag information, so as to realize encoder initialization. For video streams without abnormal tags, the encoder is initialized according to the audio and video formats in the program map stream (PSM). In the specific implementation, the streaming media server is also equipped with a flexible abnormal stream adaptation framework, which can continuously expand and adapt to new abnormal stream types. The framework includes an abnormal stream processing module, which can call the corresponding processing logic for different ErrorType tags to ensure that the streaming media server can correctly parse and play abnormal video streams.

[0093] In actual applications, abnormal streams / abnormal types may include but are not limited to the video type set by PSM not matching the actual video type, in which case the abnormal identifier / label may be expressed as ErrorType="PsmError"; the original video timestamp is abnormal, in which case the abnormal identifier / label may be expressed as ErrorType="TimestampError"; multi-Slice encoding, in which case the abnormal identifier / label may be expressed as ErrorType="MultiSliceError"; the audio and video timestamps are not synchronized, in which case the abnormal identifier / label may be expressed as ErrorType="AVSyncError"; the RTP packet is too large, in which case the abnormal identifier / label may be expressed as ErrorType="RTPSizeError"; in specific implementations, for each abnormal type, specific processing logic and solutions are provided. It is understandable that streaming media can also conveniently add newly discovered abnormal stream types to the abnormal stream adaptation framework through a dynamic extension mechanism. Through a configuration file or plug-in mechanism, administrators can add new ErrorType tags and their processing logic without restarting the streaming media server.

[0094] Based on the abnormal stream adaptation framework, a video stream processing system can be constructed. The video stream processing system can be applied to the streaming media server. The architecture of the streaming media server includes the front-end abnormal stream identification module, the middle abnormal stream adaptation framework and the back-end encoder initialization module. The front-end module is responsible for receiving and preliminarily processing the RTP stream, identifying the abnormal stream and marking it with a label. The middle module is responsible for calling the processing logic in the adaptation framework according to the label. The back-end module initializes the encoder according to the processed information to decode and play the video stream. The abnormal stream adaptation framework can identify and process various types of abnormal video streams, significantly improve the compatibility of the streaming media server with devices from different manufacturers, and ensure that the RTP streams accessed by the transmission protocols of various cameras can be parsed and played. By marking the abnormal stream with the ErrorType label and initializing the encoder according to the label information, the automatic processing of the abnormal stream can be realized, reducing the need for manual intervention and improving the intelligence level of the system. Processing abnormal streams can effectively avoid system crashes or instability caused by parsing errors, significantly improve the overall stability and reliability of the video surveillance system, and ensure the continuity and integrity of video data. The abnormal stream adaptation framework is designed flexibly and supports dynamic expansion of new abnormal stream types and processing logic. Administrators can easily add new abnormal stream processing rules through configuration files or plug-in mechanisms without restarting the system or making complex code modifications. When processing video streams, the video stream processing system based on the abnormal stream adaptation framework can not only handle specific types of abnormal streams, such as the video type set in PSM is inconsistent with the actual situation, but also deal with other complex abnormal situations, such as abnormal original video timestamps, multi-Slice encoding, audio and video timestamps are not synchronized, RTP packets are too large, etc., which comprehensively improves the adaptability of streaming media servers. By accurately identifying and processing abnormal streams, it can ensure that the audio and video encoders obtain the correct initialization parameters, improve decoding accuracy, and ensure normal playback and high-quality display of video streams. The automated abnormal stream processing mechanism reduces the workload and time of technicians in troubleshooting, improves operation and maintenance efficiency, and reduces system maintenance costs. Because the system can automatically adapt and process abnormal streams, end users will have a smoother, continuous and high-quality viewing experience when watching surveillance videos, which enhances user satisfaction with the video surveillance system.

[0095] In a specific implementation, a video stream processing system based on an abnormal stream adaptation framework may include a front-end module, an intermediate module, and a back-end module, wherein the front-end module may include an abnormal stream identification module, which is responsible for receiving and preliminarily processing the RTP stream, identifying the abnormal stream and marking it with a label. The intermediate module may be used to implement the abnormal stream adaptation framework, and call the corresponding processing logic according to the abnormal type. The back-end module may include an encoder initialization module, which initializes the encoder according to the processed information, and decodes and plays the video stream. The process of the video stream processing system processing the video stream may include: abnormal stream identification: the front-end module receives the RTP stream, parses the stream, and detects whether there is an abnormal situation, such as the video type set in the PSM is inconsistent with the actual situation. Abnormal stream labeling: if an abnormal situation is detected, the front-end module marks the stream with a corresponding ErrorType label, such as ErrorType="PsmError". Abnormal stream adaptation: the intermediate module receives the abnormal stream and identifies the ErrorType label, and calls the corresponding processing logic according to the label, such as for the abnormal stream of ErrorType="PsmError", selects to initialize the H264 encoder, and performs decoding processing. Encoder initialization: the back-end module selects a suitable encoder for initialization according to the processed information. For normal streams, select the corresponding encoder for initialization according to the audio and video format in PSM. For abnormal streams, select the corresponding encoder for initialization according to the ErrorType tag. The abnormal stream processing logic of the video stream processing system may include: PsmError exception handling. For abnormal streams where the video type set in PSM does not match the actual one, select the H264 encoder for initialization and perform decoding. During the decoding process, the video stream is re-parsed to ensure that the video data in the H264 format can be correctly parsed and played. The dynamic expansion mechanism of the video stream processing system allows administrators to add new abnormal stream processing rules and corresponding encoder initialization logic through configuration files or plug-in mechanisms to cope with new abnormal situations that may arise in the future and improve the adaptability and scalability of the system.

[0096] In this embodiment, by identifying and processing various abnormal video streams, the streaming media server can not only parse common video streams, but also adapt to abnormal video streams, thereby improving compatibility with various cameras; by marking different types of abnormal streams with specific ErrorType tags, the encoder is automatically initialized according to the tag information, and automatic adaptation of abnormal streams is achieved, ensuring that even in the face of specific marked abnormal audio and video data formats, the streaming media server can correctly parse and play; by solving the problem of video stream parsing anomalies, the overall stability and reliability of the video surveillance system are improved, and the continuity and accuracy of video surveillance data are ensured; in addition to processing specific abnormal stream types (such as the situation where the PSM is set to H265 but is actually H264 It can also be extended to handle other abnormal situations, such as abnormal original video timestamp, multi-Slice encoding, asynchronous audio and video timestamp, RTP packet size, etc.; through the dynamic expansion mechanism, newly discovered abnormal flow types and processing strategies can be easily added to the system, maintaining the flexibility and adaptability of the system, ensuring that the system can continue to adapt to new abnormal flows; by establishing a complete system architecture, including the front-end abnormal flow identification module, the middle abnormal flow adaptation framework and the back-end decoder initialization module, it ensures that all parts of the system work together to provide efficient and stable abnormal flow processing capabilities, thereby improving the compatibility, stability and decoding accuracy of the streaming media server, and adapting to a variety of camera devices and complex video streaming environments.

[0097] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0098] Based on the same inventive concept, the embodiment of the present application also provides a video stream processing device for implementing the video stream processing method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in the one or more video stream processing device embodiments provided below can refer to the limitations of the video stream processing method above, and will not be repeated here.

[0099] In an exemplary embodiment, Figure 5As shown, a video stream processing device is provided, including: a video stream parsing module 502, an abnormality analysis module 504 and a video processing module 506, wherein:

[0100] The video stream parsing module 502 is used to parse the video packets in the video stream transmitted by the camera to obtain the video stream information when the streaming media server establishes a communication connection with the camera.

[0101] The abnormality analysis module 504 is used to determine an abnormality mark for the video stream according to the video stream information, and determine a processing measure corresponding to the video stream according to the abnormality mark.

[0102] The video processing module 506 is used to execute the processing measures to obtain a processed video stream, and parse the processed video stream to play the processed video stream.

[0103] In one of the exemplary embodiments, the exception analysis module 504 is specifically used to determine the first video type information based on the program mapping stream information in the video stream information; when the first video type information is inconsistent with the video type of the video data in the video packet, generate a first exception identifier for the video stream; the first exception identifier indicates a video type mismatch.

[0104] In one of the exemplary embodiments, the anomaly analysis module 504 is specifically used to determine a video timestamp sequence based on the video element stream packets in the video stream information; generate a second anomaly identifier for the video stream based on the difference and / or size relationship between the timestamps in the video timestamp sequence; and the second anomaly identifier represents a video timestamp anomaly.

[0105] In one of the exemplary embodiments, the exception analysis module 504 is specifically used to determine a video timestamp sequence based on video element stream packets in the video stream information; determine an audio timestamp sequence based on audio element stream packets in the video stream information; generate a third exception identifier for the video stream based on a matching relationship between the video timestamp sequence and the audio timestamp sequence; the third exception identifier indicates that the video timestamp does not match the audio timestamp.

[0106] In one of the exemplary embodiments, the exception analysis module 504 is specifically used to determine the Slice encoding based on the video element stream packet in the video stream information; when the number of the Slice encoding is greater than a preset threshold, generate a fourth exception flag for the video stream; the fourth exception flag represents the Slice encoding exception.

[0107] In one of the exemplary embodiments, the exception analysis module 504 is specifically used to determine the data packet size information based on the packet header information in the video stream information; when the data packet size information is greater than a preset data packet threshold, generate a fifth exception flag for the video stream; the fifth exception flag indicates that the data packet is too large.

[0108] Each module in the above video stream processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0109] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as video stream information. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a video stream processing method is implemented.

[0110] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0111] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0112] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0113] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0114] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0115] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0116] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0117] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A video stream processing method, characterized in that: Applied to a streaming media server, the method comprises: When the streaming media server establishes a communication connection with the camera, parsing video packets in the video stream transmitted by the camera to obtain video stream information; Determine an abnormality mark for the video stream according to the video stream information, and determine a processing measure corresponding to the video stream according to the abnormality mark; The processing measures are executed to obtain a processed video stream, and the processed video stream is parsed to play the processed video stream.

2. The method according to claim 1, characterized in that The determining, according to the video stream information, an abnormality identifier for the video stream includes: Determining first video type information according to program mapping stream information in the video stream information; In the case where the first video type information is inconsistent with the video type of the video data in the video packet, a first exception flag is generated for the video stream; the first exception flag indicates that the video type does not match.

3. The method according to claim 1, characterized in that The determining, according to the video stream information, an abnormality identifier for the video stream includes: Determine a video timestamp sequence according to the video element stream packets in the video stream information; A second abnormality flag for the video stream is generated according to the difference and / or size relationship between the timestamps in the video timestamp sequence; the second abnormality flag indicates that the video timestamp is abnormal.

4. The method according to claim 1, characterized in that: The determining, according to the video stream information, an abnormality identifier for the video stream includes: Determine a video timestamp sequence according to the video element stream packets in the video stream information; Determine an audio timestamp sequence according to the audio element stream packets in the video stream information; A third abnormality flag for the video stream is generated according to the matching relationship between the video timestamp sequence and the audio timestamp sequence; the third abnormality flag indicates that the video timestamp does not match the audio timestamp.

5. The method according to claim 1, characterized in that The determining, according to the video stream information, an abnormality identifier for the video stream includes: Determine Slice encoding according to the video element stream packet in the video stream information; When the number of Slice codes is greater than a preset threshold, a fourth abnormality flag is generated for the video stream; the fourth abnormality flag indicates that Slice codes are abnormal.

6. The method according to claim 1, characterized in that The determining, according to the video stream information, an abnormality identifier for the video stream includes: Determine data packet size information according to packet header information in the video stream information; When the data packet size information is larger than a preset data packet threshold size, a fifth abnormality flag is generated for the video stream; the fifth abnormality flag indicates that the data packet is too large.

7. A video stream processing device, characterized in that: Applied to a streaming media server, the device comprises: A video stream parsing module, used for parsing video packets in a video stream transmitted by the camera to obtain video stream information when the streaming media server establishes a communication connection with the camera; An abnormality analysis module, used to determine an abnormality mark for the video stream according to the video stream information, and determine a processing measure corresponding to the video stream according to the abnormality mark; The video processing module is used to execute the processing measures to obtain a processed video stream, and to parse the processed video stream to play the processed video stream.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images