Video stream processing method and system

Abstract

The invention provides a video stream processing method and system, the system comprises a first device and a second device in communication connection with the first device, an encoder is deployed on the first device, and a decoder is deployed on the second device; the method comprises the following steps: a first device receives each video stream, wherein each video stream comprises a key frame sequence; the first device determines a target video stream according to each key frame sequence, and the target video stream is a video stream with the smallest picture scene change in each video stream; in the embodiment of the invention, the first device or the first device and the second device perform collision elimination processing on the target video stream to obtain the processed target video stream, and the collision elimination effect is evaluated according to the processed target video stream, so that the video stream collision phenomenon is reduced, and the network congestion is alleviated.

Description

Technical Field

[0001] The present application relates to the technical field of data transmission, and in particular to a video stream processing method and system. Background Art

[0002] In the field of data transmission, a video stream consists of multiple video frames, which mainly include I frames (key frames) and P frames (predicted frames). I frames are used to describe the image background and moving subjects. When decoding, only I frame data can be used to reconstruct the complete image. Since I frames contain more information than P frames, they require more bandwidth during network transmission.

[0003] When watching multiple video streams over the network, such as video conferencing or multi-channel surveillance videos, multiple video streams are transmitted simultaneously, and multiple I frames may arrive at almost the same time, i.e., I frame collision occurs. I frame collision can cause network congestion, packet loss, and other problems, seriously affecting video quality. Currently, there is no way to solve the network congestion problem caused by I frame collision.

[0004] Therefore, the existing technology cannot effectively cope with the challenges brought by I frame collision and has certain limitations. Summary of the invention

[0005] The purpose of the present application is to provide a video stream processing method and system to address the deficiencies in the above-mentioned prior art, so as to solve the problem that the prior art cannot solve the actual problem of network congestion caused by I frame collision.

[0006] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0007] In a first aspect, an embodiment of the present application provides a video stream processing method, which is applied to a video stream processing system, wherein the video stream processing system includes: a first device and a second device connected to the first device for communication, wherein an encoder is deployed on the first device, and a decoder is deployed on the second device; the method includes:

[0008] The first device receives each video stream, each of the video streams includes a key frame sequence;

[0009] The first device determines a target video stream according to each of the key frame sequences, wherein the target video stream is a video stream with the smallest scene change among the video streams;

[0010] The first device or the first device and the second device perform collision elimination processing on the target video stream to obtain a processed target video stream, and evaluate a collision elimination effect based on the processed target video stream.

[0011] As an optional implementation, determining the target video stream according to each of the key frame sequences includes:

[0012] Determine a key frame score of each video stream according to each of the key frame sequences, wherein the key frame score is used to characterize a characteristic change of a key frame in the video stream;

[0013] The target video stream is determined according to the key frame scores of each of the video streams.

[0014] As an optional implementation, determining the key frame score of each video stream according to each key frame sequence includes:

[0015] Determining the variance of the feature change rate of each key frame of each video stream according to each of the key frame sequences;

[0016] The key frame score of each video stream is determined according to the variance of the change rate of each key frame feature of each video stream and the preset weight of each key frame feature.

[0017] As an optional implementation, determining the variance of the feature change rate of each key frame of each video stream according to each key frame sequence includes:

[0018] Extracting image features from each key frame in each key frame sequence to obtain key frame features of each video stream, wherein the key frame features include pixel intensity, texture, edge, shape, and corner point;

[0019] Determining the change rate of each key feature of each video stream according to each key frame feature of each video stream;

[0020] According to the change rates of the key features of the video streams, the variances of the change rates of the key frame features of the video streams are determined.

[0021] As an optional implementation, determining the target video stream according to the key frame scores of each of the video streams includes:

[0022] The key frame scores of the video streams are minimized, and the video stream corresponding to the smallest key frame score among the key frame scores of the video streams is used as the target video stream.

[0023] As an optional implementation manner, the first device or the first device and the second device perform collision elimination processing on the target video stream to obtain a processed target video stream, and evaluate the collision elimination effect according to the processed target video stream, including:

[0024] The first device performs a first collision elimination process on the target video stream to obtain a first processed target video stream, and evaluates a collision elimination effect corresponding to the first process according to the first processed target video stream;

[0025] Alternatively, the first device and the second device perform a second collision elimination process on the target video stream to obtain a second processed target video stream, and evaluate a collision elimination effect corresponding to the second process based on the second processed target video stream.

[0026] As an optional implementation manner, the first device performs a first collision elimination process on the target video stream to obtain a first processed target video stream, including:

[0027] The first device obtains each current encoding parameter of the target video stream, and determines each target encoding parameter of the target video stream according to each current encoding parameter of the target video stream;

[0028] The first device determines each preset adjustment interval corresponding to each target encoding parameter of the target video stream, and adjusts each target encoding parameter of the target video stream according to each preset adjustment interval to obtain the first processed target video stream, so that each encoding parameter of the first processed target video stream is at a lower limit threshold of each corresponding preset adjustment interval.

[0029] As an optional implementation manner, the first device and the second device perform a second collision elimination process on the target video stream to obtain a second processed target video stream, including:

[0030] The first device sends the key frame sequence of each video stream to the second device in advance;

[0031] The first device sends the identifier of the target video stream to the second device, discards the newly generated key frame sequence of the target video stream in real time, generates key frame sequences of a specified video stream in real time and sends the key frame sequences to the second device, wherein the specified video stream is a video stream other than the target video stream;

[0032] The second device extracts the key frame sequence of the target video stream from the key frame sequences of each video stream sent in advance by the first device according to the identifier of the target video stream, and plays the video according to the key frame sequence of the target video stream and the key frame sequences of the designated video stream.

[0033] As an optional implementation, playing the video according to the key frame sequence of the target video stream and each key frame sequence of the designated video stream includes:

[0034] Importing the key frame sequence of the target video stream into the decoder of the second device in a loop for decoding, and playing the video corresponding to the target video stream in a loop;

[0035] Import each key frame sequence of the designated video stream into the decoder of the second device in real time for decoding, and play the video corresponding to the designated video stream in real time.

[0036] In a second aspect, an embodiment of the present application provides a video stream processing system, the video stream processing system comprising: a first device and a second device connected to the first device for communication, an encoder being deployed on the first device, and a decoder being deployed on the second device;

[0037] The video stream processing system is used to execute the steps of the video stream processing method described in the first aspect above.

[0038] The beneficial effects of this application are:

[0039] The present application provides a video stream processing method and system. In a scenario where multiple video streams are concurrent, the first device in the video stream processing system receives multiple video streams sent by each video stream sending source, and selects a target video stream with the smallest scene change from the multiple video streams based on the key frame sequence included in each video stream. The encoder deployed in the first device performs collision elimination processing on the target video stream, obtains the processed target video stream and sends it to the second device, and the first device evaluates the collision elimination effect based on the processed target video stream. Alternatively, the decoder deployed in the second device and the first device perform collision elimination processing on the target video stream, obtain the processed target video stream and evaluate the collision elimination effect. By taking the video stream with the smallest scene change in multiple video streams as the target video stream, and performing collision elimination processing on the target video stream at the encoding end or the decoding end, the flexibility of the collision elimination processing is achieved, the probability of key frame collision of each video stream is reduced, and the network congestion phenomenon is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 A schematic diagram of the architecture of a video stream processing system provided in an embodiment of the present application;

[0042] Figure 2 A schematic diagram of a video stream processing method provided in an embodiment of the present application;

[0043] Figure 3 A schematic diagram of a process for determining a target video stream in a video stream processing method provided in an embodiment of the present application;

[0044] Figure 4 A schematic diagram of a process for determining key frame scores of each video stream in a video stream processing method provided in an embodiment of the present application;

[0045] Figure 5 A schematic diagram of a flow chart of determining the variance of the feature change rate of each key frame of each video stream in the video stream processing method provided in an embodiment of the present application;

[0046] Figure 6 A schematic diagram of a process flow of collision elimination processing of a video stream processing method provided in an embodiment of the present application;

[0047] Figure 7 A schematic diagram of a process of obtaining a first processed target video stream according to a video stream processing method provided in an embodiment of the present application;

[0048] Figure 8 A schematic diagram of an interface of various encoding parameters of a target video stream provided in an embodiment of the present application;

[0049] Fig. 9 A schematic diagram of a process of obtaining a second processed target video stream according to a video stream processing method provided in an embodiment of the present application;

[0050] Fig.10 A schematic diagram of the process of playing a video in the video stream processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0052] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0053] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0054] In the field of data transmission, when watching multiple video streams through the network, multiple video streams are transmitted simultaneously, and I frame collision may occur, causing network congestion, packet loss and other problems, seriously affecting video quality. At present, it is impossible to effectively deal with the challenges brought by I frame collision, and there are certain limitations.

[0055] Based on the above-mentioned problems, an embodiment of the present application provides a video stream processing method, which eliminates the network congestion problem caused by the collision of multiple video streams and greatly reduces the network load by effectively identifying the target video stream with the smallest scene change from multiple video streams and performing encoding parameter adjustment or decoding strategy on the target video stream.

[0056] Figure 1 The schematic diagram of the architecture of the video stream processing system provided in the embodiment of the present application is as follows: Figure 1 As shown, the video stream processing system includes a first device and a second device, and the first device and the second device are in communication connection. Figure 1 In the video stream processing system, a second device is included as an example, the second device is a playback end of the video stream, and a decoder and a second processor are deployed on the second device ( Figure 1 The second device is used to decode the I frame sequence of each video stream and then play the video corresponding to each video stream. The video stream processing system may also include multiple second devices, each of which is respectively connected to the first device for communication to receive the I frame sequence of each video stream sent by the first device.

[0057] Continue to refer to Figure 1 The first device is also connected to multiple video stream transmission sources to receive video streams from multiple sources. The first device is equipped with an encoder and a first processor ( Figure 1The first device is used to compare the scene changes of the video streams from multiple sources and determine the target video stream with the smallest scene change. The first device can perform collision elimination processing on the target video stream in terms of adjusting the encoding parameters at the encoding end, and can also cooperate with the second device to perform collision elimination processing on the target video stream in terms of decoding at the decoding end, so that the second device can play the video after the collision elimination processing, reduce the collision phenomenon of the video stream, and reduce the occurrence of network congestion.

[0058] Figure 2 A schematic diagram of a video stream processing method provided in an embodiment of the present application, wherein the execution subject of the method is Figure 1 The video stream processing system shown in FIG. Figure 2 As shown, the method includes:

[0059] S201. A first device receives each video stream, each video stream including a key frame sequence.

[0060] Optionally, continue to refer to Figure 1 In a scenario where multiple video streams are concurrent, the first device is connected to each video stream sending source in communication, and each video stream sending source sends the collected or stored video stream to the first device by communicating with the first device. The video stream sending source can be a video stream collecting device such as a camera or an electronic device storing a video stream, and is not specifically limited here.

[0061] A video stream can be a video file that plays and transmits data in the form of frames. The video stream includes a key frame sequence, namely an I-frame sequence, that is, the video stream is composed of multiple I-frames arranged in time sequence. Among them, I-frames are independently coded and generated. I-frames can describe the image background and moving subjects in detail, and have a low compression rate. During decoding, the complete image can be reconstructed based on I-frames alone, and then the video corresponding to the video stream composed of the I-frame sequence can be fully played based on the I-frame sequence.

[0062] The first device numbers each received video stream and assigns a unique identifier flow_ID to each video stream to distinguish data of different video streams. For example, if the first device receives N video streams, the N video stream identifiers flow_ID = [1, 2, 3, ..., N]. The identifier of the i-th video stream is flow_ID i , where 1≤i≤N, i is any one of the N video streams.

[0063] S202: The first device determines a target video stream according to each key frame sequence, where the target video stream is a video stream having the smallest scene change among all video streams.

[0064] Optionally, the first device selects a target video stream from each video stream according to an I frame sequence of each video stream, so as to perform subsequent collision elimination processing based on the target video stream.

[0065] Specifically, the first device detects the degree of scene change of each video stream according to the I frame sequence of each video stream, and uses the identifier of the video stream with the smallest scene change among the video streams as the identifier of the target video stream. For example, if it is detected that the scene change of the mth video stream among N video streams is the smallest, the identifier flow_ID of the mth video stream is used as the identifier of the target video stream. m As the identifier of the target video stream, the mth video stream is selected as the target video stream, where 1≤m≤N.

[0066] S203: The first device or the first device and the second device perform collision elimination processing on the target video stream to obtain a processed target video stream, and evaluate a collision elimination effect based on the processed target video stream.

[0067] Optionally, after the first device determines the target video stream, the encoder deployed on the first device performs collision elimination processing on the target video stream corresponding to the identifier of the target video stream at the encoding end based on the identifier of the target video stream to obtain a processed target video stream, and evaluates whether the collision elimination processing of the first device at the encoding end is effective based on the processed target video stream.

[0068] Alternatively, after the first device determines the target video stream, it sends the identifier of the target video stream to the second device, and the decoder deployed on the second device performs collision elimination processing on the target video stream corresponding to the identifier of the target video stream at the decoding end based on the identifier of the target video stream to obtain a processed target video stream, and evaluates whether the collision elimination processing of the second device at the decoding end is effective based on the processed target video stream.

[0069] In this embodiment, in a scenario where multiple video streams are concurrent, the first device in the video stream processing system receives multiple video streams sent by each video stream sending source, and selects a target video stream with the smallest scene change from the multiple video streams based on the key frame sequence included in each video stream. The encoder deployed in the first device performs collision elimination processing on the target video stream, obtains the processed target video stream and sends it to the second device, and the first device evaluates the collision elimination effect based on the processed target video stream. Alternatively, the decoder deployed in the second device and the first device perform collision elimination processing on the target video stream, obtain the processed target video stream and evaluate the collision elimination effect. By taking the video stream with the smallest scene change among multiple video streams as the target video stream, and performing collision elimination processing on the target video stream at the encoding end or the decoding end, the flexibility of the collision elimination processing is achieved, the probability of key frame collision of each video stream is reduced, and network congestion is reduced.

[0070] The following is a detailed description of the process of determining the target video stream based on each key frame sequence.

[0071] Figure 3 A schematic diagram of a process for determining a target video stream in a video stream processing method provided in an embodiment of the present application, such as Figure 3 As shown, the step of determining the target video stream according to each key frame sequence in the above step S202 includes:

[0072] S301. Determine a key frame score of each video stream according to each key frame sequence, wherein the key frame score is used to characterize a characteristic change of a key frame in the video stream.

[0073] Optionally, the first processor of the first device calculates the I frame score of each video stream according to the I frame sequence of each video stream, and the I frame score of each video stream can characterize the feature change of the I frame in each video stream. The I frame score of each video stream is positively correlated with the degree of feature change of the I frame in each video stream. The higher the I frame score of the video stream, the greater the degree of feature change of the I frame in the video stream, and the greater the degree of picture scene change of the video stream. Conversely, the lower the I frame score of the video stream, the smaller the degree of feature change of the I frame in the video stream, and the smaller the degree of picture scene change of the video stream.

[0074] S302: Determine a target video stream according to the key frame scores of each video stream.

[0075] Optionally, the first processor determines a video stream having a smaller degree of picture scene change based on the characteristic that the I frame score of each video stream is positively correlated with the degree of feature change of the I frame in each video stream, and uses the identifier of the video stream as the identifier of the target video stream to determine the target video stream having the smaller degree of picture scene change among the video streams.

[0076] In this embodiment, the first device calculates the key frame score of each video stream according to the key frame sequence of each video stream, and determines the target video stream with the smallest scene change degree in each video stream based on the characteristic of positive correlation between the key frame score of each video stream and the characteristic change of the key frame in each video stream according to the characteristic of positive correlation between the key frame score of each video stream and the characteristic change of the key frame in each video stream. In this way, when performing collision elimination processing on the target video stream, the impact on video playback can be minimized while reducing video stream collision.

[0077] The following describes in detail the process of determining the key frame score of each video stream based on each key frame sequence.

[0078] Figure 4 A schematic diagram of a process for determining the key frame score of each video stream of the video stream processing method provided in an embodiment of the present application, such as Figure 4As shown, the step of determining the key frame score of each video stream according to each key frame sequence in the above step S301 includes:

[0079] S401. Determine the variance of the feature change rate of each key frame of each video stream according to each key frame sequence.

[0080] Optionally, the first processor of the first device extracts the I frame features of each I frame in the I frame sequence of each video stream at each moment according to the I frame sequence of each video stream, and compares the difference changes of the I frame features of each video stream at each moment, and calculates the variance Var_Δf of the change rate of each I frame feature of each video stream at each moment according to the difference changes of the I frame features of each video stream at each moment i j .

[0081] Among them, the larger the variance of the change rate of each I-frame feature of a video stream, the more unstable the I-frame features of the video stream at each moment, that is, the higher the degree of scene change of the video stream, the lower the probability of selecting the video stream as the target video stream.

[0082] S402: Determine the key frame score of each video stream according to the variance of the feature change rate of each key frame of each video stream and the preset feature weight of each key frame.

[0083] Optionally, the first processor calculates the variance Var_Δf of the feature change rate of each I frame of each video stream at each moment i j The weighted average operation is performed with the preset weights of each I frame feature of the corresponding category to obtain the I frame score of each video stream at each moment.

[0084] Exemplarily, each I frame feature includes pixel intensity, texture, edge, shape and corner point, wherein, since pixel intensity and texture can better reflect the changes of the picture scene than edges, shapes and corner points, the preset pixel intensity weight and the preset texture weight can be greater than the preset edge weight, the preset shape weight and the preset corner point weight, so that the changes of the two I frame features of pixel intensity and texture can better affect the I frame score of the video stream, thereby making the I frame score of the video stream characterize the feature changes of the I frames in the video stream to the greatest extent.

[0085] In this embodiment, the first device extracts the key frame features of each key frame in the key frame sequence of each video stream at each moment based on the key frame sequence of each video stream, and compares the key frame feature differences of each video stream at each moment to obtain the variance of the change rate of each key frame feature of each video stream at each moment. The variance of the change rate of each key frame feature of each video stream at each moment is weighted averaged with the preset weights of each key frame feature of the corresponding category to obtain the key frame score of each video stream at each moment. The accuracy of the key frame score of the video stream in representing the feature changes of the key frames in the video stream is improved, thereby improving the accuracy of selecting the target video stream.

[0086] The following describes in detail the process of determining the variance of the feature change rate of each key frame of each video stream based on each key frame sequence.

[0087] Figure 5 A schematic diagram of a flow chart of determining the variance of the feature change rate of each key frame of each video stream in the video stream processing method provided in an embodiment of the present application, such as Figure 5 As shown, the step of determining the variance of the feature change rate of each key frame of each video stream according to each key frame sequence in the above step S401 includes:

[0088] S501 , extracting image features from each key frame in each key frame sequence to obtain key frame features of each video stream, where the key frame features include pixel intensity, texture, edge, shape, and corner point.

[0089] Optionally, the first processor of the first device extracts image features from each I frame in the I frame sequence of each video stream through a deep learning algorithm to obtain features of each I frame of each video stream at each moment, wherein the I frame is an image frame, and the I frame features include pixel intensity, texture, edge, shape, and corner point of the image described by the I frame.

[0090] Specifically, the first processor inputs each I frame in the I frame sequence of each video stream into a convolutional neural network (CNN). CNN uses a deep learning algorithm to extract image features from each I frame in the I frame sequence of each video stream to obtain I frame features of pixel intensity, texture, edge, shape, and corner points.

[0091] For example, at time t, the extracted h-th I-frame feature of the i-th video stream is f i j (t), at the next moment after time t, that is, time t+1, the feature of the jth I frame of the i-th video stream extracted is f i j(t+1). The i-th video stream may be any one of the N video streams, and the j-th I-frame feature may be any one of the I-frame features of pixel intensity, texture, edge, shape, and corner point.

[0092] S502: Determine the change rate of each key feature of each video stream according to each key frame feature of each video stream.

[0093] Optionally, the first processor uses a frame difference method to compare the differences of various I-frame features between consecutive I-frames according to the features of each I-frame of each video stream at each moment to calculate the feature increment of each I-frame of each video stream at each moment as the feature change rate Δf of each I-frame of each video stream at each moment. i j (n).

[0094] Specifically, the I-frame feature increment Δf of the j-th I-frame feature of the i-th video stream from time t to time t+1 is determined based on the following formula: i j (t):

[0095] Δf i j (t) = f i j (t+1)-f i j (t)

[0096] Where Δf i j (t) is the I-frame feature increment of the j-th I-frame feature of the i-th video stream from time t to time t+1, that is, the I-feature change rate of the j-th I-frame feature of the i-th video stream, f i j (t+1) is the jth I-frame feature of the i-th video stream at time t+1, f i j (t) is the jth I-frame feature of the i-th video stream at time t. I-frame feature increment Δf i j The larger (t) is, the greater the change between the two frames of the i-th video stream from time t to time t+1.

[0097] Repeat the above steps to obtain the feature change rate Δf of each I frame of each video stream at each time i j (n).

[0098] S503: Determine the variance of the feature change rate of each key frame of each video stream according to the feature change rate of each key feature of each video stream.

[0099] Optionally, the first processor calculates the feature change rate Δf of each I frame of each video stream at each moment.i j (n), the expected Ave_Δf of the feature change rate of each I frame of each video stream at each moment is determined by averaging based on the following formula i h :

[0100]

[0101] Where Ave_Δf i j is the expected rate of change of the features of each I frame of each video stream at each moment, Δf i j (n) is the feature change rate of each I frame of each video stream at each moment, and N is the number of video streams.

[0102] The first processor calculates the expected Ave_Δf of the feature change rate of each I frame of each video stream at each moment. i j , based on the following formula, the variance Var_Δf of the feature change rate of each I frame of each video stream at each moment is determined i j :

[0103]

[0104] Among them, Var_Δf i j is the variance of the feature change rate of each I frame of each video stream at each moment, Δf i j (n) is the feature change rate of each I frame of each video stream at each moment, Ave_Δf i j is the expected rate of change of the features of each I frame of each video stream at each moment, and N is the number of video streams.

[0105] In this embodiment, the first device inputs the key frame sequence of each video stream into a convolutional neural network, and uses a deep learning algorithm to extract key frame features of pixel intensity, texture, edge, shape, and corner points from each key frame in the key frame sequence of each video stream. The first device uses a frame difference method to compare the differences in various key frame features between consecutive key frames based on the key frame features of each video stream at each moment to determine the change rate of each key frame feature of each video stream at each moment. According to the change rate of each key frame feature of each video stream at each moment, the expected change rate of each key frame feature of each video stream at each moment is determined, and then according to the expected change rate of each key frame feature of each video stream at each moment, the variance of the change rate of each key frame feature of each video stream at each moment is determined. Through image feature extraction and frame difference method, the variance of the change rate of each key frame feature of each video stream at each moment is accurately detected.

[0106] As an optional implementation, the step of determining the target video stream according to the key frame scores of each video stream in the above step S302 includes:

[0107] The key frame scores of each video stream are minimized, and the video stream corresponding to the smallest key frame score among the key frame scores of each video stream is used as the target video stream.

[0108] Optionally, the first processor of the first device performs a minimum operation on the key frame scores of each video stream to obtain a target video stream. Specifically, the first processor sorts the key frame scores of each video stream in order from large to small or from small to large, and takes the identifier of the video stream with the smallest key frame score among the key frame scores of each video stream as the identifier of the target video stream, so that the key frame score of the target video stream is the smallest among the key frame scores of each video stream. Since the key frame score represents the characteristic change of the key frame in the video stream, it is ensured that the characteristic change degree of the key frame of the selected target video stream is the smallest. The key frame describes the image background and the moving subject, that is, the picture scene of the target video stream has the smallest change degree. In order to minimize the adverse effects on video playback when the target video stream is subsequently subjected to collision elimination processing at the encoding end or collision elimination processing at the decoding end.

[0109] In this embodiment, the first device performs a minimum operation on the key frame scores of each video stream, and takes the identifier of the video stream with the smallest key frame score among the key frame scores of each video stream as the identifier of the target video stream, so that the key frame score of the target video stream is the smallest among the key frame scores of each video stream, and the degree of feature change of the key frame of the selected target video stream is guaranteed to be the smallest. In this way, when performing collision elimination processing on the target video stream at the encoding end or the decoding end, the impact on video playback is minimized to the greatest extent, and the negative impact on video playback is avoided while alleviating network congestion.

[0110] The following describes in detail a process in which the first device or the first device and the second device perform collision elimination processing on a target video stream to obtain a processed target video stream, and evaluates a collision elimination effect based on the processed target video stream.

[0111] Figure 6 A schematic diagram of a collision elimination process of a video stream processing method provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, in the above step S203, the first device or the first device and the second device perform collision elimination processing on the target video stream to obtain a processed target video stream, and the step of evaluating the collision elimination effect according to the processed target video stream includes:

[0112] S601: A first device performs a first collision elimination process on a target video stream to obtain a first processed target video stream, and evaluates a collision elimination effect corresponding to the first process based on the first processed target video stream.

[0113] Optionally, the first processor of the first device sends the identifier of the target video stream to the encoder of the first device, and the encoder deployed on the first device performs a first collision elimination process on the target video stream corresponding to the identifier of the target video stream at the encoding end based on the identifier of the target video stream, obtains the first processed target video stream and sends it to the first processor. The first collision elimination process on the target video stream may be to adjust the encoding parameters of the target video stream to adjust the occupied bandwidth when transmitting the target video stream.

[0114] The first processor sends the I frame sequence of the first processed target video stream and the I frame sequences of other video streams except the first processed target video stream to the second device, and calculates the I frame collision severity according to the I frame sequence of the first processed target video stream and the I frame sequences of other video streams except the first processed target video stream to obtain an I frame collision severity score, and evaluates whether the collision elimination processing of the first device at the encoding end is effective according to the I frame collision severity score and the initial I frame collision severity score calculated in advance by the first processor according to the I frame sequence of each video stream.

[0115] Specifically, if the I frame collision severity score calculated by the first processor is lower than the initial I frame collision severity score after the encoder of the first device performs the first collision elimination processing on the target video stream, it indicates that the collision elimination processing of the first device at the encoding end is effective; otherwise, it indicates that the collision elimination processing of the first device at the encoding end is invalid. Based on the method of the aforementioned embodiment, a new target video stream can be reselected and the first collision elimination processing and effect evaluation can be performed on the new target video stream until the I frame collision severity score is lower than the preset I frame collision severity score threshold.

[0116] S602, or, the first device and the second device perform a second collision elimination process on the target video stream to obtain a second processed target video stream, and evaluate a collision elimination effect corresponding to the second process based on the second processed target video stream.

[0117] Optionally, the first processor of the first device sends the identifier of the target video stream to the second processor and encoder of the second device, and the second processor and decoder perform a second collision elimination process on the target video stream corresponding to the identifier of the target video stream at the decoding end based on the identifier of the target video stream to obtain the second processed target video stream. The second collision elimination process on the target video stream may be to adjust the source of the I frame sequence of the target video stream to adjust the number of video streams transmitted in real time between the first device and the second device. The source of the I frame sequence of the target video stream includes the I frame sequence generated and transmitted in real time by the first device and the I frame sequence sent in advance by the first device.

[0118] The first device sends the I frame sequences of other video streams except the processed target video stream to the second device in real time, and the second processor calculates the I frame collision severity according to the I frame sequences of other video streams except the second processed target video stream sent by the first device to obtain the I frame collision severity score, and evaluates whether the collision elimination processing of the second device at the decoding end is effective according to the I frame collision severity score and the initial I frame collision severity score calculated in advance by the second processor according to the I frame sequence of each video stream.

[0119] Specifically, if the I frame collision severity score calculated by the second processor is lower than the initial I frame collision severity score after the second processor and decoder of the second device perform the second collision elimination processing on the target video stream, it indicates that the collision elimination processing of the second device at the decoding end is effective; otherwise, it indicates that the collision elimination processing of the second device at the decoding end is invalid. Based on the method of the aforementioned embodiment, a new target video stream can be reselected and the second collision elimination processing and effect evaluation can be performed on the new target video stream until the I frame collision severity score is lower than the preset I frame collision severity score threshold.

[0120] In this embodiment, the encoder deployed on the first device adjusts the encoding parameters of the target video stream by performing a first collision elimination process on the target video stream to obtain a first processed target video stream, and can adjust the occupied bandwidth when transmitting the target video stream, and calculate the key frame severe collision degree according to the first processed target video stream, and evaluate whether the collision elimination process of the first device at the encoding end is effective based on the key frame collision severity. Alternatively, the decoders deployed on the first device and the second device adjust the source of the key frame sequence of the target video stream by performing a second collision elimination process on the target video stream to obtain a second processed target video stream, and can adjust the number of video streams transmitted in real time between the first device and the second device, and calculate the key frame severe collision degree according to other video streams sent by the first device except the second processed target video stream, and evaluate whether the collision elimination process of the second device at the decoding end is effective based on the key frame collision severity. The flexibility of the collision elimination process is improved, and the occurrence of network congestion is reduced by adjusting the occupied bandwidth when transmitting the target video stream or the number of video streams transmitted in real time between the first device and the second device.

[0121] The following describes in detail a process in which the first device performs a first collision elimination process on the target video stream to obtain the target video stream after the first process.

[0122] Figure 7 A schematic diagram of a process of obtaining a first processed target video stream according to a video stream processing method provided in an embodiment of the present application, such as Figure 7 As shown, in the above step S601, the step of performing a first collision elimination process on the target video stream by the first device to obtain the first processed target video stream includes:

[0123] S701. The first device obtains current encoding parameters of a target video stream, and determines target encoding parameters of the target video stream according to the current encoding parameters of the target video stream.

[0124] Optionally, the encoding parameters of the video stream include at least: resolution, video frame rate, I frame interval, bit rate upper limit, encoding complexity and image quality. Among them, the bit rate upper limit is a non-adjustable encoding parameter, that is, a fixed value, and the resolution, video frame rate, I frame interval, encoding complexity and image quality are adjustable encoding parameters.

[0125] The encoder deployed on the first device obtains each current encoding parameter of the target video stream corresponding to the identifier of the target video stream based on the identifier of the target video stream. Exemplarily, Figure 8 The interface diagram of each encoding parameter of the target video stream provided in the embodiment of the present application is shown in FIG. Figure 8Among the current encoding parameters of the target video stream, the current resolution is 1920*1080P, the current video frame rate is 25FPS, the current I frame interval is 50, the upper limit of the bit rate is a fixed 2048Kbps, the current encoding complexity is medium, and the current image quality is the highest.

[0126] The encoder uses the adjustable encoding parameters in the current encoding parameters of the target video stream as target encoding parameters, that is, the target encoding parameters of the target video stream include resolution, video frame rate, I frame interval, encoding complexity and image quality.

[0127] S702. The first device determines each preset adjustment interval corresponding to each target encoding parameter of the target video stream, and adjusts each target encoding parameter of the target video stream according to each preset adjustment interval to obtain a first processed target video stream, so that each encoding parameter of the first processed target video stream is at a lower limit threshold of each corresponding preset adjustment interval.

[0128] Optionally, the encoder obtains preset adjustment intervals corresponding to target encoding parameters of the target video stream. For example, the preset adjustment intervals of resolution may include 4K, 2K, 1920*1080P and 720P, the preset adjustment intervals of video frame rate may be 1-25FPS, the preset adjustment intervals of I frame interval may be 1-100, the preset adjustment intervals of encoding complexity may include high, medium and low, and the preset adjustment intervals of image quality may include high, medium and low.

[0129] The encoder adjusts each target encoding parameter of the target video stream to the lower threshold of each corresponding preset adjustment interval, saves the configuration of each target encoding parameter, and generates the first processed target video stream after the configuration takes effect. Exemplarily, the resolution of the target video stream is adjusted from 1920*1080P to 720P, the video frame rate is adjusted from 25FPS to 1FPS, the I frame interval is adjusted from 50 to 1, the encoding complexity is adjusted from medium to low, and the image quality is adjusted from highest to lowest.

[0130] The first device sends the generated first processed target video stream and other video streams received in real time to the second device, and the second device decodes and plays the first processed target video stream and other video streams. By lowering the target encoding parameters of the target video stream, the bandwidth occupied by the first processed target video stream is greatly reduced compared to that before processing, thereby reducing the pressure of video stream transmission between the first device and the second device and reducing the network load.

[0131] In this embodiment, the encoder deployed on the first device obtains the current encoding parameters of the target video stream corresponding to the identifier of the target video stream based on the identifier of the target video stream, and uses the adjustable encoding parameters in the current encoding parameters of the target video stream as the target encoding parameters. The encoder obtains the preset adjustment intervals corresponding to the target encoding parameters of the target video stream, and adjusts the target encoding parameters of the target video stream to the lower thresholds of the corresponding preset adjustment intervals, saves the configuration of the target encoding parameters, and generates the first processed target video stream. By lowering the values ​​of the target encoding parameters, the bandwidth occupied by the first processed target video stream is greatly reduced, thereby reducing the network transmission load.

[0132] Hereinafter, a process in which the first device and the second device perform the second collision elimination processing on the target video stream to obtain the target video stream after the second processing is described in detail.

[0133] Fig. 9 A schematic diagram of a process of obtaining a second processed target video stream according to a video stream processing method provided in an embodiment of the present application, such as Fig. 9 As shown, in the above step S602, the first device and the second device perform a second collision elimination process on the target video stream to obtain the target video stream after the second process, including:

[0134] S901. The first device sends a key frame sequence of each video stream to the second device in advance.

[0135] Optionally, the first processor of the first device dynamically sends the I frame sequence of each video stream to the second processor of the second device in advance, and the second processor dynamically saves the I frame sequence of each video stream to the storage area of ​​the second device to provide data support for the subsequent extraction of the I frame sequence of the target video stream.

[0136] S902. The first device sends the identifier of the target video stream to the second device, discards the newly generated key frame sequence of the target video stream in real time, generates key frame sequences of the specified video stream in real time and sends them to the second device, wherein the specified video stream is a video stream other than the target video stream.

[0137] Optionally, the first processor of the first device sends the identifier of the target video stream to the second processor of the second device, the first processor discards the newly generated target video stream sent in real time by the target video stream sending source corresponding to the target video stream, and generates each I frame sequence of the designated video stream according to each newly generated other video stream sent in real time by each other video stream sending source other than the target video stream sending source, and sends it to the second processor of the second device. The I frame sequences of the designated video stream are each I frame sequence of other video streams other than the target video stream.

[0138] S903: The second device extracts the key frame sequence of the target video stream from the key frame sequences of the video streams pre-sent by the first device according to the identifier of the target video stream, and plays the video according to the key frame sequence of the target video stream and the key frame sequences of the designated video stream.

[0139] Optionally, the second processor extracts the I frame sequence of the target video stream from the I frame sequences of each video stream stored in the storage area according to the identifier of the target video stream sent by the first processor. At this time, the source of the I frame sequence of the target video stream is pre-sent by the first processor of the first device, rather than being generated and transmitted in real time by the first processor of the first device. The decoder deployed on the second device decodes the extracted I frame sequence of the target video stream and each I frame sequence of the newly generated designated video stream received in real time, respectively, and generates and plays the video corresponding to the pre-stored target video stream and the newly generated designated video stream.

[0140] Since the target video stream has a small degree of scene change, the I frame sequence of the pre-stored target video stream is roughly the same as that of the newly generated target video stream, so that the video corresponding to the pre-stored target video stream is roughly the same as that corresponding to the newly generated target video stream, that is, the video content corresponding to the target video stream remains basically unchanged. Even if the second device plays the video corresponding to the pre-stored target video stream, it will not affect the video content corresponding to the target video stream, and it can also save the first device from transmitting the I frame sequence of the newly generated target video stream to the second device in real time, reducing the amount of data transmission between the two devices, thereby alleviating network congestion.

[0141] In this embodiment, the second device dynamically receives and stores the key frame sequences of each video stream pre-sent by the first device in advance. The first device sends the identifier of the target video stream to the second device, discards the newly generated target video stream in real time, generates key frame sequences of other video streams except the target video stream, and sends them to the second processor of the second device. The second device extracts the key frame sequence of the target video stream from the pre-stored key frame sequences of each video stream based on the identifier of the target video stream, ensuring that the source of the key frame sequence of the target video stream is pre-sent by the first device. The decoder deployed on the second device decodes the extracted key frame sequence of the target video stream and the key frame sequences of the newly generated designated video stream respectively, generates and plays the video corresponding to the pre-stored target video stream and the newly generated designated video stream. The first device eliminates the need to transmit the key frame sequence of the newly generated target video stream to the second device in real time, reduces the amount of data transmission between the first device and the second device, and alleviates network congestion.

[0142] The following describes in detail the process of playing a video according to the key frame sequence of the target video stream and each key frame sequence of the designated video stream.

[0143] Fig.10 A schematic diagram of a video playback process of a video stream processing method provided in an embodiment of the present application, such as Fig.10 As shown, the step of playing the video according to the key frame sequence of the target video stream and each key frame sequence of the specified video stream in the above step S903 includes:

[0144] S1001. Import the key frame sequence of the target video stream into the decoder of the second device in a loop for decoding, and play the video corresponding to the target video stream in a loop.

[0145] Optionally, the second processor of the second device cyclically imports the I frame sequence of the target video stream extracted from the storage area into the decoder of the second device, and the decoder cyclically decodes the I frame sequence of the target video stream, cyclically generates and plays the video corresponding to the target video stream.

[0146] For example, in scenarios such as video conferencing, when the conference background remains basically unchanged, the video corresponding to the target video stream can be used to display the conference background. By cyclically importing, decoding and playing the I frame sequence used to describe the conference background in the target video stream, the conference background can be continuously displayed, and the process of the first device transmitting the I frame sequence describing the conference background in the target video stream to the second device is omitted.

[0147] S1002: Import each key frame sequence of the specified video stream into a decoder of the second device in real time for decoding, and play the video corresponding to the specified video stream in real time.

[0148] Optionally, at the same time, the second processor of the second device imports the newly generated I frame sequences of the specified video stream sent by the first processor of the first device in real time into the decoder of the second device in real time, and the decoder decodes the I frame sequences of the specified video stream in real time, and generates and plays the video corresponding to the specified video stream in real time.

[0149] For example, in scenarios such as video conferencing, the movements of participants change dynamically in real time, and each I-frame sequence of a designated video stream is an I-frame sequence of other video streams except a target video stream describing the conference background. Each I-frame sequence of a designated video stream is used to describe the movements of participants, etc. By real-time importing, decoding and playing each I-frame sequence in the designated video stream used to describe the movements of participants, the movement changes of participants can be displayed in real time, thereby ensuring the quality of the video conference and avoiding missing important information in the video conference.

[0150] In this embodiment, the second device imports the extracted pre-stored key frame sequence of the target video stream into the decoder of the second device in a loop, and the decoder decodes the key frame sequence of the target video stream in a loop, and generates and plays the video corresponding to the target video stream in a loop. At the same time, the second device imports each key frame sequence of the newly generated designated video stream sent by the first device in real time into the decoder of the second device in real time, and the decoder decodes each key frame sequence of the designated video stream in real time, and generates and plays the video corresponding to the designated video stream in real time. While reducing the amount of video transmission between the first device and the second device, the video playback quality is also guaranteed.

[0151] An embodiment of the present application further provides a video stream processing system, which includes: a first device and a second device communicatively connected to the first device, wherein an encoder is deployed on the first device, and a decoder is deployed on the second device.

[0152] The video stream processing system is used to execute the steps of the video stream processing method described in the above embodiments.

[0153] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0154] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk and other media that can store program code.

[0155] The above are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A video stream processing method, characterized in that: Applied to a video stream processing system, the video stream processing system comprises: a first device and a second device connected to the first device for communication, an encoder is deployed on the first device, and a decoder is deployed on the second device; the method comprises: The first device receives each video stream, each of the video streams includes a key frame sequence; The first device determines a target video stream according to each of the key frame sequences, wherein the target video stream is a video stream with the smallest scene change among the video streams; The first device or the first device and the second device perform collision elimination processing on the target video stream to obtain a processed target video stream, and evaluate a collision elimination effect based on the processed target video stream.

2. The method according to claim 1, characterized in that: Determining the target video stream according to each of the key frame sequences includes: Determine a key frame score of each video stream according to each of the key frame sequences, wherein the key frame score is used to characterize a characteristic change of a key frame in the video stream; The target video stream is determined according to the key frame scores of each of the video streams.

3. The method according to claim 2, characterized in that Determining the key frame score of each video stream according to each key frame sequence includes: Determining the variance of the feature change rate of each key frame of each video stream according to each of the key frame sequences; The key frame score of each video stream is determined according to the variance of the change rate of each key frame feature of each video stream and the preset weight of each key frame feature.

4. The method according to claim 3, characterized in that Determining the variance of the feature change rate of each key frame of each video stream according to each key frame sequence includes: Extracting image features from each key frame in each key frame sequence to obtain key frame features of each video stream, wherein the key frame features include pixel intensity, texture, edge, shape, and corner point; Determining the change rate of each key feature of each video stream according to each key frame feature of each video stream; According to the change rates of the key features of the video streams, the variances of the change rates of the key frame features of the video streams are determined.

5. The method according to claim 2, characterized in that: Determining the target video stream according to the key frame scores of each of the video streams includes: The key frame scores of the video streams are minimized, and the video stream corresponding to the smallest key frame score among the key frame scores of the video streams is used as the target video stream.

6. The method according to claim 1, characterized in that The first device or the first device and the second device perform collision elimination processing on the target video stream to obtain a processed target video stream, and evaluate a collision elimination effect according to the processed target video stream, including: The first device performs a first collision elimination process on the target video stream to obtain a first processed target video stream, and evaluates a collision elimination effect corresponding to the first process according to the first processed target video stream; Alternatively, the first device and the second device perform a second collision elimination process on the target video stream to obtain a second processed target video stream, and evaluate a collision elimination effect corresponding to the second process based on the second processed target video stream.

7. The method according to claim 6, characterized in that The first device performs a first collision elimination process on the target video stream to obtain a first processed target video stream, including: The first device obtains each current encoding parameter of the target video stream, and determines each target encoding parameter of the target video stream according to each current encoding parameter of the target video stream; The first device determines each preset adjustment interval corresponding to each target encoding parameter of the target video stream, and adjusts each target encoding parameter of the target video stream according to each preset adjustment interval to obtain the first processed target video stream, so that each encoding parameter of the first processed target video stream is at a lower limit threshold of each corresponding preset adjustment interval.

8. The method according to claim 6, characterized in that The first device and the second device perform a second collision elimination process on the target video stream to obtain a second processed target video stream, including: The first device sends the key frame sequence of each video stream to the second device in advance; The first device sends the identifier of the target video stream to the second device, discards the newly generated key frame sequence of the target video stream in real time, generates key frame sequences of a specified video stream in real time and sends the key frame sequences to the second device, wherein the specified video stream is a video stream other than the target video stream; The second device extracts the key frame sequence of the target video stream from the key frame sequences of each video stream sent in advance by the first device according to the identifier of the target video stream, and plays the video according to the key frame sequence of the target video stream and the key frame sequences of the designated video stream.

9. The method according to claim 8, characterized in that The playing of the video according to the key frame sequence of the target video stream and each key frame sequence of the designated video stream comprises: Importing the key frame sequence of the target video stream into the decoder of the second device in a loop for decoding, and playing the video corresponding to the target video stream in a loop; Import each key frame sequence of the designated video stream into the decoder of the second device in real time for decoding, and play the video corresponding to the designated video stream in real time.

10. A video stream processing system, characterized in that: The video stream processing system comprises: a first device and a second device connected to the first device for communication, wherein an encoder is deployed on the first device and a decoder is deployed on the second device; The video stream processing system is used to execute the steps of the video stream processing method according to any one of claims 1 to 9.

Images