Video stream processing method and device, electronic equipment and storage medium

Abstract

The invention provides a video stream processing method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining at least one video stream received by a message receiving window within a preset time, the video stream comprising a plurality of service messages; determining the type of each service message, and determining the average transmission rate of each type of service message according to the type of each service message; determining the remaining available bandwidth of the network according to the total bandwidth of the network, the average transmission rate of each type of service message and the priority of each type of service message; according to the residual available bandwidth of the network and the average transmission rate of the service messages of the preset type, determining whether to execute a collision elimination strategy; and if yes, determining a target video stream in the at least one video stream belonging to the preset type, and executing iterative collision elimination according to the target video stream until a preset collision elimination condition is met. The requirement of the service message with the high priority for the network bandwidth is ensured, and the problem of network congestion caused by collision of multiple video streams is eliminated.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and in particular, to a video stream processing method, apparatus, electronic device, and storage medium. Background Art

[0002] In the industrial private network environment, the service types are complex and diverse. Typical services that occupy a large amount of uplink bandwidth include industrial quality inspection camera data, environmental monitoring video data, video call data, and other services. When multiple services are concurrent, there is a possibility of collision between the packets of different services. Especially, the video service packets have strong periodicity, and the superposition of multiple video stream packets is more likely to generate peak traffic, imposing a huge pressure on the network bandwidth. When the concurrent rate of multiple types of services exceeds the network capacity, serious conflicts will occur, resulting in packet loss, seriously threatening the network service quality and affecting the network usage experience of industrial users.

[0003] Therefore, there is an urgent need for a method to solve the network congestion caused by the concurrency of multiple types of services in the industrial environment. Summary of the Invention

[0004] The purpose of the present application is to provide a video stream processing method, apparatus, electronic device, and storage medium to improve the accuracy of video stream processing for the deficiencies in the above-mentioned existing technologies.

[0005] To achieve the above purpose, the technical solutions adopted in the embodiments of the present application are as follows:

[0006] In a first aspect, an embodiment of the present application provides a video stream processing method, the method including:

[0007] Obtain at least one data stream received by a packet reception window within a preset time, where the video stream includes multiple service packets, and the video stream is sent from different service terminals to the packet reception window;

[0008] Determine the type of each service packet, and determine the average transmission rate of each type of service packet according to the type of each service packet;

[0009] Determine the remaining available network bandwidth according to the total network bandwidth, the average transmission rate of each type of service packet, and the priority of each type of service packet;

[0010] Determine whether to execute a collision elimination strategy according to the remaining available network bandwidth and the average transmission rate of a preset type of service packet;

[0011] If so, determine a target video stream in at least one video stream belonging to the preset type, and perform iterative collision elimination according to the target video stream until a preset collision elimination condition is reached.

[0012] Optionally, determining the type of each of the service messages and determining the average transmission rate of each type of service message based on the type of each service message includes:

[0013] Parsing each service message to obtain the transmission protocol of each service message, and determining the type of each service message based on the transmission protocol of each service message;

[0014] Determining the total number of each type of service message received by the message reception window within a preset time;

[0015] Dividing the total number of each type of service message by the length of the message reception window to obtain the average transmission rate of each type of service message.

[0016] Optionally, determining the remaining available network bandwidth based on the total network bandwidth, the average transmission rate of each type of service message, and the priority of each type of service message includes:

[0017] Determining the service message of the type with the highest priority based on each type of service message;

[0018] Determining the remaining available network bandwidth based on the average transmission rate of the service message of the type with the highest priority and the total network bandwidth.

[0019] Optionally, determining the remaining available network bandwidth based on the average transmission rate of the service message of the type with the highest priority and the total network bandwidth includes:

[0020] Subtracting the average transmission rate of the service message of the type with the highest priority from the total network bandwidth to obtain the remaining available network bandwidth.

[0021] Optionally, determining whether to execute a collision elimination strategy based on the remaining available network bandwidth and the average transmission rate of a preset type of service message includes:

[0022] If the remaining available network bandwidth is greater than the average output rate of the preset type of service message and the difference between the remaining available network bandwidth and the average output rate is greater than a first threshold, determine to execute the collision elimination strategy;

[0023] If the difference between the remaining available network bandwidth and the average output rate of the preset type of service message is less than a second threshold, determine the target video stream and close the target video stream.

[0024] Optionally, determining the target video stream among at least one video stream belonging to the preset type includes:

[0025] Performing feature extraction on each video stream corresponding to the service message of the preset type to obtain preset features of each video stream of the preset type at different moments;

[0026] Determine the expectation and variance of the feature change rate of the preset features of each video stream according to the preset features of each video stream at different times;

[0027] Take the video stream corresponding to the minimum value of the variance of the feature change rate as the target video stream.

[0028] Optionally, the performing iterative collision elimination according to the target video stream until a preset collision elimination condition is reached includes:

[0029] Adjust the encoding parameters of the target video stream, and obtain all current video streams received by the packet reception window within a preset time after adjusting the encoding parameters;

[0030] Perform collision detection on all the current video streams to determine the current collision time in each collision scenario;

[0031] Determine the current collision severity according to the current collision time in each collision scenario and the weight parameter corresponding to each collision scenario;

[0032] Determine whether the current collision severity is less than a preset threshold;

[0033] If so, iteratively adjust the encoding parameters of the target video stream until the collision severity meets the preset collision elimination condition; if not, select a new target video stream from all the current video streams, and re-perform collision elimination detection until the preset collision elimination condition is reached.

[0034] In a second aspect, an embodiment of the present application further provides a video stream processing device, and the device includes:

[0035] An obtaining module, configured to obtain at least one data stream received by a packet reception window within a preset time, where the video stream includes a plurality of service packets, and the video stream is sent by different service terminals to the packet reception window;

[0036] A determining module, configured to determine the type of each service packet, and determine the average transmission rate of each type of service packet according to the type of each service packet;

[0037] A determining module, configured to determine the remaining available network bandwidth according to the total network bandwidth, the average transmission rate of each type of service packet, and the priority of each type of service packet;

[0038] A determining module, configured to determine whether to execute a collision elimination strategy according to the remaining available network bandwidth and the average transmission rate of a preset type of service packet;

[0039] If so, determine a target video stream among at least one video stream belonging to the preset type, and perform iterative collision elimination according to the target video stream until a preset collision elimination condition is reached.

[0040] Optionally, the determining module is specifically configured to:

[0041] Parse each service message to obtain the transport protocol of each service message, and determine the type of each service message according to the transport protocol of each service message;

[0042] Determine the total number of service messages of each type received by the message reception window within a preset time;

[0043] Divide the total number of service messages of each type by the length of the message reception window to obtain the average transmission rate of each type of service message.

[0044] Optionally, the determining module is specifically configured to:

[0045] Determine the service message of the type with the highest priority according to each type of service message;

[0046] Determine the remaining available network bandwidth according to the average transmission rate of the service message of the type with the highest priority and the total network bandwidth.

[0047] Optionally, the determining module is specifically configured to:

[0048] Subtract the average transmission rate of the service message of the type with the highest priority from the total network bandwidth to obtain the remaining available network bandwidth.

[0049] Optionally, the determining module is specifically configured to:

[0050] If the remaining available network bandwidth is greater than the average output rate of the service message of the preset type and the difference between the remaining available network bandwidth and the average output rate is greater than a first threshold, determine to execute a collision elimination strategy;

[0051] If the difference between the remaining available network bandwidth and the average output rate of the service message of the preset type is less than a second threshold, determine the target video stream and close the target video stream.

[0052] Optionally, the determining module is specifically configured to:

[0053] Extract features from each video stream corresponding to the service message of the preset type to obtain preset features of each video stream of the preset type at different times;

[0054] According to the preset features of each video stream of the preset type at different times, determine the expectation and variance of the feature change rate of the preset features of each video stream;

[0055] Use the video stream corresponding to the minimum value of the variance of the feature change rate as the target video stream.

[0056] Optionally, the determining module is specifically configured to:

[0057] Adjust the encoding parameters of the target video stream, and obtain all the current video streams received by the packet reception window after adjusting the encoding parameters within a preset time;

[0058] Perform collision detection on all the current video streams, and determine the current collision time under each collision scenario;

[0059] Determine the current collision severity according to the current collision time under each collision scenario and the weight parameter corresponding to each collision scenario;

[0060] Determine whether the current collision severity is less than a preset threshold;

[0061] If so, iteratively adjust the encoding parameters of the target video stream until the collision severity meets the preset collision elimination condition; if not, select a new target video stream from all the current video streams, and re-perform collision elimination detection until the preset collision elimination condition is reached.

[0062] In a third aspect, an embodiment of the present application further provides an electronic device, including: a processor, a storage medium, and a bus. The storage medium stores program instructions executable by the processor. When the application program runs, the processor communicates with the storage medium through the bus, and the processor executes the program instructions to perform the steps of the video stream processing method described in the first aspect above.

[0063] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program is read and executed to perform the steps of the video stream processing method described in the first aspect above.

[0064] The beneficial effects of the present application are:

[0065] A video stream processing method, apparatus, electronic device, and storage medium provided by the present application obtain at least one video stream received by a packet reception window within a preset time; determine the types of each service packet, and determine the average transmission rate of each type of service packet according to the types of each service packet; determine the remaining available network bandwidth according to the total network bandwidth, the average transmission rate of each type of service packet, and the priority of each type of service packet; determine whether to execute a collision elimination strategy according to the remaining available network bandwidth and the average transmission rate of a preset type of service packet; if so, determine a target video stream among at least one video stream belonging to the preset type, and perform iterative collision elimination according to the target video stream until a preset collision elimination condition is reached. By identifying the types of service packets and determining the remaining available network bandwidth based on the priority of each type of service packet and the average transmission rate of each type of service packet, the priority of each type of service packet is considered when calculating the remaining available network bandwidth, which can ensure the network bandwidth requirements of high-priority service packets. At the same time, the priority of each type of service packet can be adjusted based on user needs, thereby increasing the flexibility of service guarantee; and when performing collision elimination, iterative collision elimination can be performed according to the target video stream, which can eliminate network congestion problems caused by multi-video stream collisions and greatly reduce the network load. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the accompanying drawings required for the embodiments. It should be understood that the following drawings only show some embodiments of the present application and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0067] Figure 1 It is an exemplary scenario schematic diagram provided by an embodiment of the present application;

[0068] Figure 2 It is a flowchart of a video stream processing method provided by an embodiment of the present application;

[0069] Figure 3 It is a flowchart of another video stream processing method provided by an embodiment of the present application;

[0070] Figure 4 It is a flowchart of yet another video stream processing method provided by an embodiment of the present application;

[0071] Figure 5 It is a flowchart of a method for determining a target video stream provided by an embodiment of the present application;

[0072] Figure 6Flowchart of a method for collision elimination provided by an embodiment of the present application;

[0073] Figure 7 Schematic diagram of a device for a video stream processing method provided by an embodiment of the present application;

[0074] Figure 8 Block diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0075] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. It should be understood that the accompanying drawings in the present application are only for the purposes of illustration and description, and are not used to limit the protection scope of the present application. In addition, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in the present application illustrate the operations implemented according to some embodiments of the present application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and the steps without logical context may be reversed or implemented simultaneously. In addition, those skilled in the art may add one or more other operations to the flowchart or remove one or more operations from the flowchart under the guidance of the content of the present application.

[0076] In addition, the described embodiments are only some embodiments of the present application, rather than all embodiments. The components of the embodiments of the present application described and illustrated in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application claimed, but merely represents selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts belong to the scope of protection of the present application.

[0077] It should be noted that the term "including" will be used in the embodiments of the present application to indicate the existence of the features stated thereafter, but does not exclude the addition of other features.

[0078] Industrial inspection cameras have multiple protocols, such as the Gigabit Ethernet Vision (GigE) protocol, the Universal Serial Bus (USB) protocol, CamLink, etc. Among them, the most common and representative one is the GigE Vision protocol. This standard is based on the UDP protocol and uses the GigE Vision Control Protocol (GVCP) and the GigE Vision Stream Protocol (GVSP) to configure the camera and transmit data streams. This protocol introduces the concept of channels between the sender and the receiver, including one control channel and multiple stream transmission channels. In this application, the impact of GVSP video stream transmission packets on network bandwidth is considered.

[0079] The output data of video surveillance devices are encoded video frames, mainly including I-frames and P-frames. Subsequently, they are packed into Real-time Transport Protocol (RTP) packets and further encapsulated into IP packets for transmission over the network to the client. Among them, I-frames are key frames that describe the details of the image background and moving objects, are generated by independent encoding, and a complete image can be reconstructed only using the I-frame data during decoding, with a relatively low compression ratio. P-frames are forward prediction encoded frames, and during encoding (decoding), data is compressed (decompressed) mainly based on the differences from the adjacent previous frame (I-frame or P-frame) image, with a relatively high compression ratio. One or more P-frames follow an I-frame, and they are sent cyclically in this way. Among them, multiple frames between two adjacent I-frames (including one I-frame and multiple P-frames following it immediately) form a Group of Pictures (GOP). I-frames and P-frames generally consist of multiple consecutive IP data packets. Since the size of I-frames is much larger than that of P-frames, this application focuses on I-frames in RTP packets.

[0080] Optionally, the video stream processing method provided in the embodiments of this application is applied to an electronic device, which can be, for example, a terminal device with computing and processing capabilities and a display function such as a mobile phone, a tablet computer, a laptop computer, a handheld computer, a desktop computer, etc., or it can also be a server. Specifically, it can be applied to an application program in a terminal device, such as: an application (APP) on a mobile phone, an application system on a computer, etc.

[0081] Next, the specific implementation process of the video stream processing provided in the embodiments of this application will be specifically explained.

[0082] Figure 1 This is an exemplary scenario diagram provided for the embodiments of this application, as Figure 1As shown, this scenario includes a video stream receiver and at least one service terminal. Among them, the receiver and the service terminal can be terminal devices with computing and processing capabilities and display functions such as mobile phones, tablet computers, laptop computers, personal digital assistants, desktop computers, etc. Among them, the service terminal can also be terminal devices such as industrial quality inspection cameras and video surveillance devices. The video stream receiver can receive the video streams sent by each service terminal and process the received video streams by using the video stream processing method provided in this application to eliminate the network congestion problem caused by multi-video stream collisions, greatly reducing the network load; improving the operability of the multi-video stream collision elimination technology.

[0083] Figure 2 It is a schematic flowchart of a video stream processing method provided by an embodiment of this application. The execution subject of this method is the aforementioned video stream receiver. As Figure 2 shown, this method includes:

[0084] S101. Obtain at least one video stream received by the packet reception window within a preset time.

[0085] Among them, each video stream includes multiple service packets. The video streams are sent from different service terminals to the packet reception window, and the packet reception window refers to the reception window on the video stream receiver.

[0086] Optionally, the service terminal can be terminal devices such as industrial quality inspection cameras, video surveillance devices, mobile phones, tablets, etc. When the industrial quality inspection camera generates a quality inspection video stream during industrial quality inspection and sends the generated quality inspection video stream to the video stream receiver; when the video surveillance device generates a surveillance video stream during monitoring and sends the generated surveillance video stream to the video stream receiver; when the user makes a video call on the mobile phone, the generated call video stream can be sent to the video stream receiver. And when the service terminal sends a video stream to the video stream receiver, it can also include information such as the Internet Protocol Address (IP address) of the service terminal, the Media Access Control Address (MAC address) of the service terminal, and the port number of the service terminal. Among them, the quality inspection video stream can include multiple service packets, the surveillance video stream can include multiple service packets, and the call video stream can include multiple service packets.

[0087] Exemplarily, within a preset time period, the video receiving end can receive the quality inspection video stream 1 generated by the quality inspection camera 1, the quality inspection video stream 2 generated by the quality inspection camera 2, the quality inspection video stream 3 generated by the quality inspection camera 3, the monitoring video stream 1 sent by the video monitoring device 1, the monitoring video stream 2 sent by the video monitoring device 2, the monitoring video stream 3 sent by the video monitoring device 3, the call video stream 1 sent by the user mobile phone 1, the call video stream 2 sent by the user mobile phone 2, and the call video stream 3 sent by the user mobile phone 3.

[0088] S102. Determine the types of each service message, and determine the average transmission rate of each type of service message according to the types of each service message.

[0089] Among them, the types of service messages sent by industrial quality inspection cameras are different from those sent by other service terminals. Therefore, the types of each service message can be determined by using a preset method based on the service messages in the video streams sent by each service terminal received. And after determining the type of the service message, the average transmission rate of each type of service message can also be determined according to the type of each service message and each service message by using a preset method.

[0090] Exemplarily, it can be determined that the type of the service message in the quality inspection video stream 1 is type A, the type of the service message in the quality inspection video stream 2 is type A, and the type of the service message in the quality inspection video stream 3 is type A; the type of the service message in the monitoring video stream 1 is type B, the type of the service message in the monitoring video stream 2 is type B, and the type of the service message in the monitoring video stream 3 is type B; the type of the service message in the call video stream 1 is type B, the type of the service message in the call video stream 3 is type B, and the type of the service message in the call video stream 3 is type B. Determine the average transmission rate of the service message of type A and the average transmission rate of the service message of type B.

[0091] S103. Determine the remaining available network bandwidth according to the total network bandwidth, the average transmission rate of each type of service message, and the priority of each type of service message.

[0092] Among them, the total network bandwidth is the total amount of data that the network can transmit within a unit time. In the network, it represents the amount of information transmitted from one end to the other within a specified time, that is, the data transmission rate. The average transmission rate of each type of service message is obtained from the step S102 described above, and the priority of each type of service message is pre-configured by the user. When the priority configuration function is turned on, the priority of each service message configured by the user shall prevail; when the priority configuration function is turned off, the priority of the service message of a preset type shall be taken as the highest priority.

[0093] Exemplarily, the remaining available network bandwidth can be determined using a preset method based on the total network bandwidth, the average transmission rate of service packets of type A, the average transmission rate of service packets of type B, and the priorities of service packets of each type.

[0094] It is worth noting that when configuring priorities, users can perform more fine-grained priority configuration on the basis of the priority configuration of service packets of each type. Specifically, the priorities of service packets in the video streams sent by each service terminal can also be set according to the information of each service terminal. For example, the priority of the quality inspection video stream 1 of the quality inspection camera 1 is higher than that of the quality inspection video stream 2 of the quality inspection camera 2, and the priority of the quality inspection video stream 2 of the quality inspection camera 2 is higher than that of the quality inspection video stream 3 of the quality inspection camera 3.

[0095] S104. Determine whether to execute a collision elimination strategy according to the remaining available network bandwidth and the average transmission rate of service packets of a preset type.

[0096] Among them, the average transmission rate of service packets of the preset type is obtained by calculating in step S102 described above. Among them, the collision elimination strategy is based on service packets of the preset type for collision elimination, and can eliminate the network congestion problem caused by the collision of multiple service packets during transmission in the network.

[0097] Exemplarily, if the service packets of the preset type are service packets of type B, then it can be determined whether to execute a collision elimination strategy using a preset method according to the remaining available network bandwidth and the average transmission rate of service packets of type B.

[0098] Optionally, if it is determined to execute the collision elimination strategy according to the remaining available network bandwidth and the average transmission rate of service packets of the preset type, then execute step S105 below. If it is determined not to execute the collision elimination strategy according to the remaining available network bandwidth and the average transmission rate of service packets of the preset type, then execute step S106 below.

[0099] S105. Determine the target video stream in at least one video stream belonging to the preset type, and perform iterative collision elimination according to the target video stream until a preset collision elimination condition is reached.

[0100] Optionally, the determined target video stream is the video stream corresponding to the service packets of the preset type in the foregoing S104.

[0101] Since the video stream contains service packets, that is, the type of the service packet is also the type of the video stream. For example, the type of the video stream corresponding to the service packets of type B is also type B, and the type of the video stream corresponding to the service packets of type A is also type A.

[0102] Optionally, in this embodiment, when considering the collision of video streams, it is the collision between video streams of a preset type. Then, when it is determined to execute the collision elimination strategy, a target video stream can be determined from the video streams of the preset type using a preset method. After determining the target video stream, iterative collision elimination is performed until a preset collision elimination condition is reached.

[0103] S106. Close the target video stream.

[0104] Optionally, when it is determined not to execute the collision elimination strategy, the target video stream is directly closed. Here, closing the target video stream means that the service terminal sending the target video stream no longer sends the video stream to the video stream receiving end.

[0105] In this embodiment, at least one video stream received by the packet reception window within a preset time is obtained; the types of each service packet are determined, and the average transmission rate of each type of service packet is determined according to the types of each service packet; according to the total network bandwidth, the average transmission rate of each type of service packet, and the priority of each type of service packet, the remaining available network bandwidth is determined; according to the remaining available network bandwidth and the average transmission rate of the service packets of the preset type, it is determined whether to execute the collision elimination strategy; if so, a target video stream is determined from at least one video stream belonging to the preset type, and iterative collision elimination is performed according to the target video stream until a preset collision elimination condition is reached. By identifying the types of service packets and determining the remaining available network bandwidth based on the priority of each type of service packet and the average transmission rate of each type of service packet, the priority of each type of service packet is considered when calculating the remaining available network bandwidth, which can ensure the network bandwidth requirements of service packets with high priority. At the same time, the priority of each type of service packet can be adjusted based on user needs, thereby increasing the flexibility of service guarantee; and when performing collision elimination, iterative collision elimination can be performed according to the target video stream, which can eliminate the network congestion problem caused by multi-video stream collision and greatly reduce the network load.

[0106] Figure 3 It is a schematic flowchart of another video stream processing method provided by an embodiment of this application. As Figure 3 shown, the above S102. Determine the types of each service packet, and determine the average transmission rate of each type of service packet according to the types of each service packet, may include:

[0107] S201. Analyze each service packet to obtain the transmission protocol of each service packet, and determine the type of each service packet according to the transmission protocol of each service packet.

[0108] Specifically, after receiving each video stream, the service packets in the video stream can be parsed to identify the type of the service packets. For example, the transmission protocols of the service packets in an industrial quality inspection camera include the GVSP protocol and the GVCP protocol. Since this application focuses on the impact of GVSP data stream transmission packets on network bandwidth, the type of the service packets in the industrial quality inspection camera is determined to be the GVSP type. For the video streams sent by other service terminals except the industrial quality inspection camera, such as the monitoring video stream sent by a video monitoring device, the service packets in the monitoring video stream can be parsed to determine that the transmission protocol of the monitoring video stream is the RTP protocol, and then the type of the service packets in the monitoring video stream is determined to be the RTP type. The RTP packets include I-frames and P-frames. Since the size of the I-frame is much larger than that of the P-frame, in this embodiment, the impact of the I-frame in the RTP packets on network bandwidth is concerned. Then, the types of the determined service packets may include the GVSP type and the RTP type.

[0109] S202. Determine the total number of service packets of each type received by the packet reception window within a preset time.

[0110] Specifically, after identifying the type of the service packets in each video stream, the number of service packets of each type within the preset time can be counted. Specifically, the total number of GVSP-type service packets and the total number of RTP-type service packets received by the packet reception window within the preset time can be obtained.

[0111] Exemplarily, for example, the number of GVSP-type service packets in the quality inspection video stream 1 received within the preset time is x1; the number of GVSP-type service packets in the quality inspection video stream 2 is x2; the number of GVSP-type service packets in the quality inspection video stream 3 is x3; the number of RTP-type service packets in the monitoring video stream 1 is y1; the number of RTP-type service packets in the monitoring video stream 2 is y2; the number of RTP-type service packets in the monitoring video stream 3 is y3; the number of RTP-type service packets in the call video stream 1 is y4; the number of RTP-type service packets in the call video stream 2 is y5; the number of RTP-type service packets in the call video stream 3 is y6. Then, the number of GVSP-type service packets received within the preset time, x = x1 + x2 + x3; the number of RTP-type service packets received is y = y1 + y2 + y3 + y4 + y5 + y6.

[0112] S203. Divide the total number of service packets of each type by the length of the packet reception window to obtain the average transmission rate of service packets of each type.

[0113] Wherein, the length of the packet reception window is a time window length.

[0114] Specifically, divide the total number of service packets of the GVSP type by the length of the packet reception window to obtain the average transmission rate of the service packets of the GVSP type; divide the total number of service packets of the RTP type by the length of the packet reception window to obtain the average transmission rate of the service packets of the RTP type.

[0115] Exemplarily, divide x by the length of the packet reception window to obtain the average transmission rate of the service packets of the GVSP type; divide y by the length of the packet reception window to obtain the average transmission rate of the service packets of the RTP type.

[0116] In this embodiment, it is possible to identify the types of service packets in the video stream sent by each service terminal, obtain the types of each service packet, and calculate the transmission rate of each type of service packet for subsequent calculations.

[0117] Figure 4 It is a schematic flowchart of another video stream processing method provided by an embodiment of the present application. As Figure 4 shown, the above S103, determining the remaining available network bandwidth according to the total network bandwidth, the average transmission rate of each type of service packet, and the priority of each type of service packet, may include:

[0118] S301. Determine the service packet of the type with the highest priority according to each type of service packet.

[0119] Optionally, when the priority configuration function is enabled, the priority of each type of service packet configured is used as the standard to determine the service packet of the type with the highest priority from the configured service packets of each type. For example, the service packet of the type with the highest priority is the service packet of the RTP type.

[0120] Optionally, when the priority configuration function is disabled, the priority of the service packet of the default preset type is set as the highest priority. For example, if the preset type of service type is the GVSP type, then when the priority configuration function is disabled, the priority of the service packet of the GVSP type is defaulted as the highest priority at this time.

[0121] Optionally, when the granularity of priority division is finer, the priority of the video stream sent by the service terminal is configured. For example, the priority of the quality inspection camera 1 is the highest. Then the service packet with the highest priority is the service packet of the quality inspection video stream sent by the quality inspection camera 1.

[0122] S302. Determine the remaining available bandwidth according to the average transmission rate of the service packet of the type with the highest priority and the total network bandwidth.

[0123] Exemplarily, if the service packet with the highest priority is of the RTP type, the remaining available bandwidth can be determined according to the average transmission rate of the service packet of the RTP type and the total network bandwidth; if the service packet with the highest priority is of the GVSP type, the remaining available bandwidth can be determined according to the average transmission rate of the service packet of the GVSP type and the total network bandwidth. If the service packet with the highest priority is the service packet of the video stream sent by the quality inspection camera 1, the average transmission rate of the service packets in the quality inspection video stream 1 received from the quality inspection camera 1 within a preset time can be calculated first, and then the remaining available bandwidth can be determined according to the average transmission rate of the service packets of the quality inspection camera 1 and the total network bandwidth.

[0124] Optionally, the above S302, determining the remaining available bandwidth according to the average transmission rate of the service packet of the type with the highest priority and the total network bandwidth, may include:

[0125] Specifically, the total network bandwidth can be subtracted by the average transmission rate of the service packet of the type with the highest priority to obtain the remaining available network bandwidth.

[0126] Exemplarily, if the service packet with the highest priority is of the RTP type, the total network bandwidth is subtracted by the average transmission rate of the service packet of the RTP type to obtain the remaining available bandwidth; if the service packet with the highest priority is of the GVSP type, the total network bandwidth is subtracted by the average transmission rate of the service packet of the GVSP type to obtain the remaining available bandwidth. If the service packet with the highest priority is the service packet of the video stream sent by the quality inspection camera 1, the total network bandwidth is subtracted by the average transmission rate of the service packets of the quality inspection camera 1 and the total network bandwidth to obtain the remaining available bandwidth.

[0127] In this embodiment, when determining the remaining available network bandwidth, by using the total network bandwidth minus the average transmission rate of the service packet of the type with the highest priority, the network usage requirements of high-priority services are preferentially guaranteed.

[0128] Optionally, the above S104, determining whether to execute the collision elimination strategy according to the remaining available network bandwidth and the average transmission rate of the service packet of the preset type, may include:

[0129] Specifically, if the remaining available network bandwidth is greater than the average output rate of the service packet of the preset type and the difference between the remaining available network bandwidth and the average output rate of the service packet of the preset type is greater than the first threshold, it is determined to execute the collision elimination strategy. Among them, the service packet of the preset type can be the average output rate of the service packet of the I-frame, and the average output rate of the service packet of the I-frame is the average transmission rate of the service packet of the RTP type. Among them, the first threshold can be set based on user requirements.

[0130] Optionally, when the remaining available bandwidth of the network is greater than the average output rate of the service packets of a preset type and the difference between the remaining available bandwidth of the network and the average output rate of the service packets of the preset type is greater than a first threshold, it can be indicated that the remaining amount of the remaining available bandwidth of the network is relatively large. At this time, the collision elimination strategy can be executed.

[0131] Optionally, if the difference between the remaining available bandwidth of the network and the average output rate of the service packets of a preset type is less than a second threshold, the target video stream is determined and the target video stream is closed. Wherein, the second threshold is less than the first threshold. When the difference between the remaining available bandwidth of the network and the average output rate of the service packets of the preset type is less than the second threshold, it indicates that the remaining available bandwidth of the network is already operating close to full load. At this time, the peak rate of a slight video stream collision will surely exceed the upper limit of the total network bandwidth. Therefore, after determining the target video stream, the target video stream is directly closed, so as to ensure the network usage requirements of other video services.

[0132] Figure 5 The flowchart of a method for determining a target video stream provided by an embodiment of the present application is as Figure 5 shown. The above S105, determining the target video stream among at least one video stream belonging to a preset type, may include:

[0133] S401: Extract features from each video stream corresponding to the service packets of the preset type to obtain the preset features of each video stream of the preset type at different moments.

[0134] Optionally, a convolutional neural network (CNN) can be used to extract image features from each video stream of the preset type. The extracted features may include pixel intensity, texture, edge, shape, and corner, etc. Among them, the preset feature may refer to one of the aforementioned pixel intensity, texture, edge, shape, and corner, and can be denoted as the j-th feature. Among them, the video stream of the preset type may be an RTP video stream.

[0135] Specifically, for the t-th moment, the j-th feature of the i-th RTP video stream image is f i j (t). At the next moment (t + 1), the j-th feature of the i-th video stream image is f i j (t + 1).

[0136] S402: Determine the expectation and variance of the feature change rate of the preset features of each video stream according to the preset features of each video stream of the preset type at different moments.

[0137] Specifically, the increment of the j-th feature of the i-th video stream from the t-th moment to the (t + 1)-th moment can be calculated first, Δf i j(t), the feature change rate of each video stream image within a preset time can be calculated through the formula Δf i j (t) = f i j (t + 1) - f i j (t). After that, calculate the feature change rate of each video stream image within the preset time. If the feature increment of the j-th feature of the i-th video stream can be calculated as N within the preset time, the n-th feature increment can be represented by Δf i j (n), where n ∈ (1, N). The expectation and variance of the feature change rate of the j-th feature of the i-th video stream are obtained by using the following formula (1) and formula (2).

[0138]

[0139] Among them, Ave_Δf i j is the expectation of the feature change rate of the j-th feature of the i-th video stream, and Δf i j (n) is the n-th feature increment of the j-th feature of the i-th video stream, and N is the number of feature increments.

[0140]

[0141] Among them, Var_Δf i j is the variance of the feature change rate of the j-th feature of the i-th video stream, Ave_Δf i j is the expectation of the feature change rate of the j-th feature of the i-th video stream, and Δf i j (n) is the n-th feature increment of the j-th feature of the i-th video stream, and N is the number of feature increments.

[0142] S403. Use the video stream corresponding to the minimum value of the variance of the feature change rate as the target video stream.

[0143] Exemplarily, if the variance of the feature change rate of the j-th feature of the 2nd RTP type video stream is the smallest, then use the 2nd RTP type video stream as the target video stream.

[0144] Figure 6 is the flowchart of a collision elimination method provided by an embodiment of the present application. As Figure 6 shown, in the above S105, performing iterative collision elimination according to the target video stream until the preset collision elimination condition is reached may include:

[0145] S501. Adjust the encoding parameters of the target video stream.

[0146] Among them, the encoding parameters may include resolution, video frame rate, I-frame interval, upper bitrate limit, encoding complexity, and image quality, etc. Specifically, when first adjusting the encoding parameters of the target video stream, the values of the adjustable encoding parameters of the target video stream can be directly adjusted to the minimum value that the encoding parameters can be adjusted to.

[0147] S502. Obtain all the current video streams received by the packet reception window after adjusting the encoding parameters within a preset time.

[0148] Optionally, after adjusting the encoding parameters of the target video stream, return the adjusted encoding parameters to the service terminal that sent the target video stream. When the service terminal receives the adjusted encoding parameters, use the adjusted encoding parameters to resend the video stream to the packet reception window.

[0149] Optionally, all the current video streams obtained may include the target video stream sent after adjusting the encoding parameters and may also include video streams sent by other service terminals.

[0150] S503. Perform collision detection on all the current video streams to determine the current collision time in each collision scenario.

[0151] Among them, the collision detection refers to the collision detection of I-frames in RTP-type service packets. Each collision scenario refers to the scenario of m-frame collision. If at most M-frame collision occurs in the current packet reception window, the value of M is [1, 2, 3, …, M], and a corresponding weight coefficient is set for each m-frame collision. The current collision time in each collision scenario can be used where m refers to the collision scenario of m I-frame collisions, and k is the number of times the m-frame collision scenario occurs, that is is the duration of the kth collision that occurs in the m-frame collision scenario.

[0152] Optionally, when determining the current collision time in each collision scenario, the collision flow combination in that collision scenario can also be determined. For example, during this collision detection, it can be detected that there is 1 five-frame collision, 3 three-frame collisions, and 4 two-frame collisions. Among them, the current collision time of 1 five-frame collision can be used to represent, and the collision flow combination is flow_ID = [3, 4, 5, 6, 8], where flow_ID is the encoding number of the video stream, that is, in the scenario of 1 five-frame collision, the collision flows are video stream No. 3, video stream No. 4, video stream No. 5, video stream No. 6, and video stream No. 8; the current collision time of 3 three-frame collisions is the sum of, and the collision flow combination can be flow_ID = [2, 5, 6], [5, 8, 9], [2, 5, 6]; the current collision time of 4 two-frame collisions is For the sum, the collision flow combination can be flow ID = [1, 2], [4, 7], [8, 9], [1, 2].

[0153] S504. Determine the current collision severity according to the current collision time in each collision scenario and the weight parameter corresponding to each collision scenario.

[0154] Specifically, the current collision severity can be calculated through the following formula (III).

[0155]

[0156] Where T m is the current collision time in each collision scenario, and h is the weight parameter corresponding to each collision scenario.

[0157] S505. Determine whether the current collision severity is less than a preset threshold.

[0158] Optionally, if the current collision severity is less than the preset threshold, it indicates that the adjustment of the encoding parameters of the target video stream this time is effective, and then execute S506 below; if the current collision severity is greater than the preset threshold, it indicates that the adjustment of the encoding parameters of the target video stream is ineffective, and then execute S507 below.

[0159] S506. Determine whether the current collision severity meets the preset collision elimination condition.

[0160] Optionally, if the preset collision elimination condition is met, then end; if the preset collision elimination condition is not met, then readjust the encoding parameters of the target video stream and return to execute S502. Among them, the preset collision elimination condition can refer to that the difference between the current collision severity and the preset threshold is less than the third threshold.

[0161] S507. Re-select a new target video stream from all current video streams.

[0162] Specifically, the method of re-selecting a new target video stream from all current video streams is the same as the steps of S401 to S403 described above, and will not be elaborated here. After determining the new target video stream, return to execute step S501.

[0163] Figure 7 is a schematic diagram of a device for a video stream processing method provided by an embodiment of the present application, as Figure 7 shown, the device includes:

[0164] An obtaining module 601, configured to obtain at least one data stream received by a packet reception window within a preset time, the video stream includes a plurality of service packets, and the video stream is sent from different service terminals to the packet reception window;

[0165] A determination module 602, configured to determine the type of each of the service packets, and determine the average transmission rate of each type of service packet according to the type of each service packet;

[0166] A determination module 602, configured to determine the remaining available network bandwidth according to the total network bandwidth, the average transmission rate of each type of service packet, and the priority of each type of service packet;

[0167] A determination module 602, configured to determine whether to execute a collision elimination strategy according to the remaining available network bandwidth and the average transmission rate of a service packet of a preset type;

[0168] If so, determine a target video stream among at least one video stream belonging to the preset type, and perform iterative collision elimination according to the target video stream until a preset collision elimination condition is reached.

[0169] Optionally, the determination module 602 is specifically configured to:

[0170] Parse each service packet to obtain the transmission protocol of each service packet, and determine the type of each service packet according to the transmission protocol of each service packet;

[0171] Determine the total number of each type of service packet received by the packet reception window within a preset time;

[0172] Divide the total number of each type of service packet by the length of the packet reception window to obtain the average transmission rate of each type of service packet.

[0173] Optionally, the determination module 602 is specifically configured to:

[0174] Determine the service packet of the type with the highest priority according to each type of service packet;

[0175] Determine the remaining available network bandwidth according to the average transmission rate of the service packet of the type with the highest priority and the total network bandwidth.

[0176] Optionally, the determination module 602 is specifically configured to:

[0177] Subtract the average transmission rate of the service packet of the type with the highest priority from the total network bandwidth to obtain the remaining available network bandwidth.

[0178] Optionally, the determination module 602 is specifically configured to:

[0179] If the remaining available network bandwidth is greater than the average output rate of the service packet of the preset type and the difference between the remaining available network bandwidth and the average output rate is greater than a first threshold, determine to execute a collision elimination strategy;

[0180] If the difference between the remaining available bandwidth of the network and the average output rate of the service packets of the preset type is less than the second threshold, determine the target video stream and close the target video stream.

[0181] Optionally, the determining module 602 is specifically configured to:

[0182] Extract features of each video stream corresponding to the service packets of the preset type to obtain preset features of each video stream of the preset type at different times;

[0183] According to the preset features of each video stream of the preset type at different times, determine the expectation and variance of the feature change rate of the preset features of each video stream;

[0184] Use the video stream corresponding to the minimum value of the variance of the feature change rate as the target video stream.

[0185] Optionally, the determining module 602 is specifically configured to:

[0186] Adjust the encoding parameters of the target video stream, and obtain all current video streams received by the packet reception window after adjusting the encoding parameters within a preset time;

[0187] Perform collision detection on all the current video streams to determine the current collision time in each collision scenario;

[0188] According to the current collision time in each collision scenario and the weight parameter corresponding to each collision scenario, determine the current collision severity;

[0189] Determine whether the current collision severity is less than a preset threshold;

[0190] If so, iteratively adjust the encoding parameters of the target video stream until the collision severity meets the preset collision elimination condition; if not, select a new target video stream from all the current video streams and re - perform the collision elimination detection until the preset collision elimination condition is reached.

[0191] Figure 8 It is a structural block diagram of an electronic device 700 provided by an embodiment of the present application. This electronic device can be, for example, the video stream processing described in the foregoing embodiment. As Figure 8 shown, the electronic device may include: a processor 701 and a memory 702.

[0192] Optionally, a bus 703 may also be included. Here, the memory 702 is used to store machine-readable instructions executable by the processor 701. When the electronic device 700 runs, the processor 701 communicates with the memory 702 via the bus 703. When the machine-readable instructions are executed by the processor 701, the method steps in the above method embodiments are performed.

[0193] An embodiment of the present application also provides a computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program is run by a processor, the method steps in the above method embodiments of the video stream processing method are performed.

[0194] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described systems and devices can refer to the corresponding processes in the method embodiments, which will not be elaborated in this application. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation. For another example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some communication interfaces. The indirect couplings or communication connections of the devices or modules can be in electrical, mechanical, or other forms.

[0195] In addition, in each embodiment of the present application, the functional units can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, RandomAccess Memory), magnetic disks, or optical discs that can store program codes.

[0196] The above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application.

Claims

1. A video stream processing method, characterized in that: The method comprises: Acquire at least one video stream received by a message receiving window within a preset time, wherein the video stream includes multiple service messages, and the video stream is sent to the message receiving window by different service terminals; Determining the type of each of the service messages, and determining the average transmission rate of each type of service message according to the type of each service message; Determine the remaining available network bandwidth based on the total network bandwidth, the average transmission rate of each type of service message, and the priority of each type of service message; Determining whether to execute a collision elimination strategy according to the remaining available bandwidth of the network and the average transmission rate of service messages of a preset type; If so, a target video stream among at least one video stream belonging to the preset type is determined, and iterative collision elimination is performed according to the target video stream until a preset collision elimination condition is met.

2. The video stream processing method according to claim 1, characterized in that: The determining the type of each of the service messages, and determining the average transmission rate of each type of service message according to the type of each service message, includes: Parse each service message to obtain the transmission protocol of each service message, and determine the type of each service message according to the transmission protocol of each service message; Determine the total number of service messages of each type received by the message receiving window within a preset time; The total number of service messages of each type is divided by the length of the message receiving window to obtain the average transmission rate of the service messages of each type.

3. The video stream processing method according to claim 1, characterized in that: Determining the remaining available network bandwidth according to the total network bandwidth, the average transmission rate of each type of service message, and the priority of each type of service message includes: Determine the type of service message with the highest priority according to each type of service message; The remaining available network bandwidth is determined based on the average transmission rate of the service packets of the highest priority type and the total network bandwidth.

4. The video stream processing method according to claim 3, characterized in that: Determining the remaining available network bandwidth according to the average transmission rate of the service message of the highest priority type and the total network bandwidth includes: The remaining available bandwidth of the network is obtained by subtracting the average transmission rate of the service message of the highest priority type from the total network bandwidth.

5. The video stream processing method according to claim 1, characterized in that: The determining whether to execute the collision elimination strategy according to the remaining available bandwidth of the network and the average transmission rate of the preset type of service messages includes: If the remaining available network bandwidth is greater than the average output rate of the service packets of the preset type and the difference between the remaining available network bandwidth and the average output rate is greater than a first threshold, determining to execute a collision elimination strategy; If the difference between the remaining available bandwidth of the network and the average output rate of the service messages of the preset type is less than a second threshold, the target video stream is determined and closed.

6. The video stream processing method according to claim 1, characterized in that: The determining of a target video stream among at least one video stream belonging to the preset type comprises: Extracting features of each video stream corresponding to the service message of the preset type to obtain preset features of each video stream of the preset type at different times; Determine the expectation and variance of the feature change rate of the preset features of each video stream according to the preset features of each video stream of the preset type at different times; The video stream corresponding to the minimum value of the variance of the feature change rate is taken as the target video stream.

7. The video stream processing method according to claim 1, characterized in that: The performing iterative collision elimination according to the target video stream until a preset collision elimination condition is reached includes: Adjusting the encoding parameters of the target video stream, and obtaining all current video streams received by the message receiving window within a preset time after the encoding parameters are adjusted; Performing collision detection on all current video streams to determine the current collision time under each collision scenario; Determine the current collision severity according to the current collision time under each collision scenario and the weight parameter corresponding to each collision scenario; Determining whether the current collision severity is less than a preset threshold; If so, the encoding parameters of the target video stream are iteratively adjusted until the collision severity meets the preset collision elimination condition; if not, a new target video stream is selected from all the current video streams, and the collision elimination detection is re-executed until the preset collision elimination condition is met.

8. A video stream processing device, characterized in that: include: An acquisition module, used to acquire at least one video stream received by a message receiving window within a preset time, wherein the video stream includes multiple service messages, and the video stream is sent to the message receiving window by different service terminals; A determination module, used to determine the type of each of the service messages, and determine the average transmission rate of each type of service message according to the type of each service message; A determination module, used to determine the remaining available network bandwidth according to the total network bandwidth, the average transmission rate of each type of service message and the priority of each type of service message; A determination module, used to determine whether to execute a collision elimination strategy according to the remaining available bandwidth of the network and the average transmission rate of service messages of a preset type; If so, a target video stream among at least one video stream belonging to the preset type is determined, and iterative collision elimination is performed according to the target video stream until a preset collision elimination condition is met.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program executable by the processor, and the processor implements the steps of the video stream processing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the video stream processing method according to any one of claims 1 to 7 are executed.

Images