Neural network model coding transmission method and device for monitoring video

Abstract

The embodiment of the invention discloses a neural network model coding transmission method and device for a monitoring video. A specific embodiment of the method comprises the steps of performing video coding on a shot monitoring video based on a neural network, and determining a corresponding neural network model set; performing compression coding on each neural network model to obtain a coding neural network model set; according to the coding neural network model set, carrying out video coding on the shot monitoring video to obtain a video bit stream; in response to the condition that the capacity of the coding neural network model set is smaller than or equal to a preset threshold value, adding the coding neural network model set into the video bit stream to obtain an embedded neural network video bit stream; carrying out embedded transmission on the embedded neural network video bit stream; and respectively transmitting the video bit stream and the coding neural network model set in response to the condition that the capacity of the coding neural network model set is greater than a preset threshold. According to the embodiment, network bandwidth congestion can be avoided, and waste of transmission resources is reduced.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of video coding and model compression, and more particularly to a method and device for encoding and transmitting a neural network model for monitoring video. Background Art

[0002] In video surveillance systems, encoding surveillance videos based on neural network models has become an increasingly popular technology. However, in order to effectively adapt to different application scenarios, it is often necessary to update or modify the neural network model used by the monitoring end at any time, so it is also necessary to encode and transmit the neural network model used for encoding. At present, when using a neural network model to encode surveillance videos, the commonly used methods are: using the default neural network model, that is, there is no need to transmit the neural network model; or embedding the neural network model into the video bit stream for transmission of the neural network model.

[0003] However, when the above method is used to encode surveillance videos based on a neural network model, the following technical problems often occur:

[0004] First, it is difficult to make adaptive revisions to the existing neural network model according to different application scenarios using the default neural network model, and the transmission method of embedding the neural network model into the video bit stream will cause additional bandwidth consumption, which may cause network congestion and waste of transmission resources;

[0005] Second, existing neural network model compression methods are difficult to reduce the number of model parameters while ensuring the accuracy of the compressed neural network model, and will lead to the consumption of computing resources.

[0006] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0007] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0008] Some embodiments of the present disclosure propose a neural network model encoding and transmission method and device for monitoring video to solve one or more of the technical problems mentioned in the above background technology section.

[0009] In a first aspect, some embodiments of the present disclosure provide a neural network model encoding and transmission method for surveillance videos, the method comprising: based on a neural network, performing video encoding on the captured surveillance video to obtain the encoded surveillance video and determining the corresponding neural network model set, wherein the number of neural network models in the neural network model set is the same as the number of frames of the captured surveillance video, and the neural network model includes a neural network calculation graph and a corresponding model parameter set; performing compression encoding on each neural network model in the neural network model set to obtain an encoded neural network model set; performing video encoding on the captured surveillance video according to the encoded neural network model set to obtain a video bit stream; in response to the capacity of the encoded neural network model set being less than or equal to a preset threshold, adding the encoded neural network model set to the video bit stream to obtain an embedded neural network video bit stream; performing embedded transmission on the embedded neural network video bit stream, wherein the embedded neural network transmission identifier in the embedded neural network video bit stream is 1; in response to the capacity of the encoded neural network model set being greater than a preset threshold, transmitting the video bit stream and the encoded neural network model set respectively, wherein the embedded neural network transmission identifier in the video bit stream is 0.

[0010] In a second aspect, some embodiments of the present disclosure provide a neural network model encoding and transmission device for surveillance video, the device comprising: a first video encoding unit, configured to perform video encoding on the captured surveillance video based on a neural network, to obtain the encoded surveillance video and to determine the corresponding neural network model set, wherein the number of neural network models in the above neural network model set is the same as the number of frames of the captured surveillance video, and the neural network model comprises a neural network calculation graph and a corresponding model parameter set; a compression encoding unit, configured to perform compression encoding on each neural network model in the above neural network model set, to obtain an encoded neural network model set; a second video encoding unit, configured to compress and encode the above encoded neural network model set according to the above encoded neural network model set. The surveillance video is shot and video encoding is performed to obtain a video bitstream; an adding unit is configured to, in response to the capacity of the above-mentioned encoding neural network model set being less than or equal to a preset threshold, add the above-mentioned encoding neural network model set to the above-mentioned video bitstream to obtain an embedded neural network video bitstream; an embedded transmission unit is configured to perform embedded transmission on the above-mentioned embedded neural network video bitstream, wherein the embedded neural network transmission flag in the above-mentioned embedded neural network video bitstream is 1; a separate transmission unit is configured to, in response to the capacity of the above-mentioned encoding neural network model set being greater than a preset threshold, separately transmit the above-mentioned video bitstream and the above-mentioned encoding neural network model set, wherein the embedded neural network transmission flag in the above-mentioned video bitstream is 0.

[0011] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0012] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0013] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the neural network model encoding and transmission method for monitoring video of some embodiments of the present disclosure, it is possible to avoid additional bandwidth consumption and reduce the waste of transmission resources. Specifically, the reason for the waste of transmission resources is that it is difficult to make adaptive revisions to the existing neural network model according to different application scenarios using the default neural network model, and the transmission method of embedding the neural network model into the video bit stream will cause additional bandwidth consumption, which may cause network congestion. Based on this, the neural network model encoding and transmission method for monitoring video of some embodiments of the present disclosure, first, based on the neural network, the video encoding is performed on the captured monitoring video to obtain the encoded monitoring video and determine the corresponding neural network model set. Among them, the number of neural network models in the above neural network model set is the same as the number of frames of the captured monitoring video. The neural network model includes a neural network calculation graph and a corresponding model parameter set. Through video encoding based on neural networks, the encoding problem can be transformed from the traditional local optimization relying on manual features and rules to end-to-end global optimization, which significantly improves the reconstruction quality of the encoded image. Secondly, each neural network model in the above neural network model set is compressed and encoded to obtain an encoded neural network model set. Thus, the bandwidth occupied by the neural network model during transmission can be effectively reduced. Then, according to the above-mentioned encoding neural network model set, the captured surveillance video is encoded to obtain a video bit stream. The encoded neural network model is used for video encoding and video decoding to ensure the accuracy of video encoding and decoding. Then, in response to the capacity of the above-mentioned encoding neural network model set being less than or equal to a preset threshold, the above-mentioned encoding neural network model set is added to the above-mentioned video bit stream to obtain an embedded neural network video bit stream. Therefore, the use of bandwidth and resources can be dynamically adjusted according to the size of the encoded neural network model. Next, the above-mentioned embedded neural network video bit stream is embeddedly transmitted, wherein the embedded neural network transmission identifier in the above-mentioned embedded neural network video bit stream is 1. Therefore, while avoiding bandwidth consumption, transmission delay can be reduced, thereby reducing the waste of transmission resources. Finally, in response to the capacity of the above-mentioned encoding neural network model set being greater than a preset threshold, the above-mentioned video bit stream and the above-mentioned encoding neural network model set are transmitted separately. Among them, the embedded neural network transmission identifier in the above-mentioned video bit stream is 0. When the capacity of the encoded neural network model is large, the bandwidth overload problem caused by embedded transmission can be effectively avoided by transmitting separately, reducing the waste of transmission resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0015] Figure 1 is a flowchart of some embodiments of the neural network model encoding and transmission method for monitoring video according to the present disclosure;

[0016] Figure 2 It is a schematic diagram of the structure of some embodiments of the neural network model encoding and transmission device for monitoring video according to the present disclosure;

[0017] Figure 3 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0019] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0020] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0021] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0022] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0023] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0024] Figure 1 The process 100 of some embodiments of the neural network model encoding and transmission method for monitoring video according to the present disclosure is shown. The neural network model encoding and transmission method for monitoring video includes the following steps:

[0025] Step 101, based on the neural network, the captured surveillance video is encoded to obtain the encoded surveillance video and determine the corresponding neural network model set.

[0026] In some embodiments, the execution subject of the neural network model encoding and transmission method for monitoring video can perform video encoding on the captured monitoring video based on a neural network to obtain the encoded monitoring video and determine the corresponding neural network model set. Among them, first, the captured monitoring video can be converted into video frames to obtain a video frame set. Secondly, each video frame in the above video frame set can be video encoded by a preset neural network-based video encoding algorithm to obtain an encoded monitoring video. Finally, the neural network model used in the above video encoding process can be determined as a neural network model set. Among them, the number of neural network models in the above neural network model set is the same as the number of frames of the captured monitoring video. The neural network model includes a neural network calculation graph and a corresponding model parameter set.

[0027] In practice, the image quality of surveillance videos often changes significantly due to scene changes. For example, on sunny days, there is plenty of light, but on rainy days, there may be problems such as blur, low contrast, or increased noise. This requires the use of different neural network models to process video frames in different scenes respectively to cope with video frames in different scenes. In this way, the video quality and encoding compression efficiency of video encoding can be improved.

[0028] As an example, the preset neural network-based video encoding algorithm may include but is not limited to at least one of the following: End-to-end Optimized Image Compression, Lossy Image Compression with Compressive Autoencoders, CNN-Based Inter Prediction Refinement for AVS3, etc.

[0029] Step 102, compress and encode each neural network model in the neural network model set to obtain an encoded neural network model set.

[0030] In some embodiments, the execution entity may perform compression encoding on each neural network model in the neural network model set to obtain an encoded neural network model set.

[0031] In some optional implementations of some embodiments, the execution subject compresses and encodes each neural network model in the neural network model set to obtain the encoded neural network model set, which may include the following steps:

[0032] The first step is to obtain an updated neural network identifier list, wherein the number of updated neural network identifiers in the updated neural network identifier list is the same as the number of frames of the captured surveillance video.

[0033] In practice, the above-mentioned update neural network flag can be used to determine whether the corresponding video frame should continue to use the model of the previous frame or switch to a new model for encoding. If the update neural network flag is set to 0, it means that the corresponding video frame is encoded using the neural network model used by the previous video frame. If the update neural network flag is set to 1, it means that the corresponding video frame is encoded using a neural network model different from the previous video frame.

[0034] In the second step, the first neural network model in the neural network model set is independently encoded to obtain a first encoded neural network model, and the corresponding updated neural network identifier is set to 1. The neural network model can first be compressed and converted into a data form that is easy to transmit to obtain a first encoded neural network model. The above data form may include but is not limited to at least one of the following: h5 (hierarchical data format version 5) format, ONNX (Open Neural Network Exchange) format, etc. Afterwards, the corresponding first updated neural network identifier in the above updated neural network identifier list can be set to 1.

[0035] The third step is to perform the following encoding steps on each neural network model except the first one in the above neural network model set to obtain the initial encoded neural network model set:

[0036] Sub-step 1, in response to the current neural network model being different from the previous neural network model, differentially encoding the current neural network model to obtain an initial encoded neural network model, and setting the corresponding updated neural network identifier to 1. Wherein, if the current neural network model is different from the previous neural network model, the difference between the current neural network model and the previous neural network model can be encoded to obtain the initial encoded neural network model. Thereafter, according to the sequence number of the video frame corresponding to the current neural network model, the updated neural network identifier of the corresponding sequence number can be set to 1. The above-mentioned difference portion may include different neural network layers or different model parameters between the two neural networks.

[0037] In practice, the amount of data transmitted by the model can be reduced by differentially encoding only the difference part.

[0038] Sub-step 2, in response to the current neural network model being the same as the previous neural network model, the corresponding updated neural network identifier is set to 0. If the current neural network model is the same as the previous neural network model, the updated neural network identifier of the corresponding sequence number can be set to 0 according to the sequence number of the video frame corresponding to the current neural network model.

[0039] In practice, a neural network model that is identical to a previous neural network model will not be encoded and transmitted, which can reduce the consumption of transmission resources.

[0040] The fourth step is to determine the first encoding neural network model and the initial encoding neural network model set as the encoding neural network model set.

[0041] Optionally, the execution subject independently encodes the first neural network model corresponding to the first one in the neural network model set to obtain a first encoded neural network model, including the following steps:

[0042] In the first step, the first neural network model corresponding to the above neural network model set is determined as the target neural network model.

[0043] The second step is to perform lightweight processing on the above target neural network model to obtain a lightweight neural network model.

[0044] Specifically, the execution subject performs lightweight processing on the target neural network model to obtain a lightweight neural network model, which may include the following steps:

[0045] The first sub-step is to add an intermediate fully connected layer that meets the preset conditions before the fully connected layer in the target neural network model to obtain an initial neural network model. The fully connected layer corresponds to input features and output features. The intermediate fully connected layer is used to obtain intermediate output features based on the input features. The fully connected layer is used to obtain updated output features based on the intermediate output features. The dimension of the intermediate output features is smaller than the dimension of the input features and the output features. The dimension of the updated output features is equal to the dimension of the output features.

[0046] In practice, assuming that the input and output features of the initial fully connected layer are N-dimensional and M-dimensional respectively, the corresponding fully connected layer parameter matrix is ​​N×M-dimensional. Add an intermediate fully connected layer before the above fully connected layer, the input and output features of the intermediate fully connected layer are N-dimensional and K-dimensional respectively, and the input and output features of the fully connected layer after adding are K-dimensional and M-dimensional respectively, and the corresponding fully connected layer parameter matrix is ​​N×K-dimensional and K×M-dimensional. Among them, K meets the preset conditions: K is less than N and M, and K(N+M) is less than N×M. In this way, the number of parameters of the neural network model can be reduced.

[0047] In the second sub-step, the convolutional layer in the above initial neural network model is replaced by a depth-separable convolution to obtain a lightweight neural network model.

[0048] In practice, the original convolutional layer can be replaced with a depth-wise separable convolutional layer without making additional adjustments to the input and output features. This can reduce the computational resources of the convolution operation and effectively reduce the number of parameters in the neural network model, making the neural network model more lightweight.

[0049] The third step is to perform network pruning on the above lightweight neural network model to obtain a pruned neural network model.

[0050] Specifically, the execution subject performs network pruning on the lightweight neural network model to obtain a pruned neural network model, which may include the following steps:

[0051] The first sub-step is to determine the importance scores of different channels in the above-mentioned lightweight neural network model to obtain a channel importance score set. The importance score of each channel in the above-mentioned lightweight neural network model can be determined by a preset channel importance measurement algorithm to obtain an importance score set. The channels correspond one to one to each convolution kernel in the neural network model.

[0052] In practice, the importance of each convolution kernel can be determined based on the weight of each convolution kernel in the neural network model. For those convolution kernels with lower importance, they can be deleted, and the corresponding output channels are also deleted, thus achieving the purpose of pruning.

[0053] As an example, the preset channel importance measurement algorithm may include but is not limited to at least one of the following: a filter importance evaluation algorithm based on L1 / L2 norm, an Importance Estimation for Neural Network Pruning (importance evaluation for neural network pruning) algorithm, etc.

[0054] In the second sub-step, the channel corresponding to each channel importance score that meets the preset pruning condition in the above channel importance score set is determined as the channel to be pruned, and the channel set to be pruned is obtained. Among them, the channels corresponding to the first x channel importance scores with the smallest corresponding values ​​in the above channel importance score set can be determined as the channels to be pruned, and the channel set to be pruned is obtained.

[0055] In the third sub-step, each channel to be pruned in the above-mentioned channel set to be pruned is pruned in turn to generate a pruned neural network model, and a pruned neural network model set is obtained. In which, each channel to be pruned in the above-mentioned channel set to be pruned can be deleted from the above-mentioned lightweight neural network model in turn to generate a pruned neural network model, and a pruned neural network model set is obtained. Here, the number of pruned neural network models in the above-mentioned pruned neural network model set is the same as the number of channels to be pruned in the above-mentioned channel set to be pruned.

[0056] In the fourth sub-step, the first frame of the captured surveillance video is input into each pruned neural network model in the pruned neural network model set to generate a pruned feature map, thereby obtaining a pruned feature map set.

[0057] The fifth sub-step is to input the first video frame image into the target neural network model to obtain the original feature map.

[0058] The sixth sub-step is to update the parameters of each pruned neural network model by minimizing the error between each pruned feature map in the pruned feature map set and the original feature map to generate a fine-tuned neural network model and obtain a fine-tuned neural network model set. The error can be a structural similarity index (SSIM) or a mean square error (MSE). Secondly, the error between each pruned feature map in the pruned feature map set and the original feature map can be back-propagated to update the parameters of each pruned neural network model to generate a fine-tuned neural network model and obtain a fine-tuned neural network model set.

[0059] In practice, by fine-tuning the pruned neural network model, the output of the pruned neural network model can be made as close as possible to the output of the original model, avoiding performance loss caused by pruning.

[0060] The seventh sub-step is to determine the fine-tuning neural network model that meets the preset accuracy condition in the fine-tuning neural network model set as the pre-pruned neural network model. First, the first frame of video frame image can be input into each fine-tuning neural network model in the fine-tuning neural network model set to obtain a fine-tuning feature map set. Secondly, the fine-tuning neural network model corresponding to the fine-tuning feature map with the smallest error between the fine-tuning feature map and the original feature map in the fine-tuning feature map set can be determined as the pre-pruned neural network model.

[0061] In the eighth sub-step, each model parameter in the model parameter set corresponding to the pre-pruned neural network model that satisfies the preset parameter threshold is determined as the model parameter to be pruned, and the model parameter set to be pruned is obtained. Among them, each model parameter in the model parameter set corresponding to the pre-pruned neural network model whose absolute value is less than the preset parameter threshold can be determined as the model parameter to be pruned, and the model parameter set to be pruned is obtained. The preset parameter threshold can be 0.3.

[0062] In practice, we can add L1 regularization terms to the loss function to constrain the weight parameters of the convolutional layer and batch normalization layer in the neural network model, making their parameters sparse. For neurons and connections with very small parameter values, their corresponding activation values ​​will also be close to zero, so they have little impact on the overall performance of the model. These weights close to zero can be used as objects to be pruned, thereby reducing the complexity of the model while maintaining its stable performance.

[0063] In a ninth sub-step, each parameter of the model to be trimmed in the above-mentioned parameter set of the model to be trimmed is set to 0, so as to obtain a trimmed model parameter set.

[0064] The tenth sub-step is to determine the pruned neural network model according to the pruned model parameter set, wherein the corresponding parameters in the pre-pruned neural network model can be updated according to the pruned model parameter set to obtain the pruned neural network model.

[0065] The first to tenth sub-steps of the above optional steps and their related contents are an inventive point of an embodiment of the present disclosure, which solves the above technical problem 2 "causing consumption of computing resources." The factors that lead to the above technical problem are often as follows: the existing neural network model compression method is difficult to reduce the number of model parameters while ensuring the accuracy of the compressed neural network model, and it will lead to the consumption of computing resources. If the above factors are solved, the consumption of computing resources can be avoided. In order to achieve this effect, first, by first scoring the importance of the channels, the channels with lower importance are selected as the channels to be pruned. Secondly, each pruned neural network model is fine-tuned and trained. This method avoids pruning each channel one by one before forward propagation, thereby saving computing resources and time and avoiding unnecessary computing overhead. Then, by fine-tuning the pruned neural network model, it can be ensured that the pruned network can restore the performance of the original network as much as possible. Afterwards, the fine-tuned neural network model with the highest corresponding accuracy in the fine-tuned neural network model set is determined as the pre-pruned neural network model. In this way, the accuracy loss caused by pruning can be reduced, ensuring that the accuracy will not be significantly reduced. Finally, by setting the parameter values ​​with smaller contributions to 0, the complexity of the model is further reduced, and the storage burden and computing consumption of the model are reduced. Therefore, the above overall process effectively reduces the number of model parameters while maintaining the performance of the neural network model, while optimizing the pruning efficiency, thereby reducing the consumption of computing resources.

[0066] The fourth step is to optimize the pruned neural network calculation graph corresponding to the pruned neural network model to obtain the optimized neural network model.

[0067] Specifically, the execution subject optimizes the calculation graph of the pruned neural network corresponding to the pruned neural network model to obtain the optimized neural network model, which may include the following steps:

[0068] In the first sub-step, the network layers that meet the preset conditions in the pruned neural network calculation graph corresponding to the pruned neural network model are eliminated to obtain the integrated neural network calculation graph.

[0069] In practice, if the output of a network layer is no longer passed to subsequent layers, the layer can be regarded as a redundant layer or an invalid layer and can be removed from the computational graph. As a result, the computational graph of the neural network will become more concise, thereby improving the efficiency of model reasoning and reducing computing resource consumption.

[0070] The second sub-step is to connect the convolutional layer, batch normalization layer and activation function in the above-mentioned integrated neural network calculation graph in series to obtain the merged neural network calculation graph, and determine the corresponding optimized neural network model according to the above-mentioned merged neural network calculation graph. Among them, the convolutional layer, batch normalization layer and activation function in the above-mentioned integrated neural network calculation graph can be connected in series through a preset model inference acceleration algorithm (TensorRT) to obtain the merged neural network calculation graph, and determine the corresponding optimized neural network model according to the above-mentioned merged neural network calculation graph.

[0071] The fifth step is to quantize the optimized model parameter set corresponding to the above-mentioned optimized neural network model to obtain the first encoded neural network model. Parameter quantization refers to mapping floating point numbers (such as FP32) to low-bit numbers (such as INT8) to reduce the amount of calculation and memory usage while maintaining model performance as much as possible. After quantization, the parameters of the neural network model will be compressed into a smaller numerical range, thereby accelerating the reasoning process and reducing hardware requirements.

[0072] Optionally, the execution subject quantizes the optimized model parameter set corresponding to the optimized neural network model to obtain the first encoding neural network model, which may include the following steps:

[0073] In the first sub-step, the difference between the optimized model parameter with the largest corresponding value and the optimized model parameter with the smallest corresponding value in the optimized model parameter set corresponding to the optimized neural network model is determined as the dynamic range value.

[0074] In the second sub-step, the ratio of the dynamic range value to the preset maximum representation value is determined as a scaling factor, wherein the maximum representation value may be 255, which is the maximum representation value of the INT8 data type.

[0075] In the third sub-step, the ratio of the optimized model parameter with the smallest corresponding value to the scaling factor is determined as the offset.

[0076] The fourth sub-step is to perform scaling mapping on each optimized model parameter in the optimized model parameter set according to the scaling factor and the offset to generate a mapped model parameter, thereby obtaining a mapped model parameter set. First, the ratio between the optimized model parameter and the scaling factor can be determined. Then, the sum of the ratio and the offset is determined as the mapped model parameter.

[0077] The fifth sub-step is to determine the first encoding neural network model corresponding to the above-mentioned optimized neural network model according to the above-mentioned mapped model parameter set. The above-mentioned mapped model parameter set can be determined as the parameters corresponding to the above-mentioned optimized neural network model to obtain the first encoding neural network model.

[0078] Optionally, the execution subject differentially encodes the current neural network model to obtain an initial encoded neural network model, which may include the following steps:

[0079] The first step is to determine the difference neural network calculation graph between the current neural network model and the corresponding previous neural network model. Here, the difference neural network calculation graph is used to record the difference between the current neural network model and the previous neural network model.

[0080] For example, the neurons included in the current neural network model but not included in the previous neural network model can be used as part of the difference neural network calculation graph. The neurons or network layers included in the previous neural network model but not included in the current neural network model can also be used as part of the difference neural network calculation graph, and a deletion flag can be added. The above deletion flag is used to indicate the neurons or network layers included in the previous neural network model but not included in the current neural network model. The above deletion flag can be -1.

[0081] In the second step, the model parameter difference between the current neural network model and the corresponding previous neural network model is determined as the difference model parameter value to obtain the difference model parameter value set. Among them, the difference between each model parameter of the current neural network model and the model parameter of the corresponding previous neural network model can be determined as the difference model parameter value to obtain the difference model parameter value set. Here, the above-mentioned difference model parameter value set includes the difference between the different parameter values ​​of the same neuron in the current neural network model and the corresponding previous neural network model, and also includes the model parameter values ​​corresponding to different neurons. For example, if a neuron exists in the current neural network model but does not exist in the previous neural network model, its parameter value in the current neural network model can be determined as the difference model parameter value, and an addition mark can be added to the difference model parameter value. The above-mentioned addition mark is used to indicate neurons or network layers that do not exist in the previous neural network model. The above-mentioned addition mark can be +1.

[0082] The third step is to determine the above difference neural network calculation graph and the above difference model parameter value set as the initial encoding neural network model corresponding to the current neural network model.

[0083] Step 103: perform video encoding on the captured surveillance video according to the encoding neural network model set to obtain a video bit stream.

[0084] In some embodiments, the execution subject may perform video encoding on the captured surveillance video according to the encoding neural network model set to obtain a video bit stream. Among them, the captured surveillance video may be encoded according to the encoding neural network model set by the same neural network-based video encoding algorithm as step 101 to obtain a video bit stream. Details are not given here. Here, the video bit stream is data in a binary data stream format generated after video encoding, which contains video encoding information and related control information, and is used for transmission and decoding during video transmission or playback.

[0085] Step 104, in response to the capacity of the encoded neural network model set being less than or equal to a preset threshold, the encoded neural network model set is added to the video bitstream to obtain an embedded neural network video bitstream.

[0086] In some embodiments, the execution subject may add the encoded neural network model set to the video bitstream in response to the capacity of the encoded neural network model set being less than or equal to a preset threshold value to obtain an embedded neural network video bitstream. The preset threshold value may be 25% of the video transmission bitrate. The video transmission bitrate refers to the number of bits of video data transmitted per second, usually expressed in bits per second (bps) or kilobits per second (kbps).

[0087] Step 105, embedding the embedded neural network video bit stream.

[0088] In some embodiments, the execution subject may perform embedded transmission of the embedded neural network video bitstream, wherein the embedded neural network transmission identifier in the embedded neural network video bitstream is 1. Here, the embedded neural network transmission identifier is used to indicate whether the transmitted video bitstream contains a coded neural network model.

[0089] In practice, if the size of the encoded neural network model set is limited and the impact on the transmission bandwidth is not significant, the encoded neural network model set is transmitted along with the video bit stream in an embedded manner.

[0090] Step 106, in response to the capacity of the encoding neural network model set being greater than a preset threshold, the video bit stream and the encoding neural network model set are transmitted separately.

[0091] In some embodiments, the execution subject may transmit the video bitstream and the encoded neural network model set respectively in response to the capacity of the encoded neural network model set being greater than a preset threshold, wherein the embedded neural network transmission flag in the video bitstream is 0.

[0092] In practice, when the capacity of the encoded neural network model set is large, the bandwidth overload problem caused by embedded transmission can be effectively avoided by transmitting the video bit stream and the neural network model set separately. This separate transmission method ensures the reasonable allocation of network bandwidth and avoids affecting the real-time playback of the video or causing network congestion.

[0093] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the neural network model encoding and transmission method for monitoring video of some embodiments of the present disclosure, it is possible to avoid additional bandwidth consumption and reduce the waste of transmission resources. Specifically, the reason for the waste of transmission resources is that it is difficult to make adaptive revisions to the existing neural network model according to different application scenarios using the default neural network model, and the transmission method of embedding the neural network model into the video bit stream will cause additional bandwidth consumption, which may cause network congestion. Based on this, the neural network model encoding and transmission method for monitoring video of some embodiments of the present disclosure, first, based on the neural network, the video encoding is performed on the captured monitoring video to obtain the encoded monitoring video and determine the corresponding neural network model set. Among them, the number of neural network models in the above neural network model set is the same as the number of frames of the captured monitoring video. The neural network model includes a neural network calculation graph and a corresponding model parameter set. Through video encoding based on neural networks, the encoding problem can be transformed from the traditional local optimization relying on manual features and rules to end-to-end global optimization, which significantly improves the reconstruction quality of the encoded image. Secondly, each neural network model in the above neural network model set is compressed and encoded to obtain an encoded neural network model set. Thus, the bandwidth occupied by the neural network model during transmission can be effectively reduced. Then, according to the above-mentioned encoding neural network model set, the captured surveillance video is encoded to obtain a video bit stream. The encoded neural network model is used for video encoding and video decoding to ensure the accuracy of video encoding and decoding. Then, in response to the capacity of the above-mentioned encoding neural network model set being less than or equal to a preset threshold, the above-mentioned encoding neural network model set is added to the above-mentioned video bit stream to obtain an embedded neural network video bit stream. Therefore, the use of bandwidth and resources can be dynamically adjusted according to the size of the encoded neural network model. Next, the above-mentioned embedded neural network video bit stream is embeddedly transmitted, wherein the embedded neural network transmission identifier in the above-mentioned embedded neural network video bit stream is 1. Therefore, while avoiding bandwidth consumption, transmission delay can be reduced, thereby reducing the waste of transmission resources. Finally, in response to the capacity of the above-mentioned encoding neural network model set being greater than a preset threshold, the above-mentioned video bit stream and the above-mentioned encoding neural network model set are transmitted separately. Among them, the embedded neural network transmission identifier in the above-mentioned video bit stream is 0. When the capacity of the encoded neural network model is large, the bandwidth overload problem caused by embedded transmission can be effectively avoided by transmitting separately, reducing the waste of transmission resources.

[0094] Further references Figure 2 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a neural network model encoding and transmission device for monitoring video. These device embodiments are similar to Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0095] like Figure 2 As shown, some embodiments of the neural network model encoding and transmission device 200 for surveillance video include: a first video encoding unit 201, a compression encoding unit 202, a second video encoding unit 203, an adding unit 204, an embedded transmission unit 205, and a separate transmission unit 206. The first video encoding unit 201 is configured to perform video encoding on the captured surveillance video based on a neural network to obtain the encoded surveillance video and determine the corresponding neural network model set, wherein the number of neural network models in the above neural network model set is the same as the number of frames of the captured surveillance video, and the neural network model includes a neural network calculation graph and a corresponding model parameter set; the compression encoding unit 202 is configured to perform compression encoding on each neural network model in the above neural network model set to obtain an encoded neural network model set; the second video encoding unit 203 is configured to perform video encoding on the captured surveillance video according to the above encoded neural network model set to obtain a video bit stream; The adding unit 204 is configured to add the above-mentioned encoded neural network model set to the above-mentioned video bit stream in response to the capacity of the above-mentioned encoded neural network model set being less than or equal to a preset threshold value, so as to obtain an embedded neural network video bit stream; the embedded transmission unit 205 is configured to perform embedded transmission on the above-mentioned embedded neural network video bit stream, wherein the embedded neural network transmission identifier in the above-mentioned embedded neural network video bit stream is 1; the separate transmission unit 206 is configured to transmit the above-mentioned video bit stream and the above-mentioned encoded neural network model set separately in response to the capacity of the above-mentioned encoded neural network model set being greater than a preset threshold value, wherein the embedded neural network transmission identifier in the above-mentioned video bit stream is 0.

[0096] It is understood that the units described in the device 200 are similar to those described in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 200 and the units included therein, and will not be described in detail here.

[0097] Reference below Figure 3 , which shows a structural schematic diagram of an electronic device (eg, a computing device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0098] like Figure 3 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory 302 or a program loaded from a storage device 308 into a random access memory 303. Various programs and data required for the operation of the electronic device 300 are also stored in the random access memory 303. The processing device 301, the read-only memory 302, and the random access memory 303 are connected to each other via a bus 304. An input / output interface 305 is also connected to the bus 304.

[0099] Typically, the following devices may be connected to the I / O interface 305: input devices 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 309. The communication devices 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0100] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the read-only memory 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0101] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0102] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0103] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: based on a neural network, encodes the captured surveillance video to obtain the encoded surveillance video and determines the corresponding neural network model set, wherein the number of neural network models in the neural network model set is the same as the number of frames of the captured surveillance video, and the neural network model includes a neural network calculation graph and a corresponding model parameter set; compresses and encodes each neural network model in the neural network model set to obtain an encoded neural network model set; according to the encoded neural network model set, encodes the captured surveillance video to obtain a video bit stream; in response to the capacity of the encoded neural network model set being less than or equal to a preset threshold, adds the encoded neural network model set to the video bit stream to obtain an embedded neural network video bit stream; embeds the embedded neural network video bit stream, wherein the embedded neural network transmission identifier in the embedded neural network video bit stream is 1; in response to the capacity of the encoded neural network model set being greater than a preset threshold, transmits the video bit stream and the encoded neural network model set respectively, wherein the embedded neural network transmission identifier in the video bit stream is 0.

[0104] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0105] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0106] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor. For example, they may be described as: a processor comprising a first video encoding unit, a compression encoding unit, a second video encoding unit, an adding unit, an embedded transmission unit, and a separate transmission unit. Among them, the names of these units do not constitute a limitation on the units themselves in certain cases. For example, the compression encoding unit may also be described as "a unit for performing compression encoding on each neural network model in the neural network model set."

[0107] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0108] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A neural network model encoding and transmission method for monitoring video, comprising: Based on the neural network, the video encoding is performed on the surveillance video taken, the encoded surveillance video is obtained, and the corresponding neural network model set is determined, wherein the number of neural network models in the neural network model set is the same as the number of frames of the surveillance video taken, and the neural network model includes a neural network calculation graph and a corresponding model parameter set; Performing compression encoding on each neural network model in the neural network model set to obtain an encoded neural network model set; According to the encoding neural network model set, the captured surveillance video is encoded to obtain a video bit stream; In response to the capacity of the encoding neural network model set being less than or equal to a preset threshold, adding the encoding neural network model set to the video bitstream to obtain an embedded neural network video bitstream; Performing embedded transmission on the embedded neural network video bit stream, wherein the embedded neural network transmission identifier in the embedded neural network video bit stream is 1; In response to the capacity of the encoded neural network model set being greater than a preset threshold, the video bit stream and the encoded neural network model set are transmitted separately, wherein the embedded neural network transmission identifier in the video bit stream is 0.

2. The method according to claim 1, wherein: The step of compressing and encoding each neural network model in the neural network model set to obtain an encoded neural network model set includes: Obtaining an updated neural network identifier list, wherein the number of updated neural network identifiers in the updated neural network identifier list is the same as the number of frames of the captured surveillance video; Independently encoding the first neural network model in the neural network model set to obtain a first encoded neural network model, and setting the corresponding updated neural network flag to 1; The following encoding steps are performed in sequence on each neural network model except the first one in the neural network model set to obtain an initial encoded neural network model set: In response to the current neural network model being different from the previous neural network model, differentially encoding the current neural network model to obtain an initial encoded neural network model, and setting a corresponding updated neural network flag to 1; In response to the current neural network model being the same as the previous neural network model, setting the corresponding updated neural network flag to 0; The first encoding neural network model and the initial encoding neural network model set are determined as an encoding neural network model set.

3. The method according to claim 2, wherein: The step of independently encoding the first neural network model corresponding to the first one in the neural network model set to obtain a first encoded neural network model comprises: Determine the first neural network model in the neural network model set as the target neural network model; Performing lightweight processing on the target neural network model to obtain a lightweight neural network model; Performing network pruning on the lightweight neural network model to obtain a pruned neural network model; Performing calculation graph optimization on the pruned neural network calculation graph corresponding to the pruned neural network model to obtain an optimized neural network model; Parameter quantization is performed on the optimized model parameter set corresponding to the optimized neural network model to obtain a first encoded neural network model.

4. The method according to claim 3, wherein: The step of performing lightweight processing on the target neural network model to obtain a lightweight neural network model includes: Adding an intermediate fully connected layer that meets preset conditions before the fully connected layer in the target neural network model to obtain an initial neural network model, wherein the fully connected layer corresponds to an input feature and an output feature, the intermediate fully connected layer is used to obtain an intermediate output feature according to the input feature, the fully connected layer is used to obtain an updated output feature according to the intermediate output feature, the dimension of the intermediate output feature is smaller than the dimension of the input feature and the output feature, and the dimension of the updated output feature is equal to the dimension of the output feature; The convolutional layers in the initial neural network model are replaced with depthwise separable convolutions to obtain a lightweight neural network model.

5. The method according to claim 3, wherein: The step of optimizing the pruned neural network calculation graph corresponding to the pruned neural network model to obtain the optimized neural network model includes: Eliminate the network layers that meet the preset conditions in the pruned neural network calculation graph corresponding to the pruned neural network model to obtain an integrated neural network calculation graph; The convolution layer, batch normalization layer and activation function in the integrated neural network calculation graph are connected in series to obtain a merged neural network calculation graph, and the corresponding optimized neural network model is determined according to the merged neural network calculation graph.

6. The method according to claim 3, wherein: The step of performing parameter quantization on the optimized model parameter set corresponding to the optimized neural network model to obtain a first encoding neural network model includes: Determine the difference between the optimized model parameter with the largest corresponding value and the optimized model parameter with the smallest corresponding value in the optimized model parameter set corresponding to the optimized neural network model as the dynamic range value; Determine a ratio of the dynamic range value to a preset maximum representation value as a scaling factor; Determine the ratio of the optimized model parameter with the smallest corresponding value to the scaling factor as an offset; According to the scaling factor and the offset, scaling and mapping are performed on each optimized model parameter in the optimized model parameter set to generate a mapped model parameter, thereby obtaining a mapped model parameter set; According to the mapped model parameter set, a first encoding neural network model corresponding to the optimized neural network model is determined.

7. The method according to claim 2, wherein: The differential encoding of the current neural network model to obtain an initial encoding neural network model includes: determining a difference neural network computation graph between a current neural network model and a corresponding previous neural network model; Determine the model parameter difference between the current neural network model and the corresponding previous neural network model as the difference model parameter value, and obtain the difference model parameter value set; The difference neural network calculation graph and the difference model parameter value set are determined as the initial encoding neural network model corresponding to the current neural network model.

8. A neural network model encoding and transmission device for monitoring video, comprising: A first video encoding unit is configured to perform video encoding on the captured surveillance video based on a neural network to obtain an encoded surveillance video and determine a corresponding neural network model set, wherein the number of neural network models in the neural network model set is the same as the number of frames of the captured surveillance video, and the neural network model includes a neural network calculation graph and a corresponding model parameter set; A compression encoding unit, configured to perform compression encoding on each neural network model in the neural network model set to obtain an encoded neural network model set; A second video encoding unit is configured to perform video encoding on the captured surveillance video according to the encoding neural network model set to obtain a video bit stream; An adding unit is configured to add the encoded neural network model set to the video bitstream in response to the capacity of the encoded neural network model set being less than or equal to a preset threshold value to obtain an embedded neural network video bitstream; An embedded transmission unit is configured to perform embedded transmission on the embedded neural network video bit stream, wherein the embedded neural network transmission identifier in the embedded neural network video bit stream is 1; A separate transmission unit is configured to transmit the video bit stream and the encoded neural network model set separately in response to the capacity of the encoded neural network model set being greater than a preset threshold, wherein the embedded neural network transmission identifier in the video bit stream is 0.

9. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images