An indoor monitoring video coding method based on human-computer collaborative perception

Abstract

The application belongs to the technical field of video coding, and discloses an indoor monitoring video coding method based on human-computer collaborative perception, which comprises the following steps: 1, building a human-computer collaborative video coding framework; 2, compact feature representation of specific semantic objects; 3, adaptive background updating; 4, residual coding resource allocation.The application proposes an adaptive background updating framework to dynamically update the background according to the characteristics of indoor monitoring video, considers that specific foreground semantic objects in the indoor monitoring video can be represented by compact semantic features, extracts compact semantic feature representation for specific foreground semantic objects through a feature extraction network, formulates a residual coding resource allocation strategy for a traditional encoder, avoids useless coding resource allocation, improves the human eye visual quality of the decoded video, greatly improves the compression performance of the indoor monitoring video, and realizes efficient specific semantic information retrieval and analysis.

Description

Technical Field

[0001] This invention belongs to the field of video coding technology, and in particular relates to an indoor surveillance video coding method based on human-machine collaborative perception. Background Technology

[0002] Most indoor cameras are designed for fixed scenes, and the video signals have their own characteristics: (1) The background does not change much or changes little over a period of time. Compared with applications such as broadcast television and video websites, reusing the background provides the possibility for more efficient video encoding; (2) A large number of surveillance cameras are deployed in indoor office buildings, residential buildings, etc. Most of the surveillance data is not used for manual viewing, and the information contained in the video data is mainly used for machine analysis; (3) Indoor surveillance often focuses on specific semantic information, such as facial information, human body movements and human body positions. These specific semantic information are the focus of security applications, target retrieval and other application fields.

[0003] Currently, researchers have made preliminary explorations into the characteristics of surveillance videos. Most scholars, when encoding and compressing surveillance videos, first separate the video into foreground and background, then use traditional encoding / decoding methods or deep learning networks to encode the foreground and background separately. Some scholars still use traditional encoding / decoding methods for compression: for example, Lu et al. proposed a compression method for helipad surveillance videos based on object detection. In their work, the surveillance video is separated into background images and object images (moving and stationary), stored separately on disk. However, this method's compression processing of foreground objects is relatively crude, resulting in limited encoding compression effects and inaccurate decoding of video content. For example, Ghafari et al. used a Gaussian mixture model to separate the video into background images and moving objects. The foreground video is then compressed using a traditional video encoder, while the background image is cached on the client side. However, this approach does not effectively update the background, and directly combining the background with the foreground leads to edge inconsistencies. Some scholars have used deep learning networks for encoding compression: for example, Wu et al. proposed a foreground-background parallel compression scheme, in which background information is shared between adjacent frames, and optical flow is used for motion estimation of the foreground image. However, although this method is very versatile, the representation of the foreground is not compact enough, and the encoding compression effect is limited. For example, Xia et al. proposed an end-to-end surveillance video compression scheme based on optical flow prediction network and motion compensation network. However, this method cannot handle video compression tasks with complex indoor backgrounds, and because it relies entirely on model algorithms for generation, the decoded video frames will lack a lot of details that should be in the original video.

[0004] In summary, although researchers have proposed a series of encoding and compression methods for the characteristics of surveillance videos, none of these methods have fully considered the characteristics of indoor surveillance videos. Therefore, it is necessary to study a method that can effectively compress indoor surveillance videos based on their specific characteristics. Summary of the Invention

[0005] The purpose of this invention is to provide an indoor surveillance video encoding method based on human-machine collaborative perception to solve the above-mentioned technical problems.

[0006] To address the aforementioned technical problems, the specific technical solution of the indoor surveillance video encoding method based on human-machine collaborative perception of the present invention is as follows:

[0007] An indoor surveillance video encoding method based on human-machine collaborative perception includes the following steps:

[0008] Step 1: Build a human-machine collaborative video coding framework;

[0009] Step 2: Compact feature representation of specific semantic objects;

[0010] Step 3: Adaptive background update;

[0011] Step 4: Residual coding resource allocation.

[0012] Furthermore, step 1 includes the following specific steps:

[0013] For the acquired continuous surveillance video frames (x1, x2, ...), this invention performs specific semantic target extraction and surveillance background image extraction to obtain semantic target sequences and background image sequences. The semantic target sequence is transformed into a compact feature representation via a feature extraction network, and then encoded and compressed before being transmitted to the decoding end for different machine vision analysis tasks. The machine vision task analysis network parses the compact feature representation, and the semantic analysis features obtained from the analysis, together with the semantic reference object obtained from the decoding, are used to reconstruct the semantic target sequence through a conditional generation network. The background image sequence is dynamically updated at the decoding end by encoding and transmitting a background template and necessary residuals according to an adaptive background update strategy. The dynamically updated background and the reconstructed semantic target sequence are synthesized together to form a reconstructed video frame, which is then added to the prediction residuals before being used for human viewing.

[0014] Furthermore, step 2 includes the following specific steps:

[0015] First, a semantic object detection network and a multi-object tracking network are used to continuously detect and track specific semantic objects (d0, d1, ...). While selecting and encoding semantic reference objects m, a feature extraction network E(·) is used to convert the sequence of specific semantic objects into compact feature representations (f0, f1, ...). The compact feature representations contain compact and complete semantic information. Then, the compact features (f0, f1, ...) are encoded and transmitted, and the decoder decodes them to obtain the compact feature sequence. Compact feature representation sequence Different deep neural networks A(·) are fed into the network for multi-task analysis (task0, task1, ...) to obtain the analysis results and semantic features of specific semantic objects. Simultaneously, decoding yields semantic reference objects. Semantic reference object It contains complete appearance information of a specific semantic target object, and a compact sequence of feature representations. Together, we reconstruct a specific semantic target object using a conditional generative network G(·). The formula is expressed as follows:

[0016] f0, f1,…=E(d0,d1,…) (1)

[0017]

[0018]

[0019] In the formula θ n This represents the parameters of the deep neural network used for different analysis tasks.

[0020] Furthermore, step 3 includes the following specific steps:

[0021] An adaptive weighted background update algorithm is designed to dynamically update the background. First, a kernel density clustering algorithm is used to extract background frames. Background is extracted from segments of the surveillance video to obtain a background frame sequence (b0, b1, ...). The time interval for extracting background frames is set to T. The kernel density clustering-based background extraction algorithm first extracts continuous surveillance video frames of time interval T and establishes a pixel value set for each pixel position. Pixel values ​​of frame regions containing specific semantic target objects detected by a semantic object detection network are not added to the pixel value set for calculation. Each pixel value set is calculated independently to obtain the background pixel values. Finally, the background pixel values ​​form the complete background. Taking any pixel value set as an example, some samples are randomly selected from the corresponding pixel positions in the continuous video frames. The average of the samples is taken as the classic point x of the pixel value set. Then, drifting begins from the classic point x. The update formula for the classic point is...

[0022] x = x + M h (4)

[0023] M h Let be the drift vector of the current classic point position, and its basic form is:

[0024]

[0025] S h (x)={y:(yx i ) T (yx i )<h 2} (6)

[0026] The classic point x is drifted by iteratively using the above formula, so that the position of the classic point x moves continuously towards the position with the highest distribution density in the pixel value point set until it reaches the convergence point. After obtaining a certain number of convergence points by randomly selecting different samples from the point set multiple times, these convergence points are grouped into a set and clustered again. The final convergence point is extracted as the background pixel value of the pixel corresponding to the pixel value point set.

[0027] Furthermore, after obtaining the background frame sequence (b0, b1, ...) in step 3, an adaptive background update process is performed.

[0028] First, decode the background with a frame order of nm. Set as background template, and extract background b in the order of n. n In comparison, if the difference between the two is greater than the update threshold γ, the two are subtracted to obtain the residual, which is then transmitted via encoding. Adding it to the residual yields The formula is expressed as follows:

[0029]

[0030]

[0031] In the formula, Conv2d(·) represents a two-dimensional convolution operation, and δ represents the difference threshold. This represents the total size of the image after the convolution operation.

[0032] Furthermore, step 4 includes the following steps:

[0033] Reconstructing the video The residual *r* between the original video *x* and the original video *x* is encoded. The residual is then compressed using a traditional codec. A residual coding resource allocation strategy is designed to improve the coding performance of the framework. First, a foreground mask *M* is obtained from the semantic segmentation results of a specific semantic target obtained by a semantic segmentation network. The specific semantic target region is then enlarged through a dilation operation. The enlarged foreground mask *M* guides the bit allocation at the CTU level in the traditional coding process, allocating more bits to CTUs with a higher overall foreground proportion. After bitrate allocation using the residual coding resource allocation strategy, the decoder receives the residual *r* and integrates it with the reconstructed video. The summation yields the final decoded video for human vision. The formulas for calculating the proportion of internal foreground and the proportion of overall foreground are as follows:

[0034]

[0035]

[0036] Where M x,y This represents the mask value at position (x, y). Let Pr(i) represent the total pixel value of the CTU, C represent the total number of CTUs, Pr(i) represent the foreground percentage of the i-th CTU, and T(i) represent the overall foreground percentage.

[0037] The present invention provides an indoor surveillance video coding method based on human-machine collaborative perception, which has the following advantages: The invention proposes an adaptive background update framework to dynamically update the background, taking into account the characteristics of indoor surveillance videos. It considers that specific foreground semantic objects in indoor surveillance videos can be represented by compact semantic features. A feature extraction network extracts compact semantic feature representations for these foreground semantic objects, which are used for efficient encoding compression and machine vision task analysis, and can reconstruct the specific foreground semantic objects. Furthermore, the invention develops a residual coding resource allocation strategy for traditional encoders, avoiding the allocation of useless coding resources while improving the human visual quality of the decoded video, greatly improving the compression performance of indoor surveillance videos; and achieving efficient retrieval and analysis of specific semantic information. Attached Figure Description

[0038] Figure 1 This is a flowchart of the indoor surveillance video encoding process based on human-machine collaborative perception according to the present invention. Detailed Implementation

[0039] To better understand the purpose, structure, and function of this invention, the following detailed description of an indoor surveillance video encoding method based on human-machine collaborative perception, in conjunction with the accompanying drawings, is provided.

[0040] like Figure 1As shown, the present invention provides an indoor surveillance video encoding method based on human-machine collaborative perception, comprising the following steps:

[0041] Step 1: Build a human-machine collaborative video coding framework

[0042] This invention focuses on the characteristics of indoor surveillance videos and designs a human-machine collaborative video coding framework. For acquired continuous surveillance video frames (x1, x2, ...), this invention performs specific semantic target extraction and surveillance background image extraction to obtain semantic target sequences and background image sequences. The semantic target sequence is transformed into a compact feature representation via a feature extraction network, and then transmitted to the decoding end for different machine vision analysis tasks through encoding and compression. The machine vision task analysis network parses the compact feature representation, and the semantic analysis features obtained from the analysis, together with the semantic reference objects obtained from decoding, are used to reconstruct the semantic target sequence through a conditional generation network. The background image sequence, according to an adaptive background update strategy, encodes and transmits a background template and necessary residuals to dynamically update the background image at the decoding end. The dynamically updated background and the reconstructed semantic target sequence are synthesized together to form a reconstructed video frame, which is then added to the prediction residual for human viewing.

[0043] Step 2: Compact Feature Representation of Specific Semantic Objects

[0044] Considering the temporal correlation of video signals, this invention first uses a semantic object detection network (such as YOLO, RCNN) and a multi-object tracking network (MOT) to continuously detect and track specific semantic objects (d0, d1, ...). While selecting and encoding a semantic reference object m, a feature extraction network E(·) is used to convert the sequence of specific semantic objects into a compact feature representation (f0, f1, ...). This compact feature representation contains compact and complete semantic information. Next, this invention encodes and transmits the compact features (f0, f1, ...), and the decoder decodes to obtain the compact feature sequence. Compact feature representation sequence Different deep neural networks A(·) are fed into the network for multi-task analysis (task0, task1, ...) to obtain the analysis results of specific semantic objects (such as identity, behavior) and semantic analysis features. (Including human skeleton, limb masks, etc.). Simultaneously, this invention decodes and obtains semantic reference objects. Semantic reference object It contains complete appearance information of a specific semantic target object, and a compact sequence of feature representations. Together, we reconstruct a specific semantic target object using a conditional generative network G(·). The formula is expressed as follows:

[0045] f0, f1,…=E(d0,d1,…) (1)

[0046]

[0047]

[0048] In the formula θ n This represents the parameters of the deep neural network used for different analysis tasks.

[0049] Step 3: Adaptive Background Update

[0050] In security applications, the background of cameras is mostly fixed, but changes in brightness / color and other environmental factors can cause fluctuations in the background. Therefore, this invention designs an adaptive weighted background update algorithm to dynamically update the background. This invention first uses a kernel density clustering algorithm to extract background frames, extracting background frames from segments of the surveillance video to obtain a background frame sequence (b0, b1, ...), with the interval between frame extractions set to T. The kernel density clustering-based background extraction algorithm first extracts continuous surveillance video frames of time interval T and establishes a pixel value set for each pixel location. Pixel values ​​of frame regions containing specific semantic target objects detected by a semantic object detection network are not added to the pixel value set for calculation. Each pixel value set is calculated independently to obtain the background pixel values, and the final background pixel values ​​form the complete background. Taking any pixel value set as an example, some samples are randomly selected from the corresponding pixel locations in the continuous video frames. The average of these samples is taken as the classic point x of the pixel value set, and then the drift starts from the classic point x. The update formula for the classic point is...

[0051] x = x + M h (4)

[0052] M h Let be the drift vector of the current classic point position, and its basic form is:

[0053]

[0054] S h (x)={y:(yx i ) T (yx i )<h 2} (6)

[0055] The classic point x is iteratively shifted using the above formula, causing its position to continuously move towards the location with the highest distribution density in the pixel value set, until it reaches the convergence point. After obtaining a certain number of convergence points by randomly selecting different samples from the point set multiple times, these convergence points are grouped into a set and clustered again. The final convergence point is then extracted as the background pixel value corresponding to the pixel in the pixel value set.

[0056] After obtaining the background frame sequence (b0, b1, ...), an adaptive background update process is performed. First, the decoded background with frame order nm is updated. Set as background template, and extract background b in the order of n. n In comparison, if the difference between the two is greater than the update threshold γ, the two are subtracted to obtain the residual, which is then transmitted via encoding. Adding it to the residual yields The formula is expressed as follows:

[0057]

[0058]

[0059] In the formula, Conv2d(·) represents a two-dimensional convolution operation, and δ represents the difference threshold. This represents the total size of the image after the convolution operation.

[0060] Step 4: Residual Coding Resource Allocation

[0061] From semantic target object Videos reconstructed by combining background b often exhibit inconsistent transition boundaries, therefore further processing of the reconstructed video is required. The residual r between the original video x and the original video x is encoded to improve the reconstructed video. To improve visual quality, this invention uses traditional codecs to encode and compress residuals. However, traditional encoder block coding strategies often distribute coding resources evenly, resulting in resources being consumed on background fluctuations and other information insensitive to the human eye. Therefore, this invention designs a residual coding resource allocation strategy to improve the coding performance of the framework. First, a foreground mask M is obtained from the semantic segmentation results of a specific semantic target obtained by a semantic segmentation network. The specific semantic target region is then enlarged through a dilation operation. The enlarged foreground mask M guides the bit allocation at the CTU level in the traditional coding process, allocating more bits to CTUs with higher overall foreground proportions. After bitrate allocation using the residual coding resource allocation strategy, the decoder receives the residual r and combines it with the reconstructed video. The summation yields the final decoded video for human vision. The formulas for calculating the proportion of internal foreground and the proportion of overall foreground are as follows:

[0062]

[0063]

[0064] Where M x,y This represents the mask value at position (x, y). Let Pr(i) represent the total pixel value of the CTU, and C represent the total number of CTUs. Let Pr(i) represent the foreground percentage of the i-th CTU, and T(i) represent the overall foreground percentage.

[0065] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A method for encoding indoor surveillance video based on human-machine collaborative perception, characterized in that, Includes the following steps: For the acquired continuous monitoring video frames The system extracts specific semantic targets and monitors background images separately, obtaining semantic target sequences and background image sequences. The semantic target sequence is transformed into a compact feature representation by a feature extraction network, and then transmitted to the decoder for different machine vision analysis tasks. The machine vision task analysis network parses the compact feature representation, and the semantic analysis features obtained from the analysis, together with the semantic reference objects obtained from the decoding, are used to reconstruct the semantic target sequence through a conditional generation network. The background image sequence is dynamically updated on the background image at the decoder by encoding and transmitting a background template and necessary residuals according to an adaptive background update strategy. The dynamically updated background and the reconstructed semantic target sequence are synthesized together to form a reconstructed video frame, which is then added to the prediction residual and finally used for human viewing. The formula for compact feature representation is expressed as follows: (1) The formula for the visual analysis task is expressed as follows: (2) The formula for reconstructing the semantic target sequence is as follows: (3) In the formula, ( E( represents the specific semantic target extracted; E( ) represents a feature extraction network; Represents compact features; This indicates that the decoder obtains a compact feature sequence; A( ) represents a deep neural network; This refers to a visual analysis task used to obtain analysis results and semantic analysis features of a specific semantic object, where the analysis results of the specific semantic object include identity and behavior. This represents semantic analysis features, including the human skeleton and limb masks; Represents a semantic reference object, containing complete appearance information of a specific semantic target object. This represents the parameters of the deep neural network used for different analysis tasks; This represents a conditional generation network. Represents the semantic target sequence for reconstruction; Get the background frame sequence ( After that, the background adaptive update process is carried out; First, set the frame order as follows: Decoding background Set as background template, extracting backgrounds in order n. In comparison, if the difference between the two is greater than the update threshold γ, the two are subtracted to obtain the residual, which is then transmitted via encoding. Adding it to the residual yields The formula is expressed as follows: (7) (8) In the formula Conv2d( () represents a two-dimensional convolution operation. Indicates the difference threshold. This represents the total size of the image after the convolution operation; The process of dynamically updating the background and reconstructing the semantic target sequence together to synthesize the reconstructed video frame, and adding it to the prediction residual, includes the following steps: Reconstructing the video The residual *r* between the original video I and the original video I is encoded. The residual is then compressed using a traditional codec, and a residual coding resource allocation strategy is designed. First, a foreground mask *M* is obtained from the semantic segmentation results of a specific semantic target obtained by a semantic segmentation network. This foreground mask *M* is then enlarged through a dilation operation. The enlarged foreground mask *M* guides the bit allocation at the CTU level in the traditional coding process, allocating more bits to CTUs with higher overall foreground proportions. After bitrate allocation using the residual coding resource allocation strategy, the decoder receives the residual *r* and integrates it with the reconstructed video. The summation yields the final decoded video for human vision. The formulas for calculating the proportion of internal foreground and the proportion of overall foreground are as follows: (9) (10) in express( The mask value at the position. Let Pr(i) represent the total pixel value of the CTU, C represent the total number of CTUs, Pr(i) represent the foreground percentage of the i-th CTU, and T(i) represent the overall foreground percentage.

2. The indoor surveillance video encoding method based on human-machine collaborative perception according to claim 1, characterized in that, The extraction of the monitoring background image includes the following steps: An adaptive weighted background update algorithm is designed to dynamically update the background. First, a kernel density clustering algorithm is used to extract background frames. Background is extracted from segments of the surveillance video to obtain a background frame sequence. The time interval for extracting background frames is set to T. The kernel density clustering-based background extraction algorithm first extracts consecutive surveillance video frames of time interval T and establishes a pixel value set for each pixel location. Pixel values ​​of frame regions containing specific semantic target objects detected by the semantic object detection network are not added to the pixel value set for calculation. Each pixel value set is calculated independently to obtain the background pixel values. Finally, the background pixel values ​​form the complete background. Taking any pixel value set as an example, some samples are randomly selected from the corresponding pixel locations in the consecutive video frames, and the average of the samples is taken as the classic point of that pixel value set. Then, using classic points Starting from the point of drift, the update formula for the classic point is: (4) Let be the drift vector of the current classic point position, and its basic form is: (5) (6) S h ( x The classical point is obtained by iterating through the above formula. The drift makes the classic point The position continuously moves towards the position with the highest distribution density of pixel value points until it reaches the convergence point. After obtaining a certain number of convergence points by randomly selecting different samples from the point set multiple times, these convergence points are grouped into a set and clustered again. The final convergence point extracted is used as the background pixel value of the pixel corresponding to the pixel value point set.

Images