Transmission method and device based on intelligent separation of target and background in surveillance video

Abstract

The present invention discloses a transmission method and device based on intelligent separation of target and background in surveillance video, the method comprising: obtaining an image collected by a surveillance acquisition end, the image comprising a current frame and a historical frame; analyzing the current frame and the historical frame, detecting whether the current frame has a target object; if there is no target object, calculating the background brightness information of the current frame, encoding and transmitting it for display as a new historical frame; if there is a target object, and the target object occupies a screen ratio of the current frame that is less than a predetermined value, capturing the target object information and the background brightness information, encoding and transmitting it for display as a new historical frame; otherwise, encoding and transmitting all the information of the current frame for display as a new historical frame. Compared with the prior art, the present invention effectively increases the video compression efficiency and transmission bandwidth by using the technology of separating, transmitting and re-integrating the target and the background.

Description

Technical Field

[0001] The invention relates to a transmission method and a device based on intelligent separation of a target and a background in a monitoring video, belonging to the technical field of video monitoring. Background Art

[0002] At present, the image acquisition and transmission technology of non-movable monitoring equipment uniformly adopts light-sensitive sensors to collect the images, and then compresses the collected raw data through video encoding, and then packages and transmits it through the network. This method is a standard practice at this stage, and almost all monitoring equipment adopts this traditional and simple compression and transmission technology. However, although this solution is relatively simple and easy to understand in principle, this compression and transmission technology processes all data, especially the unchanged background, compresses and transmits unnecessary background data, and reduces video compression efficiency and transmission bandwidth. Summary of the invention

[0003] Based on the above, the present invention provides a transmission method and device based on intelligent separation of target and background in surveillance video, which effectively increases video compression efficiency and transmission bandwidth to overcome the shortcomings of the prior art.

[0004] The technical solution of the present invention is: a transmission method based on intelligent separation of target and background in surveillance video, the method comprising:

[0005] Acquire an image collected by a monitoring acquisition end, wherein the image includes a current frame and a historical frame, wherein the current frame is an image collected by the monitoring acquisition end at the current moment, and the historical frame is a historical image stored by the monitoring receiving end that is closest to the current frame in time;

[0006] Analyze the current frame and the historical frame to detect whether there is a target object in the current frame;

[0007] If the target object does not exist, the background brightness information of the current frame is calculated, the background brightness information is encoded and transmitted to the monitoring receiving end, and the monitoring receiving end decodes the background brightness information and adjusts the historical frame for display as a new historical frame;

[0008] If the target object exists, and the target object occupies a smaller proportion of the current frame than a predetermined value, the target object information and the background brightness information are captured, the target object information and the background brightness information are encoded and transmitted to the monitoring receiving end, and the monitoring receiving end decodes the target object information and the background brightness information and displays them on the historical frame as a new historical frame;

[0009] Otherwise, all the information of the current frame is encoded and transmitted to the monitoring receiving end, which decodes and displays it as a new historical frame.

[0010] Optionally, the background brightness information includes an average brightness ratio, where the average brightness ratio is a ratio of an average brightness value of the current frame to an average brightness value of the historical frame.

[0011] Optionally, the method for capturing the target object information is:

[0012] The features of the entire picture of the current frame are extracted through a multi-layer convolutional network to obtain the image and coordinate information of the target object.

[0013] Optionally, the network transmission from the monitoring acquisition end to the monitoring receiving end adopts the RTP protocol, the first predetermined number of bits of data represents the payload type, and the subsequent predetermined number of bits represents the data type, and the data type respectively represents three transmission situations. The first transmission situation represents the situation where only the background brightness information is transmitted, the second transmission situation represents the situation where the target object information and the background brightness information are transmitted at the same time, and the third transmission situation represents the situation where all the information of the current frame is transmitted.

[0014] The present invention also provides a transmission device based on intelligent separation of target and background in surveillance video, the device comprising:

[0015] An image acquisition module is used to: acquire an image acquired by a monitoring acquisition end, wherein the image includes a current frame and a historical frame, wherein the current frame is an image acquired by the monitoring acquisition end at the current moment, and the historical frame is a historical image stored by the monitoring receiving end and closest to the current frame in time;

[0016] A detection and analysis module, used to: analyze the current frame and the historical frames, and detect whether there is a target object in the current frame;

[0017] A single background processing module, used to: if there is no target object, calculate the background brightness information of the current frame, encode the background brightness information and transmit it to the monitoring receiving end, and the monitoring receiving end decodes the background brightness information and adjusts the historical frame for display as a new historical frame;

[0018] The target processing module is used to: if the target object exists and the proportion of the target object in the current frame is less than a predetermined value, capture the target object information and the background brightness information, encode the target object information and the background brightness information and transmit them to the monitoring receiving end, and the monitoring receiving end decodes the target object information and the background brightness information and displays them on the historical frame as a new historical frame;

[0019] The full image processing module is used to: otherwise, encode all the information of the current frame and transmit it to the monitoring receiving end, which decodes and displays it as a new historical frame.

[0020] The beneficial effects of the present invention are as follows: according to the characteristics of the monitoring screen, the present invention divides the monitoring screen into three situations for transmission. In the first situation, that is, when the background of the monitoring screen remains unchanged and only the brightness changes, only the background brightness information is transmitted, which can save about 80% of the encoding and transmission bandwidth; in the second situation, that is, when the target object appears in the monitoring screen and the proportion of the target object is less than a predetermined value, the target object information is captured and encoded and transmitted, which can save about 50% of the encoding and transmission bandwidth; and in the remaining situations, the entire screen is transmitted in a traditional manner. Compared with the prior art, the present invention effectively increases the video compression efficiency and transmission bandwidth by separating the target and the background, transmitting and then integrating the technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flow chart of the method of the present invention;

[0022] Figure 2 Grab the conceptual diagram for the goal;

[0023] Figure 3 Grab the neural network layer graph for the target;

[0024] Figure 4 Transmitting data graphs to the network;

[0025] Figure 5 Schematic diagram of image restoration in the first case;

[0026] Figure 6 Schematic diagram of image restoration in the second case;

[0027] Figure 7 It is a structural diagram of the device of the present invention. DETAILED DESCRIPTION

[0028] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific implementation disclosed below.

[0029] See also Figure 1 The present embodiment provides a transmission method based on intelligent separation of target and background in surveillance video, the method comprising the following steps:

[0030] S1, obtaining an image collected by a monitoring acquisition end, wherein the image includes a current frame and a historical frame. The current frame is an image collected by the monitoring acquisition end at the current moment, and the historical frame is a historical image stored by a monitoring receiving end that is closest to the current frame in time.

[0031] Specifically, when the monitoring acquisition end starts to monitor images, when the monitoring receiving end is connected for the first time, the monitoring acquisition end will transmit the first image it collects to the monitoring receiving end, and the image at this time is stored in the monitoring receiving end as the initial historical frame. After the historical frame, the image collected by the monitoring acquisition end at the current moment is used as the current frame.

[0032] S2, analyzing the current frame and the historical frames to detect whether there is a target object in the current frame.

[0033] After acquiring the image at the current moment, that is, the current frame, the image of the current frame is analyzed to see whether a target object appears on the image. The target object is a target that does not appear in the historical frame.

[0034] S3: If there is no target object, the background brightness information of the current frame is calculated, the background brightness information is encoded and transmitted to the monitoring receiving end, and the monitoring receiving end decodes the background brightness information and adjusts the historical frame for display as a new historical frame.

[0035] Preferably, the background brightness information includes an average brightness ratio, and the average brightness ratio is the ratio of the average brightness value of the current frame to the average brightness value of the historical frames.

[0036] Specifically, if no target object appears in the picture, the monitoring acquisition end only needs to save the initial background picture as a historical frame, and the subsequent pictures only need to calculate the average brightness ratio between the picture and the background picture, and then send it to the monitoring receiving end. The calculation method is to convert the picture into a grayscale picture, then sum the values ​​of each pixel, and then divide the two sums to get the ratio.

[0037] The average brightness of the image is:

[0038]

[0039] Assume that the average brightness of the current image is L i , the average brightness of the background image is L b . Then the brightness ratio is:

[0040] dL=L i / L b (2)

[0041] Since the field of view of many monitoring devices is static for a long time, there is no picture change except for the background brightness. Therefore, when the monitoring acquisition end does not find any target object of interest through deep learning, the monitoring acquisition end only needs to transmit the background picture to the monitoring receiving end once. Before the target object of interest is captured, it only needs to intermittently (at a self-determined interval) transmit the brightness data P of the background picture to the monitoring receiving end, and the monitoring receiving end needs to save the background image for a long time.

[0042] S4, if the target object exists and the proportion of the target object in the current frame is less than a predetermined value, the target object information and the background brightness information are captured, the target object information and the background brightness information are encoded and transmitted to the monitoring receiving end, and the monitoring receiving end decodes the target object information and the background brightness information and displays them on the historical frame as a new historical frame;

[0043] Preferably, the method for capturing target object information is: extracting features of the entire picture of the current frame through a multi-layer convolutional network to obtain the image and coordinate information of the target object.

[0044] Specifically, during the operation of the camera, the monitoring acquisition end analyzes the screen image in real time and implements the target object detection algorithm on the image. The algorithm divides the image into S×S grids, extracts the features of the entire image through a multi-layer convolutional network, and then regresses the target's coordinates, size, category, confidence and other information according to the center point of the S×S grids. Its conceptual block diagram is as follows Figure 2 As shown in the figure, the image is divided into S×S grids to predict the center point of the target. Then the inference algorithm marks the probability of occurrence of each target in a large number of prediction boxes, and finally outputs the coordinates, size, category and confidence of the target.

[0045] The multi-layer convolutional network structure is as follows: Figure 3 As shown in the figure, the convolution layer is mainly used to extract features, and the fully connected layer is mainly used to predict category probabilities and coordinates. After the prediction is completed, each grid will output data with information dimensions of B×(4+1)+C, where B is the number of target boxes output for each grid, 4 means that each target box has 4 data of upper left corner coordinates (x, y) and width and height (w, h), 1 represents the probability of a target in the target box, and C represents the number of target categories predicted for each grid, one probability for each target, and a total of C target probabilities. Therefore, by traversing the C value of each grid and finding the maximum probability of a certain type of target, the grid predicts the coordinates and width and height of the target.

[0046] Taking yolo v1 as an example, its network structure is not complicated. It is a deep network consisting of a convolutional layer with an activation function, a pooling layer, and two fully connected layers. Its prediction steps are as follows:

[0047] 1. The image is input into the first layer of the convolutional network in a 448×448×3 matrix format;

[0048] 2. Then pass through the pooling layer, and then get the features of this level;

[0049] 3. Enter the next level of convolutional network and pooling layer one by one and obtain the corresponding layer feature data;

[0050] 4. Finally, these feature data are sent to the fully connected layer to complete regression and prediction;

[0051] 5. The best target data can be selected from the regression results of B×(4+1)+C dimensions.

[0052] When the monitoring device detects the presence of interesting targets in the monitoring screen through deep learning networks such as yolo, centerNet and MTCNN models, but the number of targets is small (can be customized according to actual conditions) or the proportion of these targets in the entire image is small (can also be customized according to actual conditions), the monitoring device only needs to extract the image of the target and the coordinates of the image (X i , Y i ), the background brightness P, image and corresponding coordinate data (X i , Y i ) is transmitted to the monitoring receiving end, which restores the background through the saved background image and background brightness P, and then decodes the target image, and uses the target coordinates (X i , Y i ) presents the target on the background to achieve the purpose of restoring the video.

[0053] S5, otherwise, all the information of the current frame is encoded and transmitted to the monitoring receiving end, which decodes and displays it as a new historical frame.

[0054] If the monitoring device finds that the background is constantly changing (not brightness changes) through image contrast technology, or detects many targets (which can be customized according to actual conditions) through the deep learning network model, or the proportion of these targets in the entire image (which can also be customized according to actual conditions) exceeds the preset value, the device automatically uses traditional methods to encode and transmit the entire image, and no longer transmits background brightness and target coordinates.

[0055] Optionally, the network transmission from the monitoring acquisition end to the monitoring receiving end adopts the RTP protocol, the first predetermined number of bits of data represents the payload type, and the subsequent predetermined number of bits represents the data type, and the data type respectively represents three transmission situations. The first transmission situation represents the situation where only the background brightness information is transmitted, the second transmission situation represents the situation where the target object information and the background brightness information are transmitted at the same time, and the third transmission situation represents the situation where all the information of the current frame is transmitted.

[0056] Specifically, this network transmission can be improved on the basis of RTP protocol. The first 7 bits of data represent the payload type TYPE, and the following 1 bit represents the data type I. The payload type of this protocol is set to 123. The data type I represents the above-mentioned transmission of only background brightness, transmission of brightness and target coordinates, and encoding of the entire picture. Please refer to Figure 4 , the meanings of the first N bytes of the data are:

[0057] The value of TYPE is fixed at 123; I is 0, which means that the network data packet only transmits the background brightness or the brightness and the target coordinates; 1 means that the whole picture is encoded and transmitted; BRIGHTNESS indicates the brightness level of the background when I is 0; if I is 1, BRIGHTNESS is invalid data; COUNTS indicates the number of targets after I is 0; if COUNTS is 0, it means that there are no targets behind, and OBJECTS DSCP and OBJECTS DATA are unavailable; CHECKSUM indicates the correctness check of OBJECTS DSCP and OBJECTS DATA data, which can be implemented using various hash algorithms of data. OBJECTS DSCP is the description data of the target. Each target is described by 12 bytes. The first 8 bytes represent the upper left corner coordinates (x, y) and width and height (w, h) of the target rectangle respectively. Each data is a 2-byte integer. The 9th and 10th bytes represent the starting position of the image data of the target in OBJECTS DATA. The 11th and 12th bytes represent the ending position of the image data of the target in OBJECTS DATA. If the number of targets is COUNTS, the length of OBJECTS DSCP is COUNTS*12. OBJECTS DATA is the image data of each target image. There are COUNTS images in total. For the specific description of each image, see OBJECTS DSCP.

[0058] After the monitoring receiver has the background image, i.e. the historical frame, it continues to receive data transmitted by the network. It first determines which case the data frame belongs to based on the I field in the protocol. If it is the first or second case, the video background can be directly restored through the background image and its light intensity ratio P. Then, each target is decoded and the target coordinates (X i , Y i ) displays the target on the background screen, and the video recovery is completed. If it is the third case, the data is received and decoded according to the traditional RTP protocol.

[0059] For the case where only the brightness ratio is transmitted, the image restoration principle diagram of the present invention is as follows: Figure 5 As shown, the dotted frame at the outer edge of the picture indicates that the picture itself is not sent through the network and only participates in the calculation of the brightness ratio. After the receiving device receives the network data packet, if it detects that the I field of the datagram is 0, it extracts the brightness value and the local background picture. Taking the RGB picture as an example, the RGB value of each pixel of the background picture is multiplied by the background brightness ratio. If the result is greater than 255, 255 is taken to obtain a new background picture, which is then displayed.

[0060] For the case of transmitting brightness ratio, position width and height and target data, the image restoration principle of this patent is as follows Figure 6 As shown, the dotted frame on the outer edge of the picture indicates that the picture itself is not sent through the network and only participates in the calculation of the brightness ratio, and the solid line on the left indicates the data to be transmitted through the network.

[0061] When calculating the brightness of the background image, we directly use formulas (1) and (2) to calculate. When calculating the position and width and height of the target, we first detect all the targets in each frame. The detection process can simultaneously obtain the position and width and height of the target. Then we encode the target. Finally, according to the transmission protocol, we attach the target data to the data of the current frame for transmission.

[0062] If there are too many targets or the background changes rapidly, the brightness ratio and target data will no longer be extracted. Instead, all the picture data will be fully compressed and encoded using traditional methods. Coding algorithms such as H.264 or H.265 can be selected. At this time, the image transmission method can use traditional methods such as RTP.

[0063] According to the test, if the monitoring screen is in the first situation of only transmitting the background brightness ratio in most cases, more than 80% of the encoding and transmission bandwidth can be saved; if the monitoring screen is often in the second situation of capturing the target and transmitting, since only the target of interest is encoded, about 50% of the encoding and transmission bandwidth can be saved; if the monitoring screen is in the third situation of full image encoding for a long time, since the third situation adopts the traditional method for encoding and transmission, there is little improvement. The present invention effectively increases the video compression efficiency and transmission bandwidth by using the technology of separating, transmitting and then integrating the target and the background.

[0064] See also Figure 7 The present embodiment also provides a transmission device based on intelligent separation of target and background in surveillance video, the device comprising: an image acquisition module, used to: acquire an image acquired by a surveillance acquisition end, the image comprising a current frame and a historical frame, the current frame being an image acquired by the surveillance acquisition end at the current moment, the historical frame being a historical image stored by a surveillance receiving end closest to the current frame; a detection and analysis module, used to: analyze the current frame and the historical frame, and detect whether the current frame has a target object; a single background processing module, used to: if there is no target object, calculate the background brightness information of the current frame, encode the background brightness information and transmit it to the surveillance receiving end, and the surveillance receiving end processes the background brightness information The target processing module is used to: if the target object exists and the proportion of the target object in the current frame is less than a predetermined value, capture the target object information and the background brightness information, encode the target object information and the background brightness information and transmit them to the monitoring receiving end, which decodes the target object information and the background brightness information and displays them on the historical frame as a new historical frame; the full image processing module is used to: otherwise, encode all the information of the current frame and transmit it to the monitoring receiving end, which decodes and displays it on the historical frame as a new historical frame.

[0065] The implementation process of the functions and effects of each module in the above-mentioned transmission device is specifically described in the implementation process of the corresponding steps in the transmission method based on intelligent separation of target and background in surveillance video, which will not be repeated here.

[0066] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A transmission method based on intelligent separation of target and background in surveillance video, characterized in that: The method comprises: Acquire an image collected by a monitoring acquisition end, wherein the image includes a current frame and a historical frame, wherein the current frame is an image collected by the monitoring acquisition end at the current moment, and the historical frame is a historical image stored by the monitoring receiving end that is closest to the current frame in time; Analyze the current frame and the historical frame to detect whether there is a target object in the current frame; If the target object does not exist, the background brightness information of the current frame is calculated, the background brightness information is encoded and transmitted to the monitoring receiving end, and the monitoring receiving end decodes the background brightness information and adjusts the historical frame for display as a new historical frame; If the target object exists, and the target object occupies a smaller proportion of the current frame than a predetermined value, the target object information and the background brightness information are captured, the target object information and the background brightness information are encoded and transmitted to the monitoring receiving end, and the monitoring receiving end decodes the target object information and the background brightness information and displays them on the historical frame as a new historical frame; Otherwise, all information of the current frame is encoded and transmitted to the monitoring receiving end, which decodes and displays the information as a new historical frame; Among them, the background brightness information includes an average brightness ratio, and the average brightness ratio is the ratio of the average brightness value of the current frame to the average brightness value of the historical frame; the method for capturing the target object information is: extracting the features of the entire picture of the current frame through a multi-layer convolutional network, and obtaining the image and coordinate information of the target object; the network transmission from the monitoring acquisition end to the monitoring receiving end adopts the RTP protocol, the first predetermined number of bits of data represents the load type, and the subsequent predetermined number of bits represents the data type, and the data type represents three transmission conditions respectively, the first transmission condition represents the situation of transmitting only the background brightness information, the second transmission condition represents the situation of transmitting the target object information and the background brightness information at the same time, and the third transmission condition represents the situation of transmitting all the information of the current frame.

2. A transmission device based on intelligent separation of target and background in surveillance video, characterized in that: The device comprises: An image acquisition module is used to: acquire an image acquired by a monitoring acquisition end, wherein the image includes a current frame and a historical frame, wherein the current frame is an image acquired by the monitoring acquisition end at the current moment, and the historical frame is a historical image stored by the monitoring receiving end and closest to the current frame in time; A detection and analysis module, used to: analyze the current frame and the historical frames, and detect whether there is a target object in the current frame; A single background processing module, used to: if there is no target object, calculate the background brightness information of the current frame, encode the background brightness information and transmit it to the monitoring receiving end, and the monitoring receiving end decodes the background brightness information and adjusts the historical frame for display as a new historical frame; The target processing module is used to: if the target object exists and the proportion of the target object in the current frame is less than a predetermined value, capture the target object information and the background brightness information, encode the target object information and the background brightness information and transmit them to the monitoring receiving end, and the monitoring receiving end decodes the target object information and the background brightness information and displays them on the historical frame as a new historical frame; The full image processing module is used to: otherwise, encode all the information of the current frame and transmit it to the monitoring receiving end, which decodes and displays it as a new historical frame; Among them, the background brightness information includes an average brightness ratio, and the average brightness ratio is the ratio of the average brightness value of the current frame to the average brightness value of the historical frame; the method for capturing the target object information is: extracting the features of the entire picture of the current frame through a multi-layer convolutional network, and obtaining the image and coordinate information of the target object; the network transmission from the monitoring acquisition end to the monitoring receiving end adopts the RTP protocol, the first predetermined number of bits of data represents the load type, and the subsequent predetermined number of bits represents the data type, and the data type represents three transmission conditions respectively, the first transmission condition represents the situation of transmitting only the background brightness information, the second transmission condition represents the situation of transmitting the target object information and the background brightness information at the same time, and the third transmission condition represents the situation of transmitting all the information of the current frame.

Images