Image target detection method, video stream compression method and device

Through the improved YOLOv8 image object detection method and differentiated compression strategy, the areas of interest and background areas in the traffic monitoring video stream are identified and separated, and the problems of waste of resources and inefficient processing in the prior art are solved, and efficient video is achieved Stream compression and processing.

CN118921475AActive Publication Date: 2024-11-08WUXI UNIV

Patent Information

Application Number
CN202411405439.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-10
Publication Date
2024-11-08
Estimated Expiration
2044-10-10

AI Technical Summary

Technical Problem

The prior art is difficult to effectively optimize the encoding efficiency and storage cost of traffic monitoring video streams, especially when there are a large number of static areas and duplicate information in complex environments, resulting in waste of resources and inefficient processing.

Method used

Using the improved YOLOv8 image object detection method, feature maps are extracted through the CSC module and the SC-Block module, feature fusion is combined with CAA and star operations, and the region of interest is identified, and the video stream is divided into video stream of interest and background video stream. Adopt differentiated compression strategies.

Benefits of technology

It improves the compression efficiency and resource utilization of video streams, reduces storage and transmission costs, ensures the real-time and accuracy of traffic monitoring video streams, and adapts to the massive video stream processing needs in complex environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118921475A_ABST
    Figure CN118921475A_ABST
Patent Text Reader

Abstract

The invention discloses an image target detection method and a video stream compression method and device, relates to the technical field of video stream compression, and solves the problem that traffic monitoring video stream optimization based on an advanced coding technology and an artificial intelligence technology is lacked in the prior art. Splitting the video stream into a plurality of images; the improved YOLOv8 performs target detection on a region of interest of an image; generating a final binary mask image from one image; an interested image and a background image are extracted from an image with an interested area through the final binary mask graph; repeating the operation until the interested image and the background image are extracted from the plurality of images, and continuing to execute the operation; a plurality of interested images are combined to form an interested video stream, and a plurality of background images are combined to form a background video stream; and merging and compressing the interested video stream and the background video stream by adopting different compression levels, and merging the compressed interested video stream and the background video stream to obtain a compressed video stream.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of video stream compression, and in particular to an image target detection method, a video stream compression method and a device. Background Art

[0002] With the development of artificial intelligence, intelligent transportation systems have become an important trend in the development of modern cities, and the storage and processing of traffic monitoring video streams are key links in intelligent transportation systems. As one of the core components of intelligent transportation systems, the real-time and accuracy of traffic monitoring video streams directly affect the effect of traffic management and urban traffic safety. Through intelligent monitoring cameras, traffic conditions on the road can be captured in real time, including information such as traffic density, speed, and vehicle type, providing real-time traffic status data for traffic management departments. These data can not only be used for real-time traffic control, but also guide traffic planning and road construction through historical data analysis to improve urban transportation efficiency. At the same time, traffic monitoring video streams can also detect and handle traffic accidents and congestion in a timely manner, providing strong support for urban traffic safety. However, the characteristics of traffic monitoring video streams require corresponding hardware devices to store huge amounts of data, but huge storage space and high-resolution imaging equipment are very expensive. For example, the captured YUV (video color coding system) video resolution is 1080p, and the number of traffic monitoring cameras in a medium-sized city is at least 5,000. At a bit rate of 4Mbps, the concurrently written bit streams in one day can reach 1648TB. In addition, due to the particularity of traffic monitoring video streams, it is usually necessary to save the video streams for a certain number of days. Faced with such a huge video stream, how to improve encoding efficiency, speed up transmission, reduce storage costs, and process and analyze the information of the video stream in a timely and effective manner has become a series of important challenges currently faced.

[0003] At present, some advanced video coding standards such as H.265 / HEVC (High Efficiency Video Coding) have been widely adopted. Compared with existing coding standards, H.265 / HEVC can reduce the size of video stream files by about 50% while maintaining the same video stream quality. But even so, it still cannot adapt well to the characteristics of traffic monitoring video streams. Due to the complexity of the road environment and the variability of traffic conditions, there are often a large number of static areas and repeated information in traffic monitoring video streams. The existing coding standards have limited effect in compressing this information, resulting in waste of resources and low processing efficiency, and cannot meet the needs of intelligent transportation system construction.

[0004] With the development of artificial intelligence technology, video coding technologies based on deep learning are emerging. Among them, traffic monitoring video stream compression based on regions of interest has become a research direction that has attracted much attention. By identifying and extracting regions of interest in traffic monitoring video streams, the size of video stream files can be effectively reduced, storage and transmission costs can be reduced, and the effective capture and transmission of traffic information can be ensured. This compression technology can process video stream data in a targeted manner according to the characteristics of the actual monitoring scene, avoid repeated encoding of static and repeated information, and improve compression efficiency and resource utilization. At the same time, for the extracted regions of interest, secondary development can also be carried out at the upper layer, such as adding functions such as danger warning, traffic flow monitoring, license plate recognition, and vehicle violation monitoring, making the intelligent video surveillance system more flexible and efficient, and improving the monitoring and analysis effect of high-risk areas.

[0005] In summary, how to optimize the data processing and storage of traffic monitoring video streams through advanced coding technology and artificial intelligence technology has become an important topic in the research of intelligent transportation systems. This will help improve the efficiency and safety of urban traffic management and provide strong technical support for the development of intelligent transportation systems. Summary of the invention

[0006] The present invention solves the problem that the prior art lacks optimization of traffic monitoring video streams based on advanced coding technology and artificial intelligence technology.

[0007] The improved YOLOv8 method for detecting an image target in the present invention sequentially inputs an image into the backbone network of YOLOv8, the neck network of YOLOv8, and the head network of YOLOv8 for detecting an image target, specifically: The image is input into the CSC module in the backbone network of YOLOv8 and the CSC module in the neck network of YOLOv8 for feature extraction and output feature maps; In the CSC module, the image is subjected to feature extraction by the CBS module to output a feature map. The feature map is divided into two branches by Split. One branch feature map is directly input into Concat, and the other branch feature map is subjected to feature fusion by multiple SC-Block modules to output the fused feature map. After the feature maps of the two branches are Concat, they are passed through the CBS module again to output the feature map. In the SC-Block module, another branch feature map is subjected to feature fusion through CAA and star operation to output a fused feature map.

[0008] Further, in one embodiment of the present invention, the feature fusion of the other branch feature map is performed through CAA and star operation, specifically: ; in, The feature map of another branch is subjected to feature fusion through CAA and star operation. is the adaptive weight coefficient, is a position extracted from the image The local characteristics of For a certain location The corresponding global context information.

[0009] Further, in one embodiment of the present invention, the feature map of another branch is subjected to feature fusion through CAA and star operation to output the fused feature map, which is specifically: The feature map of the other branch is extracted by CAA to output a feature map. The feature map is divided into the first branch and the second branch through DW-Conv. The feature maps of the first branch and the second branch are respectively input into the FC layer for operation to generate feature maps again. The star operation merges the feature maps of the first branch and the second branch to generate an updated feature map. The updated feature map is sequentially input into the FC layer and DW-Conv to generate a fused feature map.

[0010] Furthermore, in one embodiment of the present invention, the updated feature map is specifically: ; in, is the updated feature map, is the feature map, is the adaptive weight, Is a logical operation symbol.

[0011] The video stream compression method of the present invention is implemented by using the improved YOLOv8 image target detection method described in the above method, and comprises the following steps: Step S1, splitting a video stream into multiple images; Step S2, performing target detection on a region of interest of an image based on the improved YOLOv8 to form an image with the region of interest; Step S3, generating a preliminary binary mask image from an image, and sequentially performing erosion, dilation and Gaussian blur operations on the preliminary binary mask image to generate a final binary mask image; Step S4, the final binary mask image extracts an image of interest and a background image from an image having a region of interest; Step S5, repeating the operations from step S2 to step S4 until the image of interest and the background image are extracted from the multiple images, then executing the operation of step S6; Step S6, a plurality of images of interest are combined to form an interested video stream, and a plurality of background images are combined to form a background video stream; Step S7: The video stream of interest and the background video stream are respectively combined and compressed using different compression levels, and a compressed video stream is obtained based on the combined and compressed video stream of interest and background video stream.

[0012] Further, in one embodiment of the present invention, in the step S3, the operation of corroding the preliminary binary mask image is specifically: ; in, To perform the erosion operation on the preliminary binary mask image, For the images, The structural element for the initial binary mask image when performing the erosion operation. is the structural element relative to the pixel The horizontal offset of is the structural element relative to the pixel The vertical offset of is the pixel coordinate of the image, is the preliminary binary mask map; The operation of dilating the preliminary binary mask image is specifically as follows: ; in, To dilate the initial binary mask image, The structural element used to dilate the initial binary mask image.

[0013] Further, in one embodiment of the present invention, in step S3, the final binary mask image is specifically: ; in, is the final binary mask image, is the pixel coordinate of the image, is the structural element relative to the pixel The horizontal offset of is the structural element relative to the pixel The vertical offset of is the kernel function of Gaussian blur, is the standard deviation of the kernel function of Gaussian blur, It is an operation to dilate the preliminary binary mask image.

[0014] Further, in one embodiment of the present invention, in step S6, the video stream of interest is specifically: ; in, For the video stream of interest, is the final binary mask image, For the images, is the pixel coordinate of the image, is a sequence of images with a region of interest; The background video stream is specifically: ; in, For background video stream, It is the inverse mask of the final binary mask image.

[0015] The traffic monitoring video compression device of the present invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor. The processor executes the program to implement the video stream compression method as described in the above method.

[0016] The present invention solves the problem that the prior art lacks advanced coding technology and artificial intelligence technology to optimize traffic monitoring video streams. Specific beneficial effects include: 1. The improved YOLOv8 method for detecting image targets described in the present invention. In the prior art, in a complex and changeable environment, the video stream may be affected by a variety of interferences and noises, and these interference factors will directly affect the accuracy of image target detection. In addition, video streams are usually generated continuously, and the amount of data is huge, requiring the system to be able to efficiently process and analyze the video stream in a short time. In order to solve the above technical problems, the present invention designs an improved YOLOv8, which can maintain efficient and accurate detection capabilities in complex environments, and can respond and process massive video streams in real time and quickly, so as to ensure the real-time requirements in actual application scenarios; 2. The improved YOLOv8 method for detecting target objects in images described in the present invention introduces a newly designed SC-Block module through the proposed CSC module. The SC-Block module significantly reduces the computational cost through the effects of DW-Conv and star operation. Specifically, DW-Conv effectively reduces the amount of computation and parameters, while the star operation optimizes the dynamic processing of feature maps, so that the improved YOLOv8 can better process features of different scales while maintaining high efficiency; 3. The improved YOLOv8 method for detecting image targets in the present invention, the SC-Block module is combined with CAA, which can obtain the context information of distant targets, thereby improving the accuracy and robustness of detecting distant small targets in video streams; 4. The video stream compression method described in the present invention. In the prior art, the existing video encoding algorithm usually adopts a unified compression standard for the entire image, and does not perform differentiated processing according to the different image contents, resulting in waste of resources and low processing efficiency. In order to solve the above technical problems, the present invention automatically identifies and extracts the region of interest in the video stream by utilizing the improved YOLOv8 to generate a final binary mask map. The video stream is then divided into two parts, including the video stream of interest and the background video stream, and a differentiated compression strategy is adopted. The video stream of interest adopts a higher compression quality because it contains more important information, while the background video stream uses a lower compression quality, thereby effectively reducing the overall size of the video file. Thereby avoiding the technical problems of waste of resources and low processing efficiency; 5. The traffic monitoring video compression device described in the present invention lacks technical means for optimizing traffic monitoring videos based on advanced coding technology and artificial intelligence technology in the prior art. In order to solve the above technical problems, the present invention adopts the improved YOLOv8 to perform image target detection, identify and calibrate the area of ​​interest in the video stream, and then preprocesses each image to generate a final binary mask map. The final mask map is used to separate the video stream into two parts including the video stream of interest and the background video stream, and different compression levels are used respectively, thereby significantly reducing the size of the video stream file, while ensuring the effective capture of key traffic monitoring video stream information; The video stream compression method of the present invention is applied to the field of analyzing key areas and performs better than the existing uniform compression method. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which: Figure 1 It is a structural diagram of the improved YOLOv8 described in Implementation Mode 1; Figure 2 is a structural diagram of a CSC module according to the first embodiment; Figure 3 is a block diagram of the SC-Block module described in Embodiment 1; Figure 4 It is the video stream separation diagram described in Implementation Mode 2; Figure 5 It is a comparison chart of various video stream quality evaluation indicators of the existing uniform compression method described in Implementation Mode 3 and the traffic monitoring video compression device described in this implementation mode. DETAILED DESCRIPTION

[0018] The following will clearly and completely describe various embodiments of the present invention in conjunction with the accompanying drawings. The embodiments described with reference to the accompanying drawings are exemplary and intended to be used to explain the present invention, but should not be understood as limiting the present invention.

[0019] Implementation method 1: The improved YOLOv8 method for detecting an image target described in this implementation method sequentially inputs an image into the backbone network of YOLOv8, the neck network of YOLOv8, and the head network of YOLOv8 for image target detection, specifically: The image is input into the CSC module in the backbone network of YOLOv8 and the CSC module in the neck network of YOLOv8 for feature extraction and output feature maps; In the CSC module, the image is subjected to feature extraction by the CBS module to output a feature map. The feature map is divided into two branches by Split. One branch feature map is directly input into Concat, and the other branch feature map is subjected to feature fusion by multiple SC-Block modules to output the fused feature map. After the feature maps of the two branches are Concat, they are passed through the CBS module again to output the feature map. In the SC-Block module, another branch feature map is subjected to feature fusion through CAA and star operation to output a fused feature map.

[0020] In this implementation manner, the feature fusion of the other branch feature map is performed through CAA and star operation, specifically: ; in, The feature map of another branch is subjected to feature fusion through CAA and star operation. is the adaptive weight coefficient, is a position extracted from the image The local characteristics of For a certain location The corresponding global context information.

[0021] In this implementation manner, the feature map of another branch is subjected to feature fusion through CAA and star operation to output the fused feature map, specifically: The feature map of the other branch is extracted by CAA to output a feature map. The feature map is divided into the first branch and the second branch through DW-Conv. The feature maps of the first branch and the second branch are respectively input into the FC layer for operation to generate feature maps again. The star operation merges the feature maps of the first branch and the second branch to generate an updated feature map. The updated feature map is sequentially input into the FC layer and DW-Conv to generate a fused feature map.

[0022] In this implementation manner, the updated feature map is specifically: ; in, is the updated feature map, is the feature map, is the adaptive weight, Is a logical operation symbol.

[0023] In a complex and ever-changing environment, video streams may be affected by a variety of interference and noise, such as changes in lighting, fluctuations in weather conditions (such as rain, snow, haze, etc.), object occlusion, and jitter of the camera equipment. These interference factors will directly affect the accuracy of image target detection. In addition, video streams are usually generated continuously, and the amount of data is huge, requiring the system to be able to efficiently process and analyze the video stream in a short time. In order to meet this requirement, YOLOv8 (YOLO version 8) not only needs to have a high degree of accuracy, be able to accurately identify image targets and mark the boundaries of the area of ​​interest, but also must have an extremely high processing speed, and be able to respond and process massive video streams in real time and quickly to ensure the real-time requirements in actual application scenarios.

[0024] To solve the above technical problems, Figure 1 As shown, this embodiment uses YOLOv8 as the basic model to design an improved YOLOv8 image target detection method, specifically: Video streams are generally formed by continuously collecting images. Therefore, the video stream is split into multiple images. For an image target detection: First, in the backbone network of YOLOv8, a processed image is input into the Conv layer (convolutional layer) to reduce the dimension of the image. After passing through the Conv layer, the image passes through four cascaded Conv layers and the CSC module (star anchor context feature extraction module) to output a feature map. The feature map is sent to the SPPF layer (spatial pooling layer) for feature extraction again and outputs the feature map.

[0025] In the prior art, although the C2f module (feature extraction module) in the backbone network of YOLOv8 improves the feature extraction capability of the network to a certain extent, its structural complexity is relatively high, the computational cost is relatively high, and when processing continuous frames of the video stream, there are problems of repeated feature calculation and insufficient real-time performance.

[0026] In order to solve the above technical problems, the present embodiment designs a CSC module, such as Figure 2As shown in the figure, in the CSC module, first, the feature map is subjected to the CBS structure (convolution, batch normalization, SiLU activation function (adaptive activation function)) for preliminary feature extraction and conversion of the output feature map. Then, the feature map is divided into two branches through the Split operation, where one branch feature map is directly passed to the Concat layer (feature fusion layer), and the other branch feature map is processed by multiple SC-Block modules (star anchor context modules). Finally, the feature maps of the two branches are spliced ​​in the channel dimension to achieve feature fusion.

[0027] The CSC module not only maintains the original feature extraction capability of the C2f module, but also further optimizes the calculation path and reduces redundant calculations. It also significantly improves the real-time processing speed by simplifying the structure.

[0028] In the SC-Block module, such as Figure 3 As shown in the figure, first, the feature map uses CAA (context anchor attention) to capture the context information of distant targets and enhance the expressiveness of the central features during the feature extraction process, thereby outputting the feature map. After that, the extracted feature map is input into DW-Conv (depthwise separable convolution) for convolution operation and is divided into the first branch and the second branch. The feature maps of the first branch and the second branch are respectively input into the FC layer for operation to generate the feature map again. Then, the feature map is intelligently merged through the star operation and input into the FC layer and DW-Conv in sequence to generate the fused feature map.

[0029] CAA can effectively capture the contextual information of distant targets, and weightedly fuse the contextual information of distant targets with the central features through the interaction between the star operation and CAA, so that the model can maintain high accuracy when processing distant small targets. This solves the technical difficulties between the contextual information of distant targets and the central features, and overcomes the shortcomings of the existing attention mechanism in long-distance dependency modeling.

[0030] Specifically, the role of CAA and star operation is to perform feature fusion through the following formula: ; in, The feature map of another branch is subjected to feature fusion through CAA and star operation. is the adaptive weight coefficient, is a position extracted from the image The local characteristics of For a certain location The corresponding global context information.

[0031] On this basis, the star operation dynamically processes the feature map by optimizing the feature map point by point. Assume that the input feature map is ,in, is a real number domain, representing the range of each pixel value of the feature map. is the number of channels, is the height, is the width. The star operation is applied to the feature map Apply point-wise adaptive weights , generate the updated feature map: ; in, is the updated feature map, Is a logical operation symbol.

[0032] The star operation achieves adaptive weighting of feature maps by multiplying points one by one, allowing YOLOv8 to dynamically adjust the importance of each feature point according to the feature distribution, thereby optimizing the sensitivity of local features to targets of different scales. This operation also solves the problem of insufficient feature response after the introduction of CAA, making the processing of feature maps more refined.

[0033] After the feature map enhanced by CAA is combined with the feature map optimized by the star operation, the final feature map is output by the following formula: ; in, is a nonlinear activation function, is the final generated feature map.

[0034] Therefore, the role of star operation and CAA solves the problems of feature redundancy and local information loss while maintaining low computational complexity and improving the adaptability to targets of different scales.

[0035] Secondly, in the neck network of YOLOv8, such as Figure 1 As shown in the figure, the feature map first passes through the Upsample layer (upsample layer), and then is input into the Concat layer together with the feature map extracted by the CSC module in the backbone network to output the feature map. This feature map is sent to the CSC module for feature extraction and output feature map. The feature map passes through the Upsample layer, CSC module, and CSC module again to output the first size feature map. Subsequently, the first size feature map is sent to the Conv layer for dimensionality reduction, and the reduced dimensionality feature map is combined with the feature map extracted by the CSC module. Figure 1 And input the Concat layer, output the feature map, the feature map is extracted by the CSC module to output the second size feature map, the feature map is again reduced in dimension by the Conv layer, and the feature map after dimension reduction is the same as the feature map extracted by the SPPF layer. Figure 1And input the Concat layer, output the feature map, and the feature map is extracted by the CSC module to output the third size feature map.

[0036] Finally, in the head network of YOLOv8, the first-size, second-size, and third-size feature maps are all input to the Detect head for image object detection.

[0037] Therefore, the improved YOLOv8 designed in this embodiment not only has a high degree of accuracy, can accurately identify the target object and mark the boundaries of the region of interest, but also must have an extremely high processing speed, and can respond and process massive video data in real time and quickly to ensure the real-time requirements in actual application scenarios.

[0038] Implementation method 2: The video stream compression method described in this implementation method is implemented by using the improved YOLOv8 image target detection method described in implementation method 1, and includes the following steps: Step S1, splitting a video stream into multiple images; Step S2, performing target detection on a region of interest of an image based on the improved YOLOv8 to form an image with the region of interest; Step S3, generating a preliminary binary mask image from an image, and sequentially performing erosion, dilation and Gaussian blur operations on the preliminary binary mask image to generate a final binary mask image; Step S4, the final binary mask image extracts an image of interest and a background image from an image having a region of interest; Step S5, repeating the operations from step S2 to step S4 until the image of interest and the background image are extracted from the multiple images, then executing the operation of step S6; Step S6, a plurality of images of interest are combined to form an interested video stream, and a plurality of background images are combined to form a background video stream; Step S7: The video stream of interest and the background video stream are respectively combined and compressed using different compression levels, and a compressed video stream is obtained based on the combined and compressed video stream of interest and background video stream.

[0039] In this implementation manner, in step S3, the operation of corroding the preliminary binary mask image is specifically as follows: ; in, To perform the erosion operation on the preliminary binary mask image, For the images, The structural element for the initial binary mask image when performing the erosion operation. is the structural element relative to the pixel The horizontal offset of is the structural element relative to the pixel The vertical offset of is the pixel coordinate of the image, is the preliminary binary mask map; The operation of dilating the preliminary binary mask image is specifically as follows: ; in, To dilate the initial binary mask image, The structural element used to dilate the initial binary mask image.

[0040] In this implementation manner, in the step S3, the final binary mask image is specifically: ; in, is the final binary mask image, is the pixel coordinate of the image, is the structural element relative to the pixel The horizontal offset of is the structural element relative to the pixel The vertical offset of is the kernel function of Gaussian blur, is the standard deviation of the kernel function of Gaussian blur, It is an operation to dilate the preliminary binary mask image.

[0041] In this implementation manner, in step S6, the video stream of interest is specifically: ; in, For the video stream of interest, is the final binary mask image, For the images, is the pixel coordinate of the image, is a sequence of images with a region of interest; The background video stream is specifically: ; in, For background video stream, It is the inverse mask of the final binary mask image.

[0042] In a video stream, there are usually long-term unchanged background areas and moving foreground areas. The background area occupies the vast majority of the image area. These areas usually change less and contain a lot of redundant information. Although the moving foreground area is smaller in area, it contains more important information than the background area, such as dynamic objects such as vehicles and pedestrians. Such moving foreground areas are often the key to the video stream because they provide key data about the video stream information. However, existing video coding algorithms have limited effect in dealing with these characteristics. Existing video coding algorithms do not fully utilize the redundancy of the background area, resulting in repeated encoding of the static background, wasting a lot of storage space and transmission bandwidth. At the same time, since these algorithms usually use a unified compression standard for the entire image, they do not perform differentiated processing based on the different image contents, resulting in a waste of resources and low processing efficiency.

[0043] In order to solve the above technical problems, this embodiment designs a video stream compression method, which includes the following steps: Step S1, extracting the region of interest of the video stream: The video stream collected by the camera is generally formed by continuously collecting images. Therefore, the video stream is split into multiple images, denoted as , Indicates After being collected, the multiple images are usually stored and transmitted in YUV video coding format. In order to process and analyze the multiple images, they usually need to be decoded into a standard RGB image format first. The improved YOLOv8 based on the first implementation method performs target detection on the regions of interest of the multiple images to form multiple images with regions of interest.

[0044] Step S2, generate the final binary mask image: Generate a preliminary binary mask for an image .in, Represents the pixel coordinates of the image, traversing each pixel. When there is at least one bounding box ,in, For the A set of bounding boxes of the image region of interest, such that pixels If it is within this bounding box, The value of is 1, otherwise it is 0. It becomes a binary image, where each pixel is either 1, that is, it belongs to any region of interest, or 0, that is, it does not belong to any region of interest. The specific formula is as follows: ; in, After non-maximum suppression, The set of all bounding boxes in the frame, It is The coordinates of the upper left and lower right corners of the bounding box.

[0045] After generating the preliminary binary mask, further post-processing is required. These post-processing operations mainly include morphological operations such as dilation and erosion. The main purpose of the dilation operation is to expand the foreground area, that is, the area where the median value of the mask is 1, which can fill small holes in the foreground area and expand the boundary of the foreground object outward. The main purpose of the erosion operation is to shrink the foreground area, that is, the area where the median value of the mask is 1, which can remove the edge noise of the foreground object and shrink the boundary of the foreground object inward. The preliminary binary mask is eroded and then dilated. as well as They are the structural elements for corrosion and expansion operations respectively: ; ; in, To perform the erosion operation on the preliminary binary mask image, For the images, is the structural element relative to the pixel The horizontal offset of is the structural element relative to the pixel The vertical offset of is the pixel coordinate of the image, It is an operation to dilate the preliminary binary mask image.

[0046] The initial binary mask image is post-processed to improve its quality and accuracy.

[0047] Assume that the kernel function of Gaussian blur is ,in, The standard deviation of the kernel function of Gaussian blur generates the final binary mask map It can be expressed as: ; Gaussian blurring helps to reduce the discontinuity of the boundary, making the processed preliminary binary mask more suitable for subsequent image processing steps. And the edges of the preliminary binary mask after Gaussian blurring become softer and smoother, making the boundary no longer too sharp and the transition more natural.

[0048] Step S3, video stream separation: like Figure 4 As shown, using the final binary mask map Extract the image of interest and the background image from multiple images with regions of interest. Specifically: Perform a pixel-by-pixel bitwise AND operation and use the final binary mask to extract multiple images of interest from multiple images with regions of interest, i.e., keep the pixels with a mask value of 1 and set the other pixels to 0. Similarly, for each pixel position Perform pixel-by-pixel bitwise AND operations, and use the inverse mask of the final binary mask image to extract multiple background images from multiple images with regions of interest, that is, keep the pixels with inverse mask image 1, and set other pixels to 0. Multiple images of interest are combined to form an interested video stream, and multiple background images are combined to form a background video stream. Assume that the image sequence with the region of interest is , the video stream of interest and the background video stream can be expressed as: ; ; in, For the video stream of interest, For background video stream, It is the inverse mask of the final binary mask image.

[0049] Step S4, video stream hierarchical compression and merging: like Figure 4 As shown, different compression levels are used to process the two video streams of interest and background video streams. Specifically, the existing H.264 / H.265 codec framework is selected for the compression of the video stream of interest and the background video stream. The video stream compression method composed of this is called CA-X264 / CA-X265, and CRF (constant rate factor) is used to control the quality of the video stream. In CRF mode, the encoder automatically adjusts the quantization value so that the output video stream reaches the quality level specified by the user. Specifically, when a smaller CRF value is set, the encoder will maintain higher quality, but the corresponding bit rate will increase, that is, a smaller CRF value is set for the video stream of interest to ensure the integrity of key information; and when a larger CRF value is set, the encoder will compress at a higher bit rate, that is, a larger CRF value is set for the background video stream in the video stream to reduce the image quality while reducing the storage and transmission costs. By adjusting the CRF value, the quality of the video stream can be easily controlled without manually adjusting more complex parameters.

[0050] Then the two video streams are added together to obtain the final compressed and encoded video stream.

[0051] Therefore, this embodiment uses the improved YOLOv8 to automatically identify and extract the region of interest in the video stream and generate the final binary mask image. The video stream is then divided into two parts, including the video stream of interest and the background video stream, and a differentiated compression strategy is adopted. The video stream of interest uses a higher compression quality because it contains more important information, while the background video stream uses a lower compression quality, thereby effectively reducing the overall size of the video file.

[0052] Implementation method three: The traffic monitoring video compression device described in this implementation method, a memory, a processor and a computer program stored in the memory and executable on the processor, the processor executes the program to implement the video stream compression method described in implementation method two.

[0053] This embodiment loads the video stream compression method described in Embodiment 2 into a traffic monitoring video compression device, including the following steps: Step S1, marking of the region of interest: First, we collected video stream data in real traffic monitoring scenarios, and used random frame extraction to save 4,000 monitoring pictures under different weather conditions, different intersections, and different lighting conditions as a data set. Then we used annotation software to annotate them, and the annotated content was the required area of ​​interest. The annotation categories selected in this implementation were non-motor vehicles, pedestrians, ordinary vehicles, and special work vehicles.

[0054] Step S2, improve YOLOv8: In the backbone network and neck network of YOLOv8, CSC modules are designed to extract features and output feature maps. After the improvement of YOLOv8, the previously labeled dataset is used for training.

[0055] Step S3, extracting the region of interest of the video stream: On the computer side, use the cv2.VideoCapture function in OpenCV (cross-platform computer vision library) to pull the video stream of the RTSP protocol (real-time streaming protocol), then use the cap.read() function to read the image frame by frame from the video stream, and then use the cv2.cvtColor function and cv2.resize function to decode and preprocess the image read frame by frame. Use the trained ONNX format (Open Neural Network Exchange Format) model for target detection. The ONNX Runtime library provides the function of loading and reasoning models in the ONNX format. Load the model through onnxruntime.InferenceSession and use session.run for reasoning to obtain the target detection results, including the bounding boxes of multiple target areas and the corresponding confidences. Post-process the detection results, apply NMS (non-maximum suppression) to remove overlapping bounding boxes, and obtain the final list containing the coordinate information of the area of ​​interest.

[0056] Step S4, generating the final binary mask image and video stream separation: The final binary mask is generated based on the bounding box after NMS. Each pixel value in the final binary mask is either 1 or 0. The detailed process is as follows: Initialize an all-zero matrix of the same size as the image as the binary mask. Traverse all bounding boxes and set the pixel values ​​in the bounding boxes to 1. Perform erosion and dilation operations on the initialized binary mask in turn to remove noise and enhance the area of ​​interest. Use Gaussian blur operation to smooth and generate the final binary mask. Then, the smoothed final binary mask and the inverse mask of the final binary mask are bitwise ANDed with the video stream respectively to separate the video stream into two paths, one of which contains only the video stream of interest and the other contains only the background video stream.

[0057] Step S5, video stream hierarchical compression and merging: Compress the separated video streams. Use different compression quality parameters for different video streams. Compress the video stream containing the video of interest, use a higher compression quality parameter, that is, a lower CRF value, to retain more detail information. Compress the video stream containing the background, use a lower compression quality parameter, that is, a higher CRF value, to reduce the file size. Finally, add the two compressed video streams element by element to obtain the processed video stream.

[0058] In order to comprehensively evaluate the performance difference between the traffic monitoring video compression device described in this embodiment and the existing CA-Comp (uniform compression method and content-aware compression) method in video stream processing, this embodiment will use FFmpeg (computer program)'s X264 (a type of codec) and X265 (a type of codec) to conduct a detailed video stream compression study, that is, the existing uniform compression method. The focus will be on comparing the performance of these two compression methods under different CRF values, and analyzing their specific impact on video stream quality and compression efficiency. This time, 23 and 28 will be used as the default CRF values ​​in H.264 and H.265. These CRF values ​​correspond to compression ratios of approximately 1:250 and 1:375, respectively.

[0059] After that, a variety of video stream quality evaluation indicators, including PSNR (peak signal-to-noise ratio) and SSIM (structural similarity), will be used to objectively evaluate the quality of the compressed video stream. In addition, VAMF (video quality assessment method) is used to subjectively evaluate the video stream. Using the binary mask map generated by Mask_PSNR (an indicator of image quality assessment) and Mask_SSIM (an indicator of image quality assessment), the PSNR and SSIM values ​​of the region of interest in each frame are calculated. All the following evaluation indicators are calculated using the professional tool MSU Video Quality Measurement Tool.

[0060] like Figure 5 As shown, the quality of the reconstructed frame after the traffic monitoring video stream is compressed by the traffic monitoring video compression device described in this embodiment and the existing uniform compression method is given. The test video stream uses a video stream pulled in real time by traffic monitoring. Among them, "H.26..." all use the existing uniform compression method, "CA-X26..." all use the traffic monitoring video compression device described in this embodiment, ROI is the video stream of interest, and N-ROI is the background video stream.

[0061] For the traffic monitoring video compression device described in this embodiment and the existing uniform compression method, as the CRF value increases, PSNR, SSIM and VAMF all decrease significantly. In particular, when the CRF value reaches 28 or higher, the quality decreases more significantly.

[0062] In addition, under the same compression ratio, for example, number C and number D, number I and number J, the PSNR and SSIM of the existing uniform compression method are higher than those of the traffic monitoring video compression device described in this embodiment. The reason is that the background video stream accounts for a large proportion of the test video stream, about 70%, and the traffic monitoring video compression device described in this embodiment uses a higher CRF value in the background video stream, which causes the quality of the background video stream to drop significantly, thereby lowering the PSNR and SSIM values ​​of the overall video stream. Although the video stream of interest maintains a higher quality, which is in line with its original design intention, the performance of this method in terms of the overall video stream quality is compromised due to the widespread existence of the background video stream. This situation is particularly evident in the results of VAMF, Mask_PSNR and Mask_SSIM. These three indicators focus on the quality of the video stream of interest, and the traffic monitoring video compression device described in this embodiment performs better than the existing uniform compression method in these three indicators.

[0063] It can be concluded that the advantages of the traffic monitoring video compression device described in this embodiment are mainly reflected in applications with clear video streams of interest. If the importance of the video stream of interest in the video is much higher than the background video stream, the traffic monitoring video compression device described in this embodiment is obviously a better choice. On the contrary, when the importance of each part in the video stream is relatively uniform, the existing uniform compression method may still be a better choice because it can provide more consistent overall quality.

[0064] In summary, the traffic monitoring video compression device described in this embodiment achieves a higher compression rate by sacrificing the quality of the background video stream while maintaining the quality of the video stream of interest. This method may perform better than the existing uniform compression method in specific application scenarios, such as key area analysis of monitoring videos or lesion detection in medical images.

[0065] In summary, the final binary mask image is used to separate the video stream into two parts, including the video stream of interest and the background video stream, and different compression levels are used for compression through the H.264 / H.265 codec framework, thereby significantly reducing the size of the video stream file while ensuring the effective capture of key traffic monitoring information. Experimental results show that the traffic monitoring video compression device described in this embodiment is superior to the existing uniform compression method in terms of compression efficiency and video quality.

[0066] The image target detection method, video stream compression method and device proposed in the present invention are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. Improved YOLOv8 method for detecting image targets. The image is sequentially input into the YOLOv8 backbone network, the YOLOv8 neck network and the YOLOv8 head network for image target detection. The method is characterized in that: Specifically: The image is input into the CSC module in the backbone network of YOLOv8 and the CSC module in the neck network of YOLOv8 for feature extraction and output feature maps; In the CSC module, the image is subjected to feature extraction by the CBS module to output a feature map. The feature map is divided into two branches by Split. One branch feature map is directly input into Concat, and the other branch feature map is subjected to feature fusion by multiple SC-Block modules to output the fused feature map. After the feature maps of the two branches are Concat, they are passed through the CBS module again to output the feature map. In the SC-Block module, another branch feature map is subjected to feature fusion through CAA and star operation to output a fused feature map.

2. The improved YOLOv8 method for image target detection according to claim 1, characterized in that: The feature map of the other branch is subjected to feature fusion through CAA and star operation, specifically: ; in, The feature map of another branch is subjected to feature fusion through CAA and star operation. is the adaptive weight coefficient, is a position extracted from the image The local characteristics of For a certain location The corresponding global context information.

3. The improved YOLOv8 method for image target detection according to claim 1, characterized in that: The feature map of the other branch is subjected to feature fusion through CAA and star operation to output the fused feature map, specifically: The feature map of the other branch is extracted by CAA to output a feature map. The feature map is divided into the first branch and the second branch through DW-Conv. The feature maps of the first branch and the second branch are respectively input into the FC layer for operation to generate feature maps again. The star operation merges the feature maps of the first branch and the second branch to generate an updated feature map. The updated feature map is sequentially input into the FC layer and DW-Conv to generate a fused feature map.

4. The improved YOLOv8 method for image target detection according to claim 3, characterized in that: The updated feature map is specifically: ; in, is the updated feature map, is the feature map, is the adaptive weight, Is a logical operation symbol.

5. A video stream compression method, wherein the method is implemented by using the improved YOLOv8 image target detection method described in any one of claims 1 to 4, characterized in that: The following steps are involved: Step S1, splitting a video stream into multiple images; Step S2, performing target detection on a region of interest of an image based on the improved YOLOv8 to form an image with the region of interest; Step S3, generating a preliminary binary mask image from an image, and sequentially performing erosion, dilation and Gaussian blur operations on the preliminary binary mask image to generate a final binary mask image; Step S4, the final binary mask image extracts an image of interest and a background image from an image having a region of interest; Step S5, repeating the operations from step S2 to step S4 until the image of interest and the background image are extracted from the multiple images, then executing the operation of step S6; Step S6, a plurality of images of interest are combined to form an interested video stream, and a plurality of background images are combined to form a background video stream; Step S7: The video stream of interest and the background video stream are respectively combined and compressed using different compression levels, and a compressed video stream is obtained based on the combined and compressed video stream of interest and background video stream.

6. The video stream compression method according to claim 5, characterized in that: In the step S3, the operation of corroding the preliminary binary mask image is specifically as follows: ; in, To perform the erosion operation on the preliminary binary mask image, For the images, The structural element for the initial binary mask image when performing the erosion operation. is the structural element relative to the pixel The horizontal offset of is the structural element relative to the pixel The vertical offset of is the pixel coordinate of the image, is the preliminary binary mask map; The operation of dilating the preliminary binary mask image is specifically as follows: ; in, To dilate the initial binary mask image, The structural element used to dilate the initial binary mask image.

7. The video stream compression method according to claim 5, characterized in that: In the step S3, the final binary mask image is specifically: ; in, is the final binary mask image, is the pixel coordinate of the image, is the structural element relative to the pixel The horizontal offset of is the structural element relative to the pixel The vertical offset of is the kernel function of Gaussian blur, is the standard deviation of the kernel function of Gaussian blur, It is an operation to dilate the preliminary binary mask image.

8. The video stream compression method according to claim 5, characterized in that: In the step S6, the video stream of interest is specifically: ; in, For the video stream of interest, is the final binary mask image, For the images, is the pixel coordinate of the image, is a sequence of images with a region of interest; The background video stream is specifically: ; in, For background video stream, It is the inverse mask of the final binary mask image.

9. Traffic monitoring video compression device, characterized in that: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the video stream compression method as described in any one of claims 5 to 8.

Citation Information

Patent Citations

  • Video coding method and device, terminal and memory medium

    CN108810538A

  • Monitoring video secondary compression method based on connected domain analysis

    CN110719474A

  • Method and system for detecting bird species caused by mistaken collision with power transmission line based on deep learning

    CN118247554A

Cited By

  • Image processing method and device, equipment, medium and program product

    CN122053842A