Intelligent roi detection method and system based on ultra-high-definition 8k video

Abstract

The invention provides an intelligent roi detection method and system based on an ultra-high-definition 8K video, and relates to the technical field of image processing, and the method comprises the steps: obtaining a current to-be-processed frame from a picture frame sequence of the ultra-high-definition 8K video; using a background subtraction method to extract foreground areas from the current to-be-processed frame and front and back frames; calculating an optical flow between the foreground region in the current to-be-processed frame and the foreground regions in the front and back frames, and obtaining a motion vector of each foreground region; performing roi matching on a foreground region in the current to-be-processed frame and a tracking result of the preorder tracking picture frame based on the motion vector in combination with feature matching and similarity evaluation to obtain a roi matching result; and when the roi matching result determines that the current to-be-processed frame is a tracking picture frame, tracking is performed by using the to-be-tracked roi and the motion vector thereof, the roi of the current to-be-processed frame is determined, and the state is updated, so that accurate tracking of the moving target in the ultra-high-definition 8K video and extraction of the roi are realized.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an intelligent ROI detection method and system based on ultra-high-definition 8K video. Background Art

[0002] With the rapid development of ultra-high-definition video technology, 8K video resolution has become the top level of current video technology, providing users with an unprecedented visual experience. However, the massive data of ultra-high-definition 8K videos also brings unprecedented challenges to video processing and analysis. In application scenarios such as video surveillance, motion analysis, and target tracking, how to quickly and accurately extract regions of interest (ROIs) from ultra-high-definition 8K videos has become an urgent problem to be solved.

[0003] Traditional video ROI detection methods are mostly based on low-resolution videos. In the ultra-high-definition 8K video environment, these methods face problems such as large computational complexity, slow processing speed, and insufficient accuracy. In addition, the foreground and background in ultra-high-definition 8K videos are often more complex and changeable, and traditional background subtraction, motion detection and other methods are difficult to effectively deal with.

[0004] Therefore, it is necessary to provide an intelligent ROI detection method and system based on ultra-high-definition 8K video to solve the above technical problems. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides an intelligent ROI detection method and system based on ultra-high-definition 8K video, which realizes accurate tracking of moving targets and extraction of ROI in ultra-high-definition 8K video by combining background subtraction, optical flow calculation, feature matching and similarity evaluation technologies. The present invention provides an intelligent ROI detection method based on ultra-high-definition 8K video, and the detection method comprises the following steps: Get the current frame to be processed from the picture frame sequence of the ultra-high-definition 8K video; Extracting a foreground area from the current frame to be processed and its previous and next frames using a background subtraction method; Calculate the optical flow between the foreground area in the current frame to be processed and the foreground areas in the previous and next frames to obtain the motion vector of each foreground area; Based on the motion vector, and in combination with feature matching and similarity evaluation, ROI matching is performed on the foreground area in the current frame to be processed and the tracking result of the previous tracking picture frame to obtain a ROI matching result, wherein the previous tracking picture frame represents a set of frames that have been processed before the current frame to be processed and whose ROIs have been tracked; When the roi matching result determines that the current frame to be processed is a tracking picture frame, the roi to be tracked and its motion vector are used for tracking to determine the roi of the current frame to be processed and update the state.

[0006] Preferably, the obtaining the current frame to be processed from the picture frame sequence of the ultra-high-definition 8K video includes: Decode ultra-high-definition 8K video streams to obtain continuous image frame sequences; In the picture frame sequence, selecting and extracting the current frame to be processed according to a preset frame interval; The current frame to be processed is preprocessed, wherein the preprocessing includes color correction, noise removal and resolution adjustment.

[0007] Preferably, the extracting the foreground area from the current frame to be processed and its previous and next frames using a background subtraction method comprises: Constructing a background model, wherein the background model is obtained based on historical frame data in a picture frame sequence of the ultra-high-definition 8K video; The current frame to be processed and its previous and next frames are respectively compared with the background model, and the difference measurement between each pixel value and the pixel value at the corresponding position in the background model is calculated. Based on a preset threshold, the area with a pixel value exceeding the threshold value relative to the background model is identified and extracted as the foreground area.

[0008] Preferably, the step of calculating the optical flow between the foreground area in the current frame to be processed and the foreground areas in the previous and next frames to obtain the motion vector of each foreground area includes: Perform boundary segmentation on the foreground area in the current frame to be processed and its previous and next frames; The optical flow algorithm is used to calculate the optical flow field between each foreground area in the current frame to be processed after boundary segmentation and its corresponding foreground area in the previous and next frames; A motion vector of each foreground region is extracted from the optical flow field, wherein the motion vector represents a displacement direction and speed of the foreground region from a previous frame to a current frame or from a current frame to a next frame.

[0009] Preferably, the optical flow algorithm is the Horn-Schunck algorithm.

[0010] Preferably, the ROI matching is performed on the foreground area in the current frame to be processed and the tracking result of the previous tracking picture frame based on the motion vector and in combination with feature matching and similarity evaluation to obtain the ROI matching result, including: Extracting feature descriptors of each foreground area in the current frame to be processed and feature descriptors of the roi in the tracking result of the previous tracking picture frame; Calculate the feature matching score between the feature descriptor of the foreground area in the current frame to be processed and the feature descriptor of the roi in the tracking result of the previous tracking picture frame; The matching relationship is determined according to the feature matching score, a preset similarity threshold, and a motion vector of the foreground area.

[0011] Preferably, the determination of the matching relationship includes: If the feature matching score between the foreground area in the current frame to be processed and any ROI in the previous tracking picture frame is higher than the similarity threshold, and the motion vectors are consistent, the match is considered successful, and the position and state of the ROI are updated; If the feature matching scores of the foreground area in the current frame to be processed and each ROI in the previous tracking picture frame are lower than the similarity threshold, or the motion vectors are inconsistent, then the foreground area is added to the ROI to be tracked; If the feature matching scores of the roi of the previous tracking picture frame and the foreground areas in the current frame to be processed are both lower than the similarity threshold, or the motion vectors are inconsistent, the roi is deleted from the roi to be tracked.

[0012] Preferably, when the ROI matching result determines that the current frame to be processed is a tracking picture frame, tracking is performed using the ROI to be tracked and its motion vector, determining the ROI of the current frame to be processed and updating the state, including: Using the existing ROI to be tracked and its corresponding motion vector, a Kalman filter tracking algorithm is used to track the foreground area in the current frame to be processed; According to the tracking result, at least one ROI is determined as the ROI of the current frame to be processed, and the ROI state is updated, wherein the ROI state includes position, size and motion direction.

[0013] The present invention also provides an intelligent ROI detection system based on ultra-high-definition 8K video, which is used to execute the intelligent ROI detection method based on ultra-high-definition 8K video, and the detection system includes: The frame acquisition module is used to obtain the current frame to be processed from the picture frame sequence of the ultra-high-definition 8K video; A foreground extraction module, used for extracting a foreground area from the current frame to be processed and its previous and next frames using a background subtraction method; An optical flow calculation module is used to calculate the optical flow between the foreground area in the current frame to be processed and the foreground areas in the previous and next frames to obtain the motion vector of each foreground area; A roi matching module is used to perform roi matching on the foreground area in the current frame to be processed and the tracking result of the previous tracking picture frame based on the motion vector and in combination with feature matching and similarity evaluation to obtain a roi matching result, wherein the previous tracking picture frame represents a set of frames that have been processed before the current frame to be processed and whose roi has been tracked; The tracking and updating module is used to track the ROI to be tracked and its motion vector when the ROI matching result determines that the current frame to be processed is a tracking picture frame, determine the ROI of the current frame to be processed and update the state.

[0014] Compared with the related art, the intelligent ROI detection method and system based on ultra-high-definition 8K video provided by the present invention have the following beneficial effects: The present invention first obtains a continuous sequence of picture frames by decoding an ultra-high-definition 8K video stream, and selects and extracts the current frame to be processed according to a preset frame interval, and then uses the background subtraction method to extract the foreground area from the current frame to be processed and its previous and subsequent frames. Next, the optical flow algorithm is used to calculate the motion vector of the foreground area. On this basis, combined with feature matching and similarity evaluation technology, the foreground area in the current frame to be processed is matched with the tracking result of the previous tracking picture frame by ROI. Finally, according to the ROI matching result, the Kalman filter tracking algorithm is used to track the foreground area in the current frame to be processed, determine the ROI and update the status. Through the method of the present invention, accurate tracking of moving targets in ultra-high-definition 8K videos and extraction of ROI can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A flow chart of an intelligent ROI detection method based on ultra-high-definition 8K video provided by the present invention; Figure 2 A schematic diagram of the module structure of an intelligent ROI detection system based on ultra-high-definition 8K video provided by the present invention. DETAILED DESCRIPTION

[0016] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings. In addition, the embodiments of the present invention and the features in the embodiments may be combined with each other without conflict.

[0017] It should also be noted that, for ease of description, only the parts related to the present invention, but not all of the contents, are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the operations (or steps) as sequential processes, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0018] Embodiment 1 The present invention provides an intelligent ROI detection method based on ultra-high-definition 8K video. Figure 1 As shown, the detection method comprises the following steps: S1: Obtain the current frame to be processed from the picture frame sequence of the ultra-high-definition 8K video.

[0019] Specifically, step S1 includes the following steps: S11: Decode the ultra-high-definition 8K video stream to obtain a continuous sequence of picture frames.

[0020] In digital video processing, videos are usually stored in a compressed format to reduce storage space and transmission bandwidth requirements. However, before video analysis, these compressed video streams need to be decoded into original picture frame sequences. For 8K video, due to its extremely high resolution (7680x4320 pixels), the decoding process requires not only high efficiency but also image quality without loss. In this step, the decoder reads the encoding information in the video file, such as advanced encoding formats such as H.265 / HEVC, and then restores it to the original image data through the decoding algorithm.

[0021] The beneficial effect of this process is that it can accurately and losslessly restore high-quality image frames, providing a solid foundation for subsequent image processing and analysis.

[0022] S12: In the picture frame sequence, a current frame to be processed is selected and extracted according to a preset frame interval.

[0023] In this embodiment, since the high resolution of 8K video brings a huge amount of data, directly processing each frame may result in a huge waste of computing resources. Therefore, representative frames can be selected for processing by setting a certain frame interval, which can reduce the computing burden and maintain the consistency and representativeness of the video content. For example, you can choose to extract one frame every N frames as a frame to be processed, where N can be flexibly set according to the content change rate of the video and the processing requirements. This process helps to improve processing efficiency while ensuring that key information is not lost, thereby optimizing the performance of the entire system.

[0024] S13: Preprocessing the current frame to be processed, wherein the preprocessing includes color correction, noise removal and resolution adjustment.

[0025] In this embodiment, color correction is to correct possible color deviations to make the image colors more realistic and natural; noise removal is to filter various noises that may be introduced during the image acquisition process to improve the clarity of the image; and resolution adjustment is to enlarge or reduce the image when necessary to meet different processing requirements.

[0026] These preprocessing steps are crucial to improving the accuracy of ROI (region of interest) detection, because they can improve image quality and make subsequent background subtraction, optical flow calculation and other operations more effective. Through such preprocessing, the effect of the final ROI detection can be significantly improved, ensuring that the target can be accurately identified and tracked even in complex video environments.

[0027] S2: Use background subtraction method to extract the foreground area from the current frame to be processed and its previous and next frames.

[0028] Specifically, step S2 includes the following steps: S21: Build a background model, where the background model is obtained based on historical frame data in a picture frame sequence of an ultra-high-definition 8K video.

[0029] In this embodiment, the background model is constructed to distinguish between static background and dynamic foreground objects in the video. In practical applications, the background model is usually established at the beginning of the video by statistical analysis of a series of continuous frames. These initial frames should contain static scenes as much as possible to accurately capture the characteristics of the background.

[0030] The process of building the background model includes calculating the Gaussian mixture model statistical model of the average value of each pixel. In the context of 8K video applications, due to the extremely high image resolution, special attention should be paid to computational efficiency and memory usage when building the background model to avoid excessive resource consumption. The beneficial effect of this step is that it can provide a stable and reliable reference standard for subsequent foreground extraction, thereby improving the accuracy and robustness of ROI detection.

[0031] More specifically, the background model construction process includes: Select a series of consecutive frames at the beginning of the video that contain a static background as much as possible to ensure the accuracy of the background model.

[0032] For each pixel, calculate its pixel value in multiple historical frames. The distribution of each pixel can be modeled using a Gaussian mixture model (GMM), where GMM is a probabilistic model used to represent the mixed distribution of data points. In the background model, the pixel value of each pixel can be represented by one or more Gaussian distributions.

[0033] Through maximum likelihood estimation (MLE), the parameters of each Gaussian distribution, including mean, variance and weight, are estimated. These parameters reflect the statistical characteristics of pixels in different frames.

[0034] As the video plays, the background model needs to be constantly updated to adapt to changes in the environment. This can be achieved through online learning methods, for example, gradually adjusting the parameters of the Gaussian distribution to reflect the latest background information. S22: The current frame to be processed and its previous and next frames are compared with the background model respectively, by calculating the difference measure between each pixel value and the pixel value at the corresponding position in the background model, and based on a preset threshold, identifying and extracting the area whose pixel value exceeds the threshold with respect to the background model as the foreground area.

[0035] In this embodiment, this step is achieved by comparing the difference between the current frame and the background model pixel by pixel. If the difference of a certain pixel exceeds a preset threshold, the pixel is considered to belong to the foreground area. This method can effectively separate moving objects.

[0036] For 8K videos, since their high resolution may result in a large number of foreground pixels, the impact of noise needs to be considered when setting the threshold to avoid false detection or missed detection due to improper threshold setting.

[0037] More specifically, first, the difference metric between each pixel in the current frame to be processed and its previous and next frames and the pixel at the corresponding position in the background model is calculated. In the present application, the difference metric adopts the Euclidean distance.

[0038] For each pixel, the Euclidean distance between the current frame and the background model is calculated.

[0039] Next, the calculated difference measure is compared with a preset threshold to determine whether the pixel belongs to the foreground area. If it is greater than the threshold, the pixel is considered to belong to the foreground area; otherwise, the pixel is considered to belong to the background area.

[0040] The setting of the threshold can be adjusted according to the actual application scenario and video content. Usually, a suitable threshold can be determined through experiments.

[0041] After threshold processing, a binary image can be obtained, in which the pixel value of the foreground area is 1 and the pixel value of the background area is 0. This binary image is the mask of the foreground area.

[0042] In order to further refine the foreground area, the binary image can be analyzed for connected regions to extract the connected foreground area. This step can remove some small noise points and improve the accuracy of the foreground area.

[0043] Finally, the extracted foreground area is superimposed with the original image to generate an image with foreground marks. This step can be used for subsequent ROI detection and tracking S3: Calculate the optical flow between the foreground area in the current frame to be processed and the foreground areas in the previous and next frames to obtain the motion vector of each foreground area.

[0044] Specifically, step S3 includes the following steps: S31: performing boundary segmentation on the foreground area in the current frame to be processed and the frames before and after it.

[0045] After the foreground area is extracted, in order to calculate the optical flow more accurately, it is necessary to perform boundary segmentation on these foreground areas. The purpose of boundary segmentation is to clearly identify the edge contours of the foreground area so that the subsequent optical flow calculation can more accurately capture the motion information of the object. The specific implementation process is as follows: In this embodiment, boundary segmentation can be achieved by the Canny edge detection algorithm, which can detect intensity changes in an image to find the boundaries of an object. For each foreground area, an edge detection algorithm is first applied to generate a binary edge image. In this binary image, the boundaries of the foreground area are marked as white pixels, while the other parts are black pixels. In this way, the outline of the foreground area is clearly identified.

[0046] The beneficial effect of boundary segmentation is that it can significantly reduce the complexity of optical flow calculation, because the optical flow calculation only needs to be performed on the boundary pixels instead of the entire foreground area. In addition, boundary segmentation also helps to improve the accuracy of optical flow calculation, because boundary pixels usually contain more motion information.

[0047] S32: using an optical flow algorithm to calculate the optical flow field between each foreground area in the current frame to be processed after boundary segmentation and its corresponding foreground area in the previous and next frames.

[0048] In this embodiment, optical flow refers to the displacement of pixels in an image between adjacent frames, which can be used to describe the movement of an object. In this application, the Horn-Schunck algorithm is used, which is a classic global optical flow algorithm.

[0049] The specific implementation process is as follows: First, initialize the optical flow field and , respectively represent the displacement of each pixel in the horizontal direction and vertical direction. Initially, the displacement of all pixels can be set to 0.

[0050] For each pixel , calculate the current frame With the previous frame and the next frame Brightness changes between .

[0051] Calculate the horizontal and vertical gradients of each pixel and .

[0052] Use the iterative formula of the Horn-Schunck algorithm to gradually update the optical flow field and .

[0053] in, and Respectively expressed in At the iteration, the pixel Optical flow components in horizontal and vertical directions; and Respectively expressed in At the iteration, the pixel The optical flow components in the horizontal and vertical directions respectively represent the image at the pixel point Horizontal and vertical gradients at It is a smoothing parameter used to control the smoothness of the optical flow field, and the iterative process needs to be performed multiple times until the optical flow field converges.

[0054] S33: extracting a motion vector of each foreground area from the optical flow field, wherein the motion vector represents a displacement direction and speed of the foreground area from a previous frame to a current frame or from a current frame to a next frame.

[0055] In this embodiment, the step is to extract the motion vector of each foreground area from the optical flow field, wherein the motion vector represents the displacement direction and speed of the foreground area from the previous frame to the current frame or from the current frame to the next frame. After the optical flow field calculation is completed, the motion vector of each foreground area needs to be extracted from the optical flow field for subsequent ROI matching and tracking.

[0056] In this embodiment, the specific implementation process is as follows: First, based on the boundary segmentation result obtained in step S31, the boundary pixels of each foreground area are located.

[0057] For each boundary pixel of the foreground area, its corresponding horizontal displacement and vertical displacement are extracted from the optical flow field, and these displacement values ​​together constitute the motion vector of the foreground area.

[0058] In order to reduce the influence of noise, the average motion vector of each foreground area can be calculated by averaging the displacement vectors of all boundary pixels in the foreground area.

[0059] The final average motion vector represents the overall motion direction and speed of the foreground area.

[0060] Motion vectors can provide accurate motion information of the foreground area, which is critical for subsequent ROI matching and tracking. Through these motion vectors, the system can more accurately determine the motion state of the foreground area, thereby improving the accuracy of ROI detection and tracking.

[0061] S4: Based on the motion vector, and in combination with feature matching and similarity evaluation, the foreground area in the current frame to be processed is matched with the tracking result of the previous tracking picture frame to obtain the ROI matching result, wherein the previous tracking picture frame represents the set of frames that have been processed before the current frame to be processed and whose ROI has been tracked.

[0062] Specifically, step S4 includes the following steps: S41: extracting the feature descriptor of each foreground area in the current frame to be processed and the feature descriptor of the roi in the tracking result of the previous tracking picture frame.

[0063] In image processing, feature descriptors are vectors used to describe the features of a specific area in an image. These features can be color, texture, shape, etc. In this step, feature descriptors include but are not limited to SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), ORB (Oriented FAST and Rotated BRIEF), etc.

[0064] The specific implementation process is as follows: First, feature point detection algorithms are used to detect feature points in the foreground area of ​​the current frame to be processed. These feature points are usually points with unique properties in the image, such as corner points, edge points, etc.

[0065] For each detected feature point, extract the feature descriptor of the local area around it. For example, use the SIFT algorithm to extract the gradient direction histogram of the 16x16 pixel area around the feature point to form a 128-dimensional feature vector.

[0066] Similarly, feature point detection and feature descriptor extraction are performed on the ROI that has been tracked in the previous tracking picture frame to generate a corresponding feature descriptor set.

[0067] S42: Calculate the feature matching score between the feature descriptor of the foreground area in the current frame to be processed and the feature descriptor of the ROI in the tracking result of the previous tracking picture frame.

[0068] In this embodiment, the feature matching score is used to measure the similarity between two feature descriptors, wherein matching methods include but are not limited to nearest neighbor matching, ratio test, and the like.

[0069] The specific implementation process is as follows: Use the feature matching algorithm to match the feature descriptors in the current frame to be processed with the feature descriptors in the previous tracking picture frame. During the matching process, calculate the distance (Euclidean distance) between each pair of feature descriptors and record the matching results.

[0070] In order to improve the reliability of matching, a ratio test method is used. For each feature point, the two closest matching points in the previous frame are found. If the distance ratio between the closest matching point and the second closest matching point is less than a certain threshold (such as 0.7), the match is considered reliable.

[0071] According to the matching result, the matching score between the foreground area in the current frame to be processed and the ROI in the previous tracking picture frame is calculated. The matching score can be but is not limited to the number of matching feature points and the sum of the distances of the matching feature points.

[0072] S43: Determine a matching relationship according to the feature matching score, a preset similarity threshold, and a motion vector of the foreground area.

[0073] The determination of the matching relationship includes the following situations: If the feature matching score between the foreground area in the current frame to be processed and any ROI in the previous tracking picture frame is higher than the similarity threshold, and the motion vector is consistent, the match is considered successful and the position and state of the ROI are updated.

[0074] If the feature matching scores of the foreground area in the current frame to be processed and each ROI in the previous tracking picture frame are lower than the similarity threshold, or the motion vectors are inconsistent, the foreground area is added to the ROI to be tracked.

[0075] If the feature matching scores of the roi of the previous tracking picture frame and the foreground areas in the current frame to be processed are both lower than the similarity threshold, or the motion vectors are inconsistent, the roi is deleted from the roi to be tracked.

[0076] This step can accurately determine the matching relationship of the foreground area by comprehensively considering the feature matching score and motion vector, thereby improving the accuracy and robustness of ROI tracking.

[0077] S5: When the roi matching result determines that the current frame to be processed is a tracking picture frame, the roi to be tracked and its motion vector are used for tracking to determine the roi of the current frame to be processed and update the state.

[0078] Specifically, step S5 includes the following steps: S51: using the existing ROI to be tracked and its corresponding motion vector, a Kalman filter tracking algorithm is used to track the foreground area in the current frame to be processed.

[0079] In this embodiment, Kalman filtering is a recursive prediction-correction algorithm, which is widely used in the field of target tracking, especially in the processing of noise and uncertainty. In this embodiment, the Kalman filtering algorithm is used to predict and correct the position and state of the ROI to achieve continuous tracking of the foreground area.

[0080] The basic principle of Kalman filtering: The Kalman filter algorithm works in two main steps: prediction and update. The prediction step estimates the current state based on the state at the previous moment, and the update step uses the observed data at the current moment to correct the predicted state. Specifically, the Kalman filter maintains a state vector and a covariance matrix, which represent the state of the target and the uncertainty of the state estimate, respectively.

[0081] The prediction steps include: State prediction: Predict the state at the current moment based on the state at the previous moment and the state transfer matrix.

[0082] Covariance prediction: Based on the covariance matrix of the previous moment and the process noise covariance matrix, predict the covariance matrix of the current moment.

[0083] The update steps include: Calculate Kalman gain: Calculate the Kalman gain based on the predicted covariance matrix and the observed noise covariance matrix.

[0084] State update: Update the current state based on the current observation data and Kalman gain.

[0085] Covariance update: Update the covariance matrix at the current moment according to the Kalman gain.

[0086] Through the Kalman filter algorithm, the position and state of the ROI can be effectively predicted and corrected, thereby achieving continuous tracking of the foreground area. The Kalman filter can handle noise and uncertainty, improving the robustness and accuracy of tracking. In addition, the recursive nature of the Kalman filter makes it suitable for real-time processing, and can quickly respond to the motion changes of the target, ensuring the real-time and stability of tracking.

[0087] S52: According to the tracking result, at least one ROI is determined as the ROI of the current frame to be processed, and the ROI state is updated, wherein the ROI state includes position, size and motion direction.

[0088] In this embodiment, according to the tracking result of Kalman filtering, the ROI that best matches the current foreground area is selected. This can be done by calculating the similarity between the foreground area and the candidate ROI (including but not limited to overlap rate, feature matching score, etc.) to select the optimal ROI.

[0089] After determining the optimal ROI, update its state information, including position, size and direction of movement. The position can be directly obtained from the result of Kalman filtering, while the size and direction of movement can be calculated from the bounding box and optical flow information of the foreground area.

[0090] Through this step, the tracking results of the Kalman filter can be converted into specific ROI information, ensuring that each foreground area has a clear identification and status description. This not only helps with subsequent ROI detection and tracking, but also provides basic data support for other advanced analysis tasks (such as behavior recognition, anomaly detection, etc.). In addition, the updated ROI state can be used for tracking initialization of the next frame, forming a closed-loop tracking system, improving the continuity and accuracy of the overall tracking.

[0091] Embodiment 2 The present invention also provides an intelligent ROI detection system based on ultra-high-definition 8K video, which is used to perform an intelligent ROI detection method based on ultra-high-definition 8K video, referring to Figure 2As shown, the detection system includes: The frame acquisition module 100 is used to obtain the current frame to be processed from the picture frame sequence of the ultra-high-definition 8K video.

[0092] The foreground extraction module 200 is used to extract the foreground area from the current frame to be processed and its previous and next frames using a background subtraction method.

[0093] The optical flow calculation module 300 is used to calculate the optical flow between the foreground area in the current frame to be processed and the foreground areas in the previous and next frames to obtain the motion vector of each foreground area.

[0094] The ROI matching module 400 is used to perform ROI matching on the foreground area in the current frame to be processed and the tracking result of the previous tracking picture frame based on the motion vector and in combination with feature matching and similarity evaluation to obtain the ROI matching result, wherein the previous tracking picture frame represents a set of frames that have been processed before the current frame to be processed and whose ROI has been tracked.

[0095] The tracking and updating module 500 is used to track the ROI to be tracked and its motion vector when the ROI matching result determines that the current frame to be processed is a tracking picture frame, determine the ROI of the current frame to be processed and update the state.

[0096] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0097] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0098] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

Claims

1. An intelligent ROI detection method based on ultra-high-definition 8K video, characterized in that: The detection method comprises the following steps: Get the current frame to be processed from the picture frame sequence of the ultra-high-definition 8K video; Extracting a foreground area from the current frame to be processed and its previous and next frames using a background subtraction method; Calculate the optical flow between the foreground area in the current frame to be processed and the foreground areas in the previous and next frames to obtain the motion vector of each foreground area; Based on the motion vector, and in combination with feature matching and similarity evaluation, ROI matching is performed on the foreground area in the current frame to be processed and the tracking result of the previous tracking picture frame to obtain a ROI matching result, wherein the previous tracking picture frame represents a set of frames that have been processed before the current frame to be processed and whose ROIs have been tracked; When the roi matching result determines that the current frame to be processed is a tracking picture frame, the roi to be tracked and its motion vector are used for tracking to determine the roi of the current frame to be processed and update the state.

2. According to claim 1, an intelligent ROI detection method based on ultra-high-definition 8K video is characterized in that: The step of obtaining a current frame to be processed from a picture frame sequence of an ultra-high-definition 8K video includes: Decode ultra-high-definition 8K video streams to obtain continuous image frame sequences; In the picture frame sequence, selecting and extracting the current frame to be processed according to a preset frame interval; The current frame to be processed is preprocessed, wherein the preprocessing includes color correction, noise removal and resolution adjustment.

3. According to claim 2, an intelligent ROI detection method based on ultra-high-definition 8K video is characterized in that: The extracting of the foreground area from the current frame to be processed and its previous and next frames by using the background subtraction method comprises: Constructing a background model, wherein the background model is obtained based on historical frame data in a picture frame sequence of the ultra-high-definition 8K video; The current frame to be processed and its previous and next frames are respectively compared with the background model, and the difference measurement between each pixel value and the pixel value at the corresponding position in the background model is calculated. Based on a preset threshold, the area with a pixel value exceeding the threshold value relative to the background model is identified and extracted as the foreground area.

4. According to claim 3, an intelligent ROI detection method based on ultra-high-definition 8K video is characterized in that: The step of calculating the optical flow between the foreground area in the current frame to be processed and the foreground areas in the previous and next frames to obtain the motion vector of each foreground area includes: Perform boundary segmentation on the foreground area in the current frame to be processed and its previous and next frames; The optical flow algorithm is used to calculate the optical flow field between each foreground area in the current frame to be processed after boundary segmentation and its corresponding foreground area in the previous and next frames; A motion vector of each foreground region is extracted from the optical flow field, wherein the motion vector represents a displacement direction and speed of the foreground region from a previous frame to a current frame or from a current frame to a next frame.

5. According to claim 4, an intelligent ROI detection method based on ultra-high-definition 8K video is characterized in that: The optical flow algorithm is the Horn-Schunck algorithm.

6. The intelligent ROI detection method based on ultra-high-definition 8K video according to claim 5 is characterized in that: The step of performing ROI matching on the foreground area in the current frame to be processed and the tracking result of the previous tracking picture frame based on the motion vector and combining feature matching and similarity evaluation to obtain the ROI matching result includes: Extracting feature descriptors of each foreground area in the current frame to be processed and feature descriptors of the roi in the tracking result of the previous tracking picture frame; Calculate the feature matching score between the feature descriptor of the foreground area in the current frame to be processed and the feature descriptor of the roi in the tracking result of the previous tracking picture frame; The matching relationship is determined according to the feature matching score, a preset similarity threshold, and a motion vector of the foreground area.

7. The intelligent ROI detection method based on ultra-high-definition 8K video according to claim 6 is characterized in that: The determination of the matching relationship includes: If the feature matching score between the foreground area in the current frame to be processed and any ROI in the previous tracking picture frame is higher than the similarity threshold, and the motion vectors are consistent, the match is considered successful, and the position and state of the ROI are updated; If the feature matching scores of the foreground area in the current frame to be processed and each ROI in the previous tracking picture frame are lower than the similarity threshold, or the motion vectors are inconsistent, then the foreground area is added to the ROI to be tracked; If the feature matching scores of the roi of the previous tracking picture frame and the foreground areas in the current frame to be processed are both lower than the similarity threshold, or the motion vectors are inconsistent, the roi is deleted from the roi to be tracked.

8. The intelligent ROI detection method based on ultra-high-definition 8K video according to claim 7 is characterized in that: When the ROI matching result determines that the current frame to be processed is a tracking picture frame, tracking is performed using the ROI to be tracked and its motion vector, determining the ROI of the current frame to be processed and updating the state, including: Using the existing ROI to be tracked and its corresponding motion vector, a Kalman filter tracking algorithm is used to track the foreground area in the current frame to be processed; According to the tracking result, at least one ROI is determined as the ROI of the current frame to be processed, and the ROI state is updated, wherein the ROI state includes position, size and motion direction.

9. An intelligent ROI detection system based on ultra-high-definition 8K video, used to execute an intelligent ROI detection method based on ultra-high-definition 8K video according to any one of claims 1 to 8, characterized in that: The detection system comprises: The frame acquisition module is used to obtain the current frame to be processed from the picture frame sequence of the ultra-high-definition 8K video; A foreground extraction module, used for extracting a foreground area from the current frame to be processed and its previous and next frames using a background subtraction method; An optical flow calculation module is used to calculate the optical flow between the foreground area in the current frame to be processed and the foreground areas in the previous and next frames to obtain the motion vector of each foreground area; A roi matching module is used to perform roi matching on the foreground area in the current frame to be processed and the tracking result of the previous tracking picture frame based on the motion vector and in combination with feature matching and similarity evaluation to obtain a roi matching result, wherein the previous tracking picture frame represents a set of frames that have been processed before the current frame to be processed and whose roi has been tracked; The tracking and updating module is used to track the ROI to be tracked and its motion vector when the ROI matching result determines that the current frame to be processed is a tracking picture frame, determine the ROI of the current frame to be processed and update the state.

Images