Ground area detection method and device in video monitoring scene, equipment and medium

Abstract

The invention relates to the technical field of video monitoring, solves the problem that ground area detection cannot be accurately carried out in the prior art, and provides a ground area detection method and device in a video monitoring scene, equipment and a medium. The method comprises the following steps: inputting a real-time image into a pre-trained human body target detection model, monitoring the rotation of a camera by using a visual mileage calculation method, and if a human body target is not recognized in a current monitoring scene and a current monitoring view angle is not rotated, carrying out filtering processing and color space conversion on the real-time image to obtain a color space image; performing feature extraction on the color space image, and fusing the extracted local features and global features to obtain multi-scale features; and determining the position information of the ground area according to the multi-scale features. According to the invention, the accuracy of ground area detection is improved.

Description

[0001] This application is a divisional application of the invention patent application filed on December 15, 2023, with the invention name “Human fall detection method, device, equipment and medium combined with depth measurement” and application number 202311742651.3. Technical Field

[0002] The present invention relates to the field of video surveillance technology, and in particular to a ground area detection method, device, equipment and medium in a video surveillance scene. Background Art

[0003] In video surveillance, accurate detection of ground areas is of great significance for key safety applications such as human fall detection. Falls often occur in ground areas. Therefore, accurate identification of ground areas can lay the foundation for subsequent behavior recognition and anomaly detection. By positioning the ground area, the posture changes of the human body relative to the ground can be more effectively judged, especially in scenarios such as elderly care and public safety. Timely detection of falls and issuing alarms can significantly reduce the risk of accidental injuries. In addition, ground area detection can also reduce misjudgment and noise interference, improve the accuracy of fall detection, and enable the system to focus on target activities on the ground, ignore irrelevant environmental factors, and ensure the reliability and efficiency of detection.

[0004] Existing ground detection technologies often have problems with insufficient accuracy in complex monitoring scenarios, which are mainly reflected in the following aspects. First, many algorithms rely on single features, such as color or texture information, which can easily lead to misjudgment when there are changes in lighting, shadow interference, or diverse ground materials. Second, some technologies lack adaptability to changes in camera perspectives in dynamic scenes, making it difficult to stably identify ground areas at different angles. In addition, existing ground detection technology methods are prone to over-segmentation or missed detection when dealing with scenes with rich details or complex scenes, and cannot accurately capture the complete outline of the ground area. These problems limit the practicality of ground detection, especially in applications that require high-precision ground recognition, such as fall detection scenarios.

[0005] Therefore, how to accurately detect ground areas is an urgent problem to be solved. Summary of the invention

[0006] In view of this, the present invention provides a ground area detection method, device, equipment and medium in a video surveillance scenario, so as to solve the problem that the prior art cannot accurately perform ground area detection.

[0007] The technical solution adopted by the present invention is:

[0008] In a first aspect, the present invention provides a ground area detection method in a video surveillance scenario, characterized in that the method comprises:

[0009] The real-time image obtained by decomposing the real-time video stream in the video surveillance scene is input into the pre-trained human target detection model to identify whether a human target appears in the current surveillance scene;

[0010] The visual odometry calculation method is used to monitor the rotation of the camera and to identify whether the current monitoring view has rotated by comparing the posture changes between adjacent frames.

[0011] If no human target is identified in the current monitoring scene and the current monitoring angle of view is not rotated, the real-time image is filtered and the color space is converted to obtain a color space image;

[0012] Extracting features from the color space image, fusing the extracted local features with the global features, and obtaining multi-scale features;

[0013] According to the multi-scale features, the probability of each pixel being in the ground area is obtained, a preset probability threshold is obtained, and the probability map is converted into a binary segmentation map;

[0014] Morphological processing is performed on the binary segmentation map, and the pixel position information in the ground area is used as the ground area position information.

[0015] Preferably, if no human target is identified in the current monitoring scene and the current monitoring angle of view is not rotated, the real-time image is segmented to obtain the ground area position information, and the method further includes:

[0016] Inputting each of the real-time images into a pre-trained human body detection model, and outputting human body region position information;

[0017] Performing depth analysis on the human body region position information and the ground region position information to obtain an average ground region depth sequence and an average human body region depth sequence corresponding to multiple frames of time-series images;

[0018] The depth difference between the average depth sequence of the human body area and the average depth sequence of the ground area is calculated, and a human fall is detected based on the depth difference sequence.

[0019] Preferably, the performing depth analysis on the human body region position information and the ground region position information to obtain the ground region average depth sequence and the human body region average depth sequence corresponding to multiple frames of time-series images comprises:

[0020] Obtaining a training image with a deep label in a video surveillance scene, inputting the training image into a deep learning network, and obtaining a training model;

[0021] Input the real-time image into the training model for forward propagation to obtain depth values ​​corresponding to all pixels in the real-time image;

[0022] According to the human body area position information and the ground area position information, combined with the depth values ​​corresponding to all pixels in the real-time image, a human body area depth value sequence and a ground area depth value sequence corresponding to multiple frames of time-series images are obtained;

[0023] The ground area average depth sequence and the human area average depth sequence are obtained according to the human area depth value sequence and the ground area depth value sequence.

[0024] Preferably, the obtaining of a human region depth value sequence and a ground region depth value sequence corresponding to multiple frames of sequential images based on the human region position information and the ground region position information in combination with depth values ​​corresponding to all pixels in the real-time image comprises:

[0025] Outputting depth values ​​corresponding to pixel points in the human body region to the human body region depth value sequence according to the human body region position information;

[0026] According to the ground area position information, the depth values ​​corresponding to the pixel points in the ground area are output to the ground area depth value sequence.

[0027] Preferably, the obtaining of the ground area average depth sequence and the human area average depth sequence according to the human area depth value sequence and the ground area depth value sequence comprises:

[0028] Acquire a first pixel point sequence in the human body region and a second pixel point sequence in the ground region according to the human body region position information and the ground region position information;

[0029] Obtaining a first sliding window and a second sliding window according to the first pixel point sequence and the second pixel point sequence;

[0030] According to the first sliding window and the second sliding window, a first filtered depth value sequence and a second filtered depth value sequence are obtained;

[0031] According to the first filtered depth value sequence and the second filtered depth value sequence, sliding window processing is performed on the human body area depth value sequence and the ground area depth value sequence to obtain an average depth of the ground area and an average depth of the human body area.

[0032] Preferably, the calculating of the depth difference between the average depth sequence of the human body area and the average depth sequence of the ground area, and detecting a human fall according to the depth difference sequence comprises:

[0033] Get the preset depth difference threshold;

[0034] Comparing each depth difference in the depth difference sequence with the depth difference threshold in sequence to obtain a comparison result;

[0035] When the comparison results corresponding to the continuous multiple frames of images all show that the depth difference is less than the depth difference threshold, the result of the human body falling is that the human body falls on the ground.

[0036] Preferably, if no human target is identified in the current monitoring scene and the current monitoring angle of view is not rotated, filtering and color space conversion are performed on the real-time image to obtain the color space image, which includes:

[0037] Get the pre-trained human target detection model based on yolov6s architecture;

[0038] Inputting the real-time image into the human target detection model to identify whether a human target appears in the current monitoring scene;

[0039] The visual odometry calculation method is used to monitor the rotation of the camera and identify whether the current monitoring angle of view rotates.

[0040] In a second aspect, the present invention provides a ground area detection device in a video surveillance scenario, characterized in that the device comprises:

[0041] The human target detection module is used to input the real-time image obtained by decomposing the real-time video stream in the video surveillance scene into the pre-trained human target detection model to identify whether a human target appears in the current surveillance scene;

[0042] The view rotation monitoring module is used to monitor the camera rotation by using the visual mileage calculation method, and to identify whether the current monitoring view rotates by comparing the posture changes between adjacent frames;

[0043] A color space image acquisition module is used to filter and convert the real-time image into a color space to obtain a color space image if no human target is identified in the current monitoring scene and the current monitoring angle of view is not rotated;

[0044] A feature extraction module, used to extract features from the color space image, fuse the extracted local features with the global features, and obtain multi-scale features;

[0045] A binary segmentation map acquisition module is used to obtain the probability of each pixel point being in the ground area based on the multi-scale features, obtain a preset probability threshold, and convert the probability map into a binary segmentation map;

[0046] The ground area position information acquisition module is used to perform morphological processing on the binary segmentation map and use the pixel point position information in the ground area as the ground area position information.

[0047] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising: at least one processor, at least one memory, and computer program instructions stored in the memory, which, when executed by the processor, implement the method of the first aspect in the above-mentioned embodiment.

[0048] In a fourth aspect, an embodiment of the present invention further provides a storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method of the first aspect in the above-mentioned embodiment is implemented.

[0049] In summary, the beneficial effects of the present invention are as follows:

[0050] The present invention provides a ground area detection method, device, equipment and medium in a video surveillance scene. The method comprises: inputting a real-time image obtained by decomposing a real-time video stream in a video surveillance scene into a pre-trained human target detection model to identify whether a human target appears in the current surveillance scene; using a visual mileage calculation method to monitor the rotation of the camera, and identifying whether the current surveillance view angle rotates by comparing the posture changes between adjacent frames; if no human target is identified in the current surveillance scene and the current surveillance view angle does not rotate, filtering and color space conversion are performed on the real-time image to obtain a color space image; feature extraction is performed on the color space image, and the extracted local features are merged with the global features to obtain multi-scale features; based on the multi-scale features, the probability of each pixel point being in the ground area is obtained, a preset probability threshold is obtained, and the probability map is converted into a binary segmentation map; morphological processing is performed on the binary segmentation map, and the pixel point position information in the ground area is used as the ground area position information. The present invention effectively improves the accuracy of ground area detection by combining human target detection and visual mileage monitoring. When no human target is detected and the viewing angle of the monitoring camera remains stable, the real-time image is filtered and the color space is converted, thereby reducing noise interference and enhancing the available information of the image. Subsequently, multi-scale features are extracted by fusing local and global features to capture the complex details of the ground area. These multi-scale features make the probability estimation of pixel points in the ground area more accurate. Combined with morphological processing, the ground area can be clearly segmented to reduce misjudgment. It is particularly suitable for complex monitoring scenes, providing stable and efficient ground area recognition, and ensuring accurate positioning of key areas in monitoring scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the drawings required for use in the embodiment of the present invention will be briefly introduced below. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work, and these are all within the protection scope of the present invention.

[0052] Figure 1 This is a schematic diagram of the overall working process of the ground area detection method in the video surveillance scenario in Example 1 of the present invention;

[0053] Figure 2 Schematic diagram of the process of identifying a human target and camera rotation in Embodiment 1 of the present invention;

[0054] Figure 3 It is a schematic diagram of the process of performing image segmentation on the real-time image in Embodiment 1 of the present invention;

[0055] Figure 4 This is a schematic diagram of the process of performing in-depth analysis in Example 1 of the present invention;

[0056] Figure 5 This is a schematic diagram of a process for determining a depth value sequence in Embodiment 1 of the present invention;

[0057] Figure 6 It is a schematic diagram of a process for determining the average depth in Example 1 of the present invention;

[0058] Figure 7 A schematic diagram of a process for determining a human fall detection result in Embodiment 1 of the present invention;

[0059] Figure 8 This is a structural block diagram of a ground area detection device in a video surveillance scenario in Embodiment 2 of the present invention;

[0060] Fig. 9 It is a schematic diagram of the structure of the electronic device in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described in conjunction with the drawings in the embodiment of the present invention. It should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. In the description of the present invention, it should be understood that the orientation or position relationship indicated by the terms "center", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such a process, method, article or device. In the absence of further restrictions, the elements defined by the phrase "comprising..." do not exclude the existence of other identical elements in the process, method, article or device comprising the elements. If there is no conflict, the embodiments of the present invention and the various features in the embodiments can be combined with each other, all within the protection scope of the present invention.

[0062] Example 1

[0063] See also Figure 1 Embodiment 1 of the present invention discloses a ground area detection method in a video surveillance scenario, the method comprising:

[0064] S1: Acquire a real-time video stream in a video surveillance scenario, and decompose the real-time video stream into multiple frames of real-time images;

[0065] Specifically, a real-time video stream captured by a tilted monocular camera in a video surveillance scene is obtained, and the obtained video stream is decoded and converted into an image sequence. This step is performed using a common video codec (such as H.264, H.265); the decoded video stream is decomposed into multiple frames of images, each frame represents a moment of the video stream, and the image sequence is composed of these frames; a monocular camera refers to a camera with only one optical lens, through which image information of a plane view can be obtained. Commonly used monocular cameras include ordinary digital cameras, network cameras, industrial cameras, etc. On the one hand, when installing a monocular camera, if it is installed vertically, the field of view of the camera will be very limited. By installing it tilted, its field of view can be expanded and the monitoring effect can be improved; on the other hand, when installing a monocular camera, if it is installed vertically, the camera will be easily discovered by people. Installing it tilted can make the camera more concealed and improve safety.

[0066] In one embodiment, see Figure 2 , said S2 before includes:

[0067] S201: Get the pre-trained human target detection model based on yolov6s architecture;

[0068] S202: Inputting the real-time image into the human target detection model to identify whether a human target appears in the current monitoring scene;

[0069] S203: Using the visual odometry calculation method, the camera rotation is monitored to identify whether the current monitoring angle of view is rotated.

[0070] Specifically, a machine learning or deep learning algorithm, such as the yolov6s algorithm, is used to learn a large amount of labeled training data to establish a human detection model to detect whether there is a human body in the current video surveillance scene; the visual mileage calculation method is used to compare the posture changes between adjacent frames to determine whether the camera has rotated. When it is determined by the above method that no human body is detected in the current video surveillance scene and the camera stops rotating, it is considered that the preset start condition is met and the ground area in the image is segmented.

[0071] S2: If no human target is identified in the current monitoring scene and the current monitoring view angle is not rotated, the real-time image is segmented to obtain the ground area position information;

[0072] Specifically, a starting condition is pre-set, and the preset starting condition includes: no human body is detected in the current video surveillance scene and the camera stops rotating. Only when the preset starting condition is met, the ground area position information in the monocular image is segmented out. On the one hand, the interference of human body information in the surveillance scene on the segmented ground area is avoided; on the other hand, when the camera does not rotate, the surveillance scene does not change much, and a human fall will definitely cause a drastic change in the surveillance scene. Therefore, image segmentation is only performed when the camera stops rotating, avoiding unnecessary waste of work resources.

[0073] In one embodiment, see Figure 3 , said S2 comprises:

[0074] S21: If no human target is identified in the current monitoring scene and the current monitoring angle of view is not rotated, filtering and color space conversion are performed on the real-time image to obtain a color space image, wherein the real-time image is a monocular image;

[0075] Specifically, a monocular image is obtained, which is a color image. The input color image is preprocessed. The noise that may exist in the image will affect the segmentation result, so Gaussian filtering is used to eliminate the noise in the image; the preprocessed color image is converted into HSV color space. The reason for choosing HSV color space is that HSV color space can decompose color information into three components: hue, saturation and brightness. The hue component corresponds to the human eye's perception of color, so it can more accurately distinguish elements of different colors, so as to better distinguish ground and non-ground elements.

[0076] S22: extracting features from the color space image, fusing the extracted local features with the global features, and obtaining multi-scale features;

[0077] Specifically, the converted color space image is segmented at multiple scales to obtain multiple sub-regions. For each sub-region, the LBP operator is used to extract local features. The LBP operator is an algorithm for texture feature extraction. It can compare the adjacent pixels around a pixel with the pixel to obtain a binary code to describe the texture features around the pixel. For the entire image, the pre-trained ResNet model is used to extract global features. The ResNet model is a deep convolutional neural network that can perform high-level feature extraction and classification on the image. Through the MLFN feature fusion network, the extracted local features and global features are fused to construct a multi-scale feature representation. Through the application of multi-scale segmentation and the LBP operator, sub-regions with local texture features can be extracted, which enhances the distinguishability of features. At the same time, the use of the ResNet model to extract global features can capture the contextual information of the entire image and improve the accuracy of classification. Finally, by fusing local features and global features, the feature expression ability and classification performance can be further improved.

[0078] S23: Obtaining the probability of each pixel point being in the ground area according to the multi-scale features, and obtaining a probability map;

[0079] S24: Obtain a preset probability threshold, and convert the probability map into a binary segmentation map;

[0080] Specifically, the above local features and global features are spliced ​​together to form a feature vector for each pixel. These feature vectors are used to train a classifier, which divides each pixel into two categories: ground elements and non-ground elements. After the training is completed, the classifier is used to classify each pixel to obtain a probability map. This probability map represents the probability that each pixel belongs to a ground element, and the value range is between 0 and 1. The Otsu method is used to determine a suitable probability threshold. The Otsu method is an adaptive threshold selection method based on histogram, which can automatically determine an optimal threshold to divide the image into two parts. The Otsu method regards the grayscale value of the image as a probability distribution, and determines an optimal grayscale threshold by minimizing the weighted sum of the intra-class variance and the inter-class variance. In our method, we regard the probability map as a grayscale image, determine an optimal probability threshold by the Otsu method, and convert the probability map into a binary segmentation map, which divides each pixel in the image into two categories: ground elements and non-ground elements. The probability map method can be used to calculate the probability value of a pixel belonging to a ground element based on its color and texture characteristics, thereby avoiding the problem of inaccurate segmentation caused by changes in pixel color and texture characteristics in traditional segmentation methods. At the same time, the Otsu method can be used to adaptively determine the probability threshold, making the segmentation result more accurate.

[0081] S25: Perform morphological processing on the binary segmentation map, and use the pixel point position information in the ground area as the ground area position information.

[0082] Specifically, morphological operations (opening, closing, etc.) are performed on the binary segmentation map. The opening operation eliminates small noises and small connections, thereby obtaining a more accurate segmentation result. The closing operation can fill the holes and cracks inside the object, making the object more complete and connected. Such operations can improve the accuracy of subsequent processing. The pixels of the ground area are saved in coordinate order to obtain the ground area position information P (p1, p2, p3..., Pn), where p1, p2, p3..., Pn represent the coordinate information corresponding to the first pixel point to the coordinate information corresponding to the nth pixel point in the camera coordinate system.

[0083] S3: inputting each of the real-time images into a pre-trained human body detection model, and outputting human body region position information;

[0084] S4: performing depth analysis on the human body region position information and the ground region position information to obtain an average depth sequence of the ground region and an average depth sequence of the human body region corresponding to multiple frames of time-series images;

[0085] Specifically, the position of the human body is detected through target detection algorithms such as yolov6s, and the pixel set of the human body area is obtained. Based on the pixel set of the human body area, the position information of the human body area is obtained. Combined with the above-mentioned ground area position information, we can convert the image captured by the monocular camera into a three-dimensional scene. Then, the average depth of the ground area and the average depth of the human body area are calculated; because a fallen human body usually has abnormal depth changes, such as a sudden decrease or increase in the distance between the human body and the ground, the average depth of the ground area and the average depth of the human body area are used to detect human falls, thereby reducing false detections and missed detections, and improving the accuracy and stability of human fall detection.

[0086] In one embodiment, see Figure 4 , said S4 comprises:

[0087] S41: Obtain a training image with a deep label in a video surveillance scene, input the training image into a deep learning network, and obtain a training model;

[0088] Specifically, first, it is necessary to obtain a training image with a depth label and input it into a deep learning network for training to obtain a model that can perform depth estimation on monocular images. During the training process, the deep learning network will learn the correspondence between the depth information in the input training image and the pixel value of the training image.

[0089] S42: Inputting the real-time image into the training model for forward propagation to obtain depth values ​​corresponding to all pixels in the real-time image;

[0090] Specifically, after obtaining the trained model, the required monocular image is input into the model for forward propagation to obtain the depth value corresponding to each pixel in the input monocular image. These depth values ​​provide more accurate data support for subsequent fall detection.

[0091] S43: acquiring a human region depth value sequence and a ground region depth value sequence corresponding to multiple frames of sequential images according to the human region position information and the ground region position information and in combination with depth values ​​corresponding to all pixels in the real-time image;

[0092] In one embodiment, see Figure 5 , the S43 comprises:

[0093] S431: outputting depth values ​​corresponding to pixel points in the human body region to the human body region depth value sequence according to the human body region position information;

[0094] Specifically, for the human body area, we can determine the coordinate position of each pixel in the human body area based on the segmentation results of the previous human body detection model, and then filter out the depth value belonging to the human body area from the depth values ​​corresponding to all pixels in the above monocular image, and output it to the human body area depth value sequence.

[0095] S432: Outputting depth values ​​corresponding to pixel points in the ground area to the ground area depth value sequence according to the ground area position information.

[0096] Specifically, the position information of the ground area is obtained, and a set of depth values ​​corresponding to the pixels in the ground area is taken as the ground area depth value sequence D (d1, d2, d3, ..., dn), wherein d1, d2, d3, ..., dn represent the depth value corresponding to the first pixel to the nth pixel in the segmented ground area.

[0097] S44: deriving the ground area average depth sequence and the human area average depth sequence according to the human area depth value sequence and the ground area depth value sequence.

[0098] Specifically, according to the human body area depth value sequence and the ground area depth value sequence, the ground area average depth and the human body area average depth are obtained. After the ground area average depth and the human body area average depth are obtained, it is possible to more accurately judge whether the human body has fallen. When falling, the average depth of the human body area will be significantly reduced, and compared with the average depth of the ground area, the average depth of the human body area will have a large deviation. Therefore, by comparing the average depth of the human body area and the average depth of the ground area, it is possible to more reliably judge whether the human body has fallen and avoid misjudgment.

[0099] In one embodiment, see Figure 6 , the S44 comprises:

[0100] S441: acquiring a first pixel point sequence in the human body region and a second pixel point sequence in the ground region according to the human body region position information and the ground region position information;

[0101] S442: Obtaining a first sliding window and a second sliding window according to the first pixel point sequence and the second pixel point sequence;

[0102] S443: Obtaining a first filtered depth value sequence and a second filtered depth value sequence according to the first sliding window and the second sliding window;

[0103] Specifically, for each pixel in the first pixel sequence, its adjacent points (such as w adjacent points before and after) are taken to form a 2w+1 sliding window, and the sliding window is used as the first sliding window. The median of the depth values ​​corresponding to all pixels in the first sliding window is calculated, and it is used as the depth value after filtering. The first filtered depth value sequence is obtained by calculating the depth values ​​of all pixels in the first pixel sequence after filtering. Similarly, the second filtered depth value sequence can be obtained. It should be noted that the size of the sliding window can be adjusted according to the actual application requirements. Using a sliding window to perform median filtering of the depth value can effectively eliminate isolated noise points and false detection points in the depth map, making the depth map smoother and more reliable. Since depth maps often have noise and false detection points in actual applications, if they are not filtered, they may have a greater impact on subsequent algorithms. Therefore, the use of sliding window median filtering can effectively improve the robustness and accuracy of the algorithm. In addition, the size of the sliding window can be adjusted according to the actual application requirements to achieve the best filtering effect.

[0104] S444: Perform sliding window processing on the human body area depth value sequence and the ground area depth value sequence according to the first filtered depth value sequence and the second filtered depth value sequence to obtain an average depth of the ground area and an average depth of the human body area.

[0105] Specifically, the first filtered depth value sequence D1_filtered and the second filtered depth value sequence D2_filtered are obtained, and the average depth of the ground area and the average depth of the human body area are calculated using the following formula:

[0106] D_person_avg=(d11_filtered+d12_filtered+……+d1n_filtered) / n;

[0107] D_ground_avg=(d21_filtered+d22_filtered+……+d2m_filtered) / m;

[0108] Among them, D_person_avg represents the average depth of the human body area, D_ground_avg represents the average depth of the ground area, d11_filtered, d12_filtered, d12_filtered, ... d1n_filtered respectively represent the filtered depth values ​​corresponding to the first pixel to the nth pixel in the first filtered depth value sequence, d22_filtered, d22_filtered, ... d2m_filtered respectively represent the filtered depth values ​​corresponding to the first pixel to the mth pixel in the second filtered depth value sequence, n is the number of points in the first filtered depth value sequence, and m is the number of points in the second filtered depth value sequence.

[0109] S5: Calculate the depth difference between the average depth sequence of the human body area and the average depth sequence of the ground area, and detect human falls based on the depth difference sequence.

[0110] Specifically, by acquiring the average depth sequence of the human body area corresponding to multiple frames of monocular images, more accurate human body position and posture information is obtained, and then based on the depth difference sequence between the average depth sequence of the human body area and the average depth of the ground area, the human fall behavior is more accurately detected.

[0111] In one embodiment, see Figure 7 , the S5 comprises:

[0112] S51: Obtain a preset depth difference threshold;

[0113] Specifically, in actual applications, due to different ways of acquiring depth information in different scenes and cameras, the required depth difference threshold will also be different. If the threshold is set too small, it may lead to a high false detection rate, that is, a situation where there is no fall is misjudged as a fall; if the threshold is set too large, it may lead to a high missed detection rate, that is, a situation where a fall is not detected. Therefore, a suitable threshold T is pre-set to determine whether the depth difference between the human body and the ground is small enough. This threshold can be adjusted according to the actual scene and camera parameters.

[0114] S52: Compare each depth difference in the depth difference sequence with the depth difference threshold in sequence to obtain a comparison result;

[0115] Specifically, for each frame of the input monocular image, the difference between the average depth of the human body and the depth of the ground is calculated according to the following formula:

[0116] de lta_D_i=abs(D_person_avg_i–D_ground_avg)

[0117] Among them, delta_D_i is the depth difference between the average depth of the human body in the i-th frame image and the ground depth, D_person_avg_i is the average depth of the human body area in the i-th frame image, D_ground_avg is the average depth of the ground area, and abs is the absolute value; the calculated depth difference is compared with the depth difference threshold in sequence. When the depth difference is less than the depth difference threshold, it is considered that the human body has fallen; when the depth difference is not less than the depth difference threshold, it is considered that the human body has not fallen. Judging according to the set depth difference threshold, it is possible to accurately detect whether the human body has fallen, thereby achieving effective video monitoring and safety management.

[0118] S53: When the comparison results corresponding to the continuous multiple frames of images all show that the depth difference is less than the depth difference threshold, the result of the human body falling is that the human body falls on the ground.

[0119] Specifically, traverse the depth difference de lta_D_i of all frames. If the depth difference de lta_D_i of multiple consecutive frames (for example, 3 consecutive frames or more) is less than the threshold T, then we can assume that the human body has fallen to the ground. The depth difference of multiple consecutive frames is used to determine whether the human body has fallen, thereby improving the accuracy and reliability of detection. Since the position and posture of the human body will change greatly when falling, simply judging the depth difference of a certain frame may lead to misjudgment. However, by judging the depth difference of multiple consecutive frames, the possibility of misjudgment can be reduced, thereby improving the accuracy of fall detection. At the same time, by adjusting the number of frames and the threshold size of multiple consecutive frames, the sensitivity and real-time performance of the detection can also be controlled to better meet the needs of different scenarios.

[0120] Example 2

[0121] See also Figure 8 Embodiment 2 of the present invention further provides a ground area detection device in a video surveillance scenario, the device comprising:

[0122] An image acquisition module is used to acquire a real-time video stream in a video surveillance scenario and decompose the real-time video stream into multiple frames of real-time images;

[0123] An image segmentation module is used to segment the real-time image to obtain ground area location information if no human target is identified in the current monitoring scene and the current monitoring angle of view is not rotated;

[0124] In one embodiment, if no human target is identified in the current monitoring scene and the current monitoring angle of view is not rotated, the real-time image is segmented to obtain the ground area position information, and the following steps are also included:

[0125] A human target detection model acquisition unit is used to obtain a pre-trained human target detection model based on the yolov6s architecture;

[0126] A human target detection unit is used to input the real-time image into the human target detection model to identify whether a human target appears in the current monitoring scene;

[0127] The monitoring view rotation recognition unit is used to monitor the camera rotation by using the visual mileage calculation method and to recognize whether the current monitoring view rotation occurs.

[0128] In one embodiment, the image segmentation module includes:

[0129] A filtering and color space conversion unit, used for filtering and color space conversion of the real-time image to obtain a color space image if no human target is identified in the current monitoring scene and the current monitoring angle of view is not rotated, wherein the real-time image is a monocular image;

[0130] A feature extraction and fusion unit, used to extract features from the color space image, fuse the extracted local features with the global features, and obtain multi-scale features;

[0131] A probability map acquisition unit, used to acquire the probability of each pixel point being in the ground area according to the multi-scale features, and obtain a probability map;

[0132] A probability map conversion unit, used to obtain a preset probability threshold and convert the probability map into a binary segmentation map;

[0133] The morphological processing unit is used to perform morphological processing on the binary segmentation map, and use the pixel point position information in the ground area as the ground area position information.

[0134] A human body detection module, used to input each of the real-time images into a pre-trained human body detection model and output human body region position information;

[0135] A depth analysis module, used to perform depth analysis on the human body region position information and the ground region position information, and obtain an average depth sequence of the ground region and an average depth sequence of the human body region corresponding to multiple frames of time-series images;

[0136] In one embodiment, the in-depth analysis module includes:

[0137] A model training unit, used to obtain a training image with a deep label in a video surveillance scene, input the training image into a deep learning network, and obtain a training model;

[0138] A forward propagation unit, used to input the real-time image into the training model for forward propagation to obtain depth values ​​corresponding to all pixels in the real-time image;

[0139] A depth value sequence acquisition unit is used to acquire a human region depth value sequence and a ground region depth value sequence corresponding to multiple frames of time-series images based on the human region position information and the ground region position information and in combination with the depth values ​​corresponding to all pixels in the real-time image;

[0140] In one embodiment, the depth value sequence acquisition unit includes:

[0141] A human body region depth value sequence acquisition subunit, configured to output depth values ​​corresponding to pixel points in the human body region to the human body region depth value sequence according to the human body region position information;

[0142] The ground area depth value sequence acquisition subunit is used to output the depth values ​​corresponding to the pixel points in the ground area to the ground area depth value sequence according to the ground area position information.

[0143] The average depth sequence unit acquisition unit is used to obtain the ground area average depth sequence and the human area average depth sequence according to the human area depth value sequence and the ground area depth value sequence.

[0144] In one embodiment, the average depth sequence unit acquisition unit includes:

[0145] A pixel point sequence and acquisition subunit, used to acquire a first pixel point sequence in the human body area and a second pixel point sequence in the ground area according to the human body area position information and the ground area position information;

[0146] A sliding window acquisition subunit, used to obtain a first sliding window and a second sliding window according to the first pixel point sequence and the second pixel point sequence;

[0147] A filtering processing subunit, configured to obtain a first filtered depth value sequence and a second filtered depth value sequence according to the first sliding window and the second sliding window;

[0148] The sliding window processing subunit is used to perform sliding window processing on the human body area depth value sequence and the ground area depth value sequence according to the first filtered depth value sequence and the second filtered depth value sequence to obtain the average depth of the ground area and the average depth of the human body area.

[0149] The fall detection module calculates the depth difference between the average depth sequence of the human body area and the average depth sequence of the ground area, and detects human falls based on the depth difference sequence.

[0150] In one embodiment, the fall detection module comprises:

[0151] A depth difference threshold acquisition unit, used to acquire a preset depth difference threshold;

[0152] a depth difference comparison unit, configured to compare each depth difference in the depth difference sequence with the depth difference threshold in sequence to obtain a comparison result;

[0153] The fall judgment unit is used to determine that the human body has fallen to the ground when the comparison results corresponding to multiple consecutive frames of images all show that the depth difference is less than the depth difference threshold.

[0154] Specifically, a ground area detection device in a video surveillance scenario provided by an embodiment of the present invention is used, and the device includes: an image acquisition module, which is used to acquire a real-time video stream in a video surveillance scenario, and decompose the real-time video stream into multiple frames of real-time images; an image segmentation module, which is used to perform image segmentation on the real-time image if no human target is identified in the current monitoring scenario and the current monitoring angle of view is not rotated, so as to obtain ground area position information; a human body detection module, which is used to input each of the real-time images into a pre-trained human body detection model, and output human body area position information; a depth analysis module, which is used to perform depth analysis on the human body area position information and the ground area position information, and obtain a ground area average depth sequence and a human body area average depth sequence corresponding to multiple frames of time-series images; a fall detection module, which performs depth difference calculation on the human body area average depth sequence and the ground area average depth sequence, and detects human falls based on the depth difference sequence. This device relies on real-time video streams in video surveillance scenarios, which are obtained through common surveillance cameras. There is no need to introduce additional depth sensors, thereby reducing hardware costs; pre-training of human detection models means that there is no need to train the model from scratch, reducing development costs. Many pre-trained models have been trained on large-scale data for human detection in general scenarios; the location information of the ground area is obtained through real-time image segmentation and implemented through common computer vision algorithms without the need for complex depth image processing, which helps to reduce costs; by analyzing the average depth sequence of the ground and human areas corresponding to multiple frames of time-series images, the system's adaptability to dynamic environmental changes is increased, thereby reducing the probability of misjudgment; through depth difference calculation, the system can pay more attention to changes in relative depth rather than just absolute depth values, which helps to improve the stability of detection.

[0155] Example 3

[0156] In addition, combined Figure 1 The ground area detection method in the video surveillance scenario described in Embodiment 1 of the present invention may be implemented by an electronic device. Fig. 9 A schematic diagram of the hardware structure of an electronic device provided in Embodiment 3 of the present invention is shown.

[0157] The electronic device may include a processor and a memory storing computer program instructions.

[0158] Specifically, the processor may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present invention.

[0159] The memory may include a large capacity memory for data or instructions. By way of example and not limitation, the memory may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory may include a removable or non-removable (or fixed) medium. Where appropriate, the memory may be inside or outside a data processing device. In a particular embodiment, the memory is a non-volatile solid-state memory. In a particular embodiment, the memory includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM) or a flash memory or a combination of two or more of these.

[0160] The processor implements the ground area detection method in any one of the video surveillance scenarios in the above embodiments by reading and executing computer program instructions stored in the memory.

[0161] In one example, the electronic device may further include a communication interface and a bus. Fig. 9 As shown, the processor, memory, and communication interface are connected via a bus and communicate with each other.

[0162] The communication interface is mainly used to implement communication between the modules, devices, units and / or equipment in the embodiments of the present invention.

[0163] The bus includes hardware, software or both, and the parts of the device are coupled to each other. For example, but not limitation, the bus may include an accelerated graphics port (AGP) or other graphics bus, an enhanced industrial standard architecture (EISA) bus, a front-side bus (FSB), a hypertransport (HT) interconnection, an industrial standard architecture (ISA) bus, an infinite bandwidth interconnection, a low pin count (LPC) bus, a memory bus, a microchannel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a serial advanced technology attachment (SATA) bus, a video electronics standard association local (VLB) bus or other suitable bus or a combination of two or more of these. In suitable cases, the bus may include one or more buses. Although the embodiments of the present invention describe and illustrate specific buses, the present invention contemplates any suitable bus or interconnection.

[0164] Example 4

[0165] In addition, in combination with the ground area detection method in the video surveillance scenario in the above embodiment 1, embodiment 4 of the present invention can also provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any of the ground area detection methods in the video surveillance scenario in the above embodiments is implemented.

[0166] In summary, the embodiments of the present invention provide a method, apparatus, device and medium for detecting a ground area in a video surveillance scenario.

[0167] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0168] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0169] It should also be noted that the exemplary embodiments mentioned in the present invention describe some methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiments, or in a different order from the embodiments, or several steps can be performed simultaneously.

[0170] The above is only a specific implementation of the present invention. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system, module and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be covered within the protection scope of the present invention.

Claims

1. A ground area detection method in a video surveillance scenario, characterized in that: The method comprises: The real-time image obtained by decomposing the real-time video stream in the video surveillance scene is input into the pre-trained human target detection model to identify whether a human target appears in the current surveillance scene; The visual odometry calculation method is used to monitor the rotation of the camera and to identify whether the current monitoring view has rotated by comparing the posture changes between adjacent frames. If no human target is identified in the current monitoring scene and the current monitoring angle of view is not rotated, the real-time image is filtered and the color space is converted to obtain a color space image; Extracting features from the color space image, fusing the extracted local features with the global features, and obtaining multi-scale features; According to the multi-scale features, the probability of each pixel being in the ground area is obtained, a preset probability threshold is obtained, and the probability map is converted into a binary segmentation map; Morphological processing is performed on the binary segmentation map, and the pixel position information in the ground area is used as the ground area position information.

2. The ground area detection method in a video surveillance scene according to claim 1, characterized in that: If no human target is identified in the current monitoring scene and the current monitoring angle of view is not rotated, the real-time image is segmented to obtain the ground area position information, and the method further includes: Inputting each of the real-time images into a pre-trained human body detection model, and outputting human body region position information; Performing depth analysis on the human body region position information and the ground region position information to obtain an average ground region depth sequence and an average human body region depth sequence corresponding to multiple frames of time-series images; The depth difference between the average depth sequence of the human body area and the average depth sequence of the ground area is calculated, and a human fall is detected based on the depth difference sequence.

3. The ground area detection method in a video surveillance scene according to claim 2, characterized in that: The performing depth analysis on the human body region position information and the ground region position information to obtain the ground region average depth sequence and the human body region average depth sequence corresponding to multiple frames of time-series images comprises: Obtaining a training image with a deep label in a video surveillance scene, inputting the training image into a deep learning network, and obtaining a training model; Input the real-time image into the training model for forward propagation to obtain depth values ​​corresponding to all pixels in the real-time image; According to the human body area position information and the ground area position information, combined with the depth values ​​corresponding to all pixels in the real-time image, a human body area depth value sequence and a ground area depth value sequence corresponding to multiple frames of time-series images are obtained; The ground area average depth sequence and the human area average depth sequence are obtained according to the human area depth value sequence and the ground area depth value sequence.

4. The ground area detection method in a video surveillance scene according to claim 3 is characterized in that: The method of obtaining a human region depth value sequence and a ground region depth value sequence corresponding to multiple frames of sequential images based on the human region position information and the ground region position information and combining the depth values ​​corresponding to all pixels in the real-time image comprises: Outputting depth values ​​corresponding to pixel points in the human body region to the human body region depth value sequence according to the human body region position information; According to the ground area position information, the depth values ​​corresponding to the pixel points in the ground area are output to the ground area depth value sequence.

5. The ground area detection method in a video surveillance scene according to claim 3 is characterized in that: The step of obtaining the ground area average depth sequence and the human area average depth sequence according to the human area depth value sequence and the ground area depth value sequence comprises: Acquire a first pixel point sequence in the human body region and a second pixel point sequence in the ground region according to the human body region position information and the ground region position information; Obtaining a first sliding window and a second sliding window according to the first pixel point sequence and the second pixel point sequence; According to the first sliding window and the second sliding window, a first filtered depth value sequence and a second filtered depth value sequence are obtained; According to the first filtered depth value sequence and the second filtered depth value sequence, sliding window processing is performed on the human body area depth value sequence and the ground area depth value sequence to obtain an average depth of the ground area and an average depth of the human body area.

6. The ground area detection method in a video surveillance scene according to any one of claims 2 to 5, characterized in that: The calculating of the depth difference between the average depth sequence of the human body area and the average depth sequence of the ground area, and detecting a human fall according to the depth difference sequence comprises: Get the preset depth difference threshold; Comparing each depth difference in the depth difference sequence with the depth difference threshold in sequence to obtain a comparison result; When the comparison results corresponding to the continuous multiple frames of images all show that the depth difference is less than the depth difference threshold, the result of the human body falling is that the human body falls on the ground.

7. The ground area detection method in a video surveillance scene according to claim 1, characterized in that: If no human target is identified in the current monitoring scene and the current monitoring angle of view is not rotated, filtering and color space conversion are performed on the real-time image to obtain a color space image, including: Get the pre-trained human target detection model based on yolov6s architecture; Inputting the real-time image into the human target detection model to identify whether a human target appears in the current monitoring scene; The visual odometry calculation method is used to monitor the rotation of the camera and identify whether the current monitoring angle of view rotates.

8. A ground area detection device in a video surveillance scene, characterized in that: The device comprises: The human target detection module is used to input the real-time image obtained by decomposing the real-time video stream in the video surveillance scene into the pre-trained human target detection model to identify whether a human target appears in the current surveillance scene; The view rotation monitoring module is used to monitor the camera rotation by using the visual mileage calculation method, and to identify whether the current monitoring view rotates by comparing the posture changes between adjacent frames; A color space image acquisition module is used to filter and convert the real-time image into a color space to obtain a color space image if no human target is identified in the current monitoring scene and the current monitoring angle of view is not rotated; A feature extraction module, used to extract features from the color space image, fuse the extracted local features with the global features, and obtain multi-scale features; A binary segmentation map acquisition module is used to obtain the probability of each pixel point being in the ground area based on the multi-scale features, obtain a preset probability threshold, and convert the probability map into a binary segmentation map; The ground area position information acquisition module is used to perform morphological processing on the binary segmentation map and use the pixel point position information in the ground area as the ground area position information.

9. An electronic device, characterized in that: include: At least one processor, at least one memory and computer program instructions stored in the memory, when the computer program instructions are executed by the processor, implement the method according to any one of claims 1 to 7.

10. A storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images