Personnel stay monitoring method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a kind of personnel stay monitoring method, the method comprises: obtaining the current monitoring frame of monitoring area, based on image similarity, multiple shooting images are carried out rough deduplication processing, obtain multiple to be fused images, the transparency of the overlapping area between each to-be-fused image is calculated, and based on the transparency of the overlapping area, to-be-fused image is carried out image fusion, obtain fusion image, the number of persons in fusion image is detected, obtain the number of persons in fusion image, if the number of persons in fusion image satisfies pre-set number, current monitoring frame is determined as current effective frame, determine whether continuous effective frame satisfies pre-set condition, if continuous effective frame satisfies pre-set condition, it is determined that there is personnel stay in monitoring area.The personnel stay of large public area is monitored by the above method, the number of persons in flow state can be more accurately counted and analyzed, so as to improve the accuracy and efficiency of personnel stay monitoring method.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of urban management, and in particular to a personnel retention monitoring method and device, an electronic device and a storage medium. BACKGROUND

[0002] In the traditional personnel retention monitoring technology, multiple monitoring devices are arranged in a specific area, and images of the area are captured by the devices. However, according to the traditional personnel retention monitoring technology, due to the position setting of the monitoring devices or the image processing method, the images captured by the monitoring devices may have problems such as overlap and blur, which affects the accurate counting of the number of personnel in the specific area. Therefore, the existing personnel retention monitoring method has the problem that the obtained images may have overlap and blur, which leads to inaccurate counting of the number of personnel in the specific area. SUMMARY

[0003] The embodiments of the present application provide a personnel retention monitoring method, which aims to solve the problem that the existing personnel retention monitoring method cannot accurately count the number of personnel in a specific area through deployed monitoring devices. By performing coarse deduplication processing on multiple captured images in the specific area, multiple to-be-fused images are obtained, and based on the transparency of the overlapping area between each to-be-fused image, image fusion is performed on each to-be-fused image, and after person number detection and continuous valid frame confirmation on the fused image, it is determined whether there is personnel retention in the specific area. By determining the personnel retention condition through continuous valid frames, the number of personnel in a flowing state can be more accurately counted and analyzed, thereby improving the accuracy and efficiency of the personnel retention monitoring method.

[0004] In a first aspect, the embodiments of the present application provide a personnel retention monitoring method, characterized in that the method comprises the following steps:

[0005] obtaining a current monitoring frame of a monitoring area, the current monitoring frame comprising multiple captured images captured by multiple monitoring devices at a current time, the monitoring area being monitored by the multiple monitoring devices;

[0006] performing coarse deduplication processing on the multiple captured images based on image similarity to obtain multiple to-be-fused images, the number of to-be-fused images being less than or equal to the number of captured images;

[0007] calculating the transparency of the overlapping area between each to-be-fused image, and performing image fusion on the to-be-fused images based on the transparency of the overlapping area to obtain a fused image;

[0008] performing person number detection on the fused image to obtain the number of persons in the fused image;

[0009] If the number of people in the fused image meets the preset number, then the current monitoring frame is determined as the current valid frame;

[0010] Determine whether consecutive valid frames meet preset conditions. The consecutive valid frames include current valid frames and historical valid frames. Two adjacent valid frames in the consecutive valid frames have a preset frame extraction interval.

[0011] If the consecutive valid frames meet the preset conditions, it is determined that there are people staying in the monitoring area.

[0012] Optionally, obtaining the current monitoring frame of the monitoring area includes:

[0013] Obtain the area and the speed of personnel flow in the monitored area;

[0014] Based on the area of ​​the monitoring area and the speed of personnel flow, a preset frame extraction interval is determined;

[0015] Based on the preset frame extraction interval, the current monitoring frame of the monitoring area is determined.

[0016] Optionally, the coarse deduplication process based on image similarity to obtain multiple images to be fused includes:

[0017] A hash calculation is performed on the multiple captured images to obtain the hash code of each captured image;

[0018] Based on the hash codes of the captured images, the Hamming distance between each captured image is obtained;

[0019] The Hamming distance between each of the captured images is compared with a preset Hamming distance threshold. Among the captured images whose Hamming distance is greater than the preset Hamming distance threshold, the captured images that need to be deduplicated are identified and removed to obtain multiple images to be fused.

[0020] Optionally, calculating the transparency of the overlapping regions between the images to be fused, and performing image fusion on the images to be fused based on the transparency of the overlapping regions to obtain a fused image, includes:

[0021] Extract feature values ​​of the overlapping regions between the images to be fused;

[0022] Based on the feature values ​​of the overlapping regions between the images to be fused, the similarity of the overlapping regions between the images to be fused is calculated.

[0023] The transparency of the overlapping regions between the images to be fused is determined based on the similarity of the overlapping regions between the images to be fused.

[0024] Based on the transparency of the overlapping areas between the images to be fused, the images to be fused are fused to obtain a fused image.

[0025] Optionally, determining the transparency of the overlapping regions between the images to be fused based on the similarity of the overlapping regions between the images to be fused includes:

[0026] Based on the similarity of the overlapping regions between the images to be fused, the heatmap distribution of the overlapping regions between the images to be fused is determined;

[0027] The transparency of the overlapping regions between the images to be fused is determined based on the heatmap distribution of the overlapping regions between the images to be fused.

[0028] Optionally, determining the transparency of the overlapping regions between the images to be fused based on the heatmap distribution of the overlapping regions between the images to be fused includes:

[0029] Based on the heat map distribution of the overlapping regions between the images to be fused, the thermal proportion of the overlapping regions between the images to be fused is determined.

[0030] The transparency of the overlapping regions between the images to be fused is determined based on the thermal proportion of the overlapping regions between the images to be fused.

[0031] Optionally, determining that there are people staying in the monitoring area if the consecutive valid frames meet the preset conditions includes:

[0032] Obtain historical valid frames preceding the current valid frame, wherein there is a preset frame extraction interval between the last frame of the historical valid frames and the current valid frame;

[0033] Determine consecutive valid frames based on historical valid frames and current valid frames;

[0034] Calculate the number of consecutive valid frames;

[0035] Based on the number of frames, determine the frame extraction duration for consecutive valid frames;

[0036] If the frame extraction duration of consecutive valid frames is greater than or equal to the preset frame extraction duration, it is determined that there are people staying in the target area.

[0037] Secondly, embodiments of the present invention also provide a personnel stay monitoring device, the personnel stay monitoring device comprising:

[0038] The first acquisition module is used to acquire the current monitoring frame of the monitoring area. The current monitoring frame includes multiple images captured by multiple monitoring devices at the current time. The monitoring area is monitored by the multiple monitoring devices.

[0039] The deduplication module is used to perform coarse deduplication on multiple captured images based on image similarity to obtain multiple images to be fused, wherein the number of images to be fused is less than or equal to the number of captured images;

[0040] The calculation module is used to calculate the transparency of the overlapping areas between the images to be fused, and to perform image fusion on the images to be fused based on the transparency of the overlapping areas to obtain a fused image;

[0041] The detection module is used to detect the number of people in the fused image to obtain the number of people in the fused image;

[0042] The first determining module is used to determine the current monitoring frame as the current valid frame if the number of people in the fused image meets the preset number.

[0043] The second determining module is used to determine whether consecutive valid frames meet preset conditions. The consecutive valid frames include current valid frames and historical valid frames. Two adjacent valid frames in the consecutive valid frames have a preset frame extraction interval.

[0044] The third determining module is used to determine that there are people staying in the monitoring area if the consecutive valid frames meet the preset conditions.

[0045] Thirdly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the personnel stay monitoring method provided in embodiments of the present invention.

[0046] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the personnel stay monitoring method provided in the embodiments of the present invention.

[0047] In this embodiment of the invention, the current monitoring frame of the monitoring area is acquired. Multiple captured images are coarsely deduplicated based on image similarity to obtain multiple images to be fused. The transparency of the overlapping areas between the images to be fused is calculated, and the images to be fused are fused based on the transparency of the overlapping areas to obtain a fused image. People are detected in the fused image to obtain the number of people in the fused image. If the number of people in the fused image meets a preset number, the current monitoring frame is determined as the current valid frame. It is then determined whether consecutive valid frames meet preset conditions. If consecutive valid frames meet the preset conditions, it is determined that there are people lingering in the monitoring area. By coarsely deduplicating multiple captured images within a specific area to obtain multiple images to be fused, and by fusing the images based on the transparency of the overlapping areas between them, and then performing people detection and confirming consecutive valid frames in the fused image, it is determined whether there are people lingering in the specific area. Determining the number of people lingering in the specific area through consecutive valid frames allows for more accurate statistics and analysis of the number of people in a moving state, thereby improving the accuracy and efficiency of the people lingering monitoring method. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart of a method for monitoring personnel stay provided in an embodiment of the present invention;

[0050] Figure 2 This is a flowchart of another method for monitoring personnel stay provided in an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of the structure of a personnel stay monitoring device provided in an embodiment of the present invention;

[0052] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] like Figure 1 As shown, Figure 1 This is a flowchart of a method for monitoring personnel stay according to an embodiment of the present invention. The method includes the following steps:

[0055] 101. Obtain the current monitoring frame of the monitoring area.

[0056] In this embodiment of the invention, the aforementioned method for monitoring personnel lingering can be applied to an urban management platform. This platform stores monitoring image information of various public areas and has data acquisition, data analysis, data transmission, and data storage functions. The monitoring image information of the public areas can be captured by multiple monitoring devices deployed within the public areas and acquired and stored by the urban management platform.

[0057] The aforementioned monitoring areas can be defined by urban management personnel based on urban development plans, delineating areas where population surveys or movement tracking are necessary. Specifically, multiple monitoring devices are typically required to monitor these areas comprehensively, thus fully capturing the scene. Examples include tracking the occupancy of people in prison public areas, large public areas, and public areas with high population flow, such as subway platforms.

[0058] The aforementioned current monitoring frame can be multiple images captured by multiple monitoring devices at the current moment. Specifically, city management personnel can set the frame extraction interval of the monitoring frame according to the actual implementation plan, and extract the monitoring images acquired by the monitoring devices according to the set frame extraction interval. The frame extraction interval can be set to check every 30 minutes if the population flow in the area is fast and the area is wide, that is, every 30 minutes is one frame interval. According to the set frame extraction interval, each monitoring image after 30 minutes is selected and used as the current monitoring frame.

[0059] In one possible embodiment, urban management personnel use the aforementioned urban management platform to set a preset frame extraction interval based on the area and pedestrian flow speed within the monitoring area, and select monitoring images acquired by monitoring devices within the monitoring area according to the preset frame extraction interval, using the selected monitoring image as the current monitoring frame.

[0060] 102. Perform coarse deduplication on multiple captured images based on image similarity to obtain multiple images to be fused.

[0061] In this embodiment of the invention, the aforementioned image similarity can be used as a standard to measure the degree of similarity between two images. By calculating the similarity between each image, a rough deduplication of images can be performed, that is, some suspected or confirmed duplicate images can be roughly removed. The number of images to be fused is less than or equal to the number of captured images, and there are no similar or overlapping areas between the images to be fused. Specifically, image processing algorithms such as image hashing algorithms, feature extraction algorithms, duplicate image identification, and Phash algorithms can be used to perform deduplication processing between images. After the deduplication process, at least one or more duplicate photos need to be retained and added to the list of images to be fused.

[0062] In one possible embodiment, the urban management platform performs similarity calculations on multiple acquired monitoring images, and performs deduplication processing on the multiple monitoring images based on the calculation results to obtain multiple images to be fused, and stores them in the database of the urban management platform.

[0063] 103. Calculate the transparency of the overlapping areas between the images to be fused, and perform image fusion based on the transparency of the overlapping areas to obtain the fused image.

[0064] In this embodiment of the invention, the aforementioned transparency refers to the degree of light transmission between images. In image fusion technology, this degree of light transmission can be reflected in the pixel coverage effect of the upper image on the lower image. Specifically, in image fusion, when the upper image covers the lower image, if the transparency of the upper image is 100%, the details of the lower image will be fully revealed. Conversely, when the transparency of the upper image is 0%, the details of the lower image will be completely covered by the upper image, thus achieving the effect that only the upper image can be seen.

[0065] Image fusion, as described above, refers to the process of combining information from multiple images into a single image using an image fusion algorithm. The fused image is then called the merged image. Specifically, this can be achieved using image processing algorithms such as averaging and weighted algorithms, pixel grayscale selection algorithms, and PCA fusion methods.

[0066] In one possible embodiment, the aforementioned urban management platform extracts features from monitoring images, calculates the corresponding similarity based on the extracted features, and then maps the similarity levels to color shades using a preset color mapping method, forming a heatmap similar to a heat distribution. This heatmap is then used to weightedly fuse the monitoring images, resulting in a fused image. The feature extraction can be performed using a convolutional neural network. Specifically, a feature extraction model to be trained is first constructed and then subjected to supervised training. The scene image is input into the model, and the output is a feature image of the scene image. The model is trained with the feature image similarity to the scene image being close to 1 as optimal. When the output feature image similarity is close to 1, the trained feature extraction model is obtained. The trained feature extraction model is then used to extract features from the monitoring images so that similarity can be calculated subsequently based on these features.

[0067] 104. Perform people detection on the fused image to obtain the number of people in the fused image.

[0068] In this embodiment of the invention, the aforementioned number of people can be detected by using head and shoulder vision algorithms, feature clustering algorithms, and region growing algorithms to identify the fused image. Specifically, in this embodiment, the number of people in the fused image is obtained by identifying the head and shoulder parts of the human body in the fused image and calculating the number of people based on the head and shoulder vision algorithm.

[0069] 105. If the number of people in the fused image meets the preset number, then the current monitoring frame is determined as the current valid frame.

[0070] In this embodiment of the invention, the preset number of people can be determined based on the flow speed of people within the monitoring area and the area itself. Specifically, the preset number of people can be set according to the rule that a larger monitoring area corresponds to a higher preset number of people, or a slower flow speed of people within the monitoring area corresponds to a higher preset number of people. The preset number of people can be determined according to the following formula:

[0071]

[0072] Where P is the preset population ratio parameter, A is the area parameter of the monitoring area, B is the population flow speed parameter of the monitoring area, and T is the population flow rate parameter of the monitoring area. s The standard is the reasonable number of people that can be accommodated within one square meter. As can be seen from the formula, the larger the area parameter of the monitoring area and the smaller the personnel flow speed parameter of the monitoring area, the larger the preset number of people parameter corresponding to the monitoring area, that is, the larger the preset number of people should be.

[0073] The aforementioned currently valid frame can be a fused image where the current monitoring frame meets the preset number of people. Specifically, when the number of people in the fused image meets the preset number, the monitoring frame corresponding to that fused image is determined as the currently valid frame. For example, considering the area of ​​the monitoring area and the speed of people flow, the preset number is set to 1000 people. When the number of people in the fused image calculated by the aforementioned people detection algorithm is 1000 or 1001, the number of people in the fused image is greater than or equal to the preset number, and the monitoring frame corresponding to the current fused image is then taken as the currently valid frame.

[0074] 106. Determine whether consecutive valid frames meet the preset conditions.

[0075] In this embodiment of the invention, the aforementioned consecutive valid frames include, but are not limited to, the current valid frame and historical valid frames. The historical valid frames can be calculated from the previous invalid frame until the next invalid frame appears, and the consecutive valid frames between these two invalid frames can all be considered historical valid frames. Specifically, there is a preset frame skipping interval between the historical valid frames and the current valid frame, and there is also a preset frame skipping interval between two adjacent valid frames in the aforementioned consecutive valid frames.

[0076] Generally, multiple consecutive currently valid frames can be treated as a whole. By setting the frame extraction interval and the number of valid frames, the total duration of the valid frames can be calculated. The total duration is then compared with the frame extraction interval. If the total duration of the valid frames is greater than or equal to the frame extraction interval, then the consecutive valid frames are determined to meet the preset conditions.

[0077] 107. If consecutive valid frames meet the preset conditions, it is determined that there are people staying in the monitoring area.

[0078] In this embodiment of the invention, the aforementioned personnel congestion refers to a situation where personnel remain in a certain place or location for a period exceeding the originally planned or stipulated time, and are unable to leave on time. Specifically, in this embodiment, personnel congestion may occur due to factors such as control measures in the monitored area, resulting in slow personnel flow. For example, during peak hours on the subway, subway platforms may implement personnel flow control measures, causing the entry or exit queues to form only one long line, thus resulting in personnel congestion.

[0079] In one possible embodiment, after processing the aforementioned consecutive valid frames, if the processing result indicates that the consecutive valid frames meet the aforementioned preset conditions, the urban management platform determines that the monitoring area corresponding to the consecutive valid frames may be in a situation where people are staying, and then notifies urban management personnel to manage the aforementioned monitoring area through a terminal or other device.

[0080] Optionally, in the step of obtaining the current monitoring frame of the monitoring area, the area of ​​the monitoring area and the flow speed of people can also be obtained, and then a preset frame extraction interval can be determined based on the area of ​​the monitoring area and the flow speed of people. Finally, the current monitoring frame of the monitoring area can be determined based on the preset frame extraction interval.

[0081] In this embodiment of the invention, the area of ​​the monitoring area can be estimated by estimating the area of ​​the region in the monitoring image. Specifically, after acquiring monitoring images of multiple monitoring areas, the monitoring images are deduplicated to obtain a general scene image of the monitoring area. The area occupied by the image is then estimated based on reference objects within the image. It should be noted that the reference objects can be selected according to specific embodiments to reduce errors. For example, a trash can or tile with a standard volume can be used as a reference object to estimate the area of ​​the monitoring area. The personnel flow speed can be calculated as the ratio of the distance traveled by a person from their position at a first moment to their position at a second moment, to the time taken. Specifically, the movement distance is obtained by recording the position information at the first moment and the position information at the second moment, and the ratio of the movement distance to the time information is used as the personnel flow speed based on the time information at the recorded position.

[0082] More specifically, the preset frame skipping interval can be set based on the monitoring area and the flow speed of people, according to the following formula:

[0083] Q = ∑(0.7G + 0.3H) × T1

[0084] Where Q is the preset frame sampling interval, usually measured in frames; G is the parameter representing the influence of the monitoring area's area on the frame sampling interval; H is the parameter representing the influence of people's flow speed on the frame sampling interval, the specific influence of which needs to be determined based on the environmental or behavioral factors present in the specific implementation plan; and T1 is the frequency of frames that a normal human eye can observe. From the above formula, it can be seen that the larger the monitoring area and the faster the people's flow speed, the larger the preset frame sampling interval, meaning more different information is captured, which is more beneficial for accurate statistics.

[0085] In one possible embodiment, when the urban management platform acquires the monitoring images of the monitoring area, it analyzes the monitoring images to obtain a preset frame extraction interval, and determines the content of the current monitoring frame based on the preset frame extraction interval.

[0086] Optionally, in the step of performing coarse deduplication on multiple captured images based on image similarity to obtain multiple images to be fused, hash calculation can also be performed on multiple captured images to obtain the hash code of each captured image. Then, based on the hash code of the captured images, the Hamming distance between each captured image can be obtained. Finally, the Hamming distance between each captured image is compared with a preset Hamming distance threshold. Among the captured images whose Hamming distance is greater than the preset Hamming distance threshold, the captured images that need to be deduplicated are identified and removed to obtain multiple images to be fused.

[0087] In this embodiment of the invention, the hash calculation can be a process of calculating the hash of the multiple captured images using a perceptual hash algorithm or a mean hash algorithm. The hash code can be a string of values ​​obtained after calculating the hash of the multiple captured images. Specifically, the hash code obtained for the same image will be different depending on the hash algorithm used. Therefore, when using a hash algorithm to calculate the similarity of images, it is necessary to select the same hash algorithm. For example, the mean hash algorithm compares the average pixel value of the image with the value of each pixel in the image. It converts the image into a string of hash values ​​by assigning a value greater than or equal to the average pixel value of the image as 1, and assigning a value less than the average pixel value as 0. The similarity between the images is then determined based on these hash values.

[0088] The Hamming distance mentioned above refers to the number of hash codes that need to be converted when synchronizing two hash codes. For example, when synchronizing hash codes 110000 and 111000, regardless of which hash code is used for the conversion, only the third bit needs to be changed from 1 to 0 or 0 to 1. That is, only one bit of hash code needs to be changed, and the Hamming distance between the two hash codes is 1. The above-mentioned preset Hamming distance threshold can be set according to the actual scenario and needs. Specifically, the areas where the shooting ranges of multiple devices overlap in the current scene can be counted, and all overlapping areas can be divided into groups of two devices. According to the proportion of the overlapping area in the device's shooting area, they can be arranged from largest to smallest, and then the Hamming distance threshold can be calculated according to the following formula:

[0089]

[0090] Where V is the Hamming distance threshold, W represents the proportion of the overlapping area in the device's shooting area, N represents the number of areas where this overlap proportion occurs, and M represents the total number of overlapping shooting areas. It can be seen that the larger the proportion of the overlapping area in the device's shooting area, and the more areas where this overlap proportion occurs, the larger the Hamming distance threshold should be set.

[0091] In one possible embodiment, after acquiring multiple captured images, the urban management platform performs hash calculations on the multiple captured images to obtain the hash codes corresponding to each captured image, calculates the Hamming distance between them, sets the Hamming distance threshold according to the Hamming distance threshold setting formula, and determines that images with a Hamming distance greater than the preset Hamming distance threshold are duplicate images that need to be removed. The platform then performs deduplication processing on the multiple captured images to finally obtain multiple fused images.

[0092] Optionally, in the steps of calculating the transparency of the overlapping regions between the images to be fused and performing image fusion based on the transparency of the overlapping regions to obtain the fused image, the feature values ​​of the overlapping regions between the images to be fused can be extracted. Then, based on the feature values ​​of the overlapping regions between the images to be fused, the similarity of the overlapping regions between the images to be fused can be calculated. Then, based on the similarity of the overlapping regions between the images to be fused, the transparency of the overlapping regions between the images to be fused can be determined. Finally, based on the transparency of the overlapping regions between the images to be fused, the images to be fused can be fused to obtain the fused image.

[0093] In this embodiment of the invention, the feature values ​​of the overlapping regions between the images to be fused can refer to identifiable and highly recognizable environmental features, such as a green ball with thorns on sandy soil, which evokes the image of a cactus. The similarity of the overlapping regions between the images to be fused can be calculated based on the proportion of the feature values ​​in the area of ​​the overlapping regions, thus obtaining the similarity between the overlapping regions. Based on the similarity, the transparency of the overlapping regions between the images to be fused is determined. The transparency of the overlapping regions between the images to be fused can refer to the light transmittance of each layer of the image between the overlapping regions. For example, when there are three layers of images between the overlapping regions, if the transparency of the upper layer image is 100%, the transparency of the middle layer image is 100%, and the transparency of the lower layer image is 0%, then the pixel information of the overlapping image is entirely the image information of the lower layer image. Therefore, the transparency of the image can be determined according to the similarity between the images in the overlapping regions, using a weighted method.

[0094] Optionally, in the step of determining the transparency of the overlapping regions between images to be fused based on the similarity of the overlapping regions between the images to be fused, the heatmap distribution of the overlapping regions between the images to be fused can also be determined based on the similarity of the overlapping regions between the images to be fused, and then the transparency of the overlapping regions between the images to be fused can be determined based on the heatmap distribution of the overlapping regions between the images to be fused.

[0095] In this embodiment of the invention, the heat map distribution of the overlapping areas between the above-mentioned images to be fused can refer to the process of calculating the proportion of each image to be fused in the overlapping areas based on the similarity of the overlapping areas between the above-mentioned images to be fused, mapping the similarity to the corresponding color according to a preset color mapping method, and generating an image with color distribution for the corresponding overlapping areas according to the rule that the higher the similarity, the redder the color, and the lower the similarity, the bluer the color.

[0096] In one possible embodiment, after obtaining the similarity of the overlapping areas between the images to be fused, the urban management platform generates a fused image with a red and blue color distribution based on the calculation of the similarity and the rule that the higher the similarity, the redder the color, and the lower the similarity, the bluer the color. Based on the fused image with the red and blue color distribution, the transparency of the image corresponding to the redder overlapping area is adjusted to be lower during the fusion process, and the transparency of the image corresponding to the bluer overlapping area is adjusted to be higher during the fusion process.

[0097] Optionally, in the step of determining the transparency of the overlapping regions between the images to be fused based on the heat map distribution of the overlapping regions between the images to be fused, the thermal proportion of the overlapping regions between the images to be fused can also be determined based on the heat map distribution of the overlapping regions between the images to be fused, and then the transparency of the overlapping regions between the images to be fused can be determined based on the thermal proportion of the overlapping regions between the images to be fused.

[0098] In this embodiment of the invention, the aforementioned thermal proportion refers to the proportion of an image in the overlapping area of ​​the heatmap of the fused image. Specifically, the transparency of the image during fusion can be adjusted based on the proportion of the overlapping area occupied by the image during fusion. For example, if the image occupies a high proportion of the overlapping area during fusion, it indicates that the image is too similar to other images, i.e., its features are less distinct, and its transparency during fusion should be reduced. Conversely, if the image occupies a low proportion of the overlapping area during fusion, it indicates that the image is too similar to other images, i.e., its features are more distinct, and its transparency during fusion should be increased.

[0099] In one possible embodiment, when the aforementioned urban management platform invokes the personnel lingering monitoring method, it performs a weighted calculation based on the proportion of a certain image in the heatmap of the merged image, and dynamically adjusts its transparency. Specifically, during image merging, the transparency of the previously merged image can be adjusted according to the number of times the images have been merged. For example, after the first image is merged with the second image, the transparency of the first and second images is adjusted to obtain the first merged image. When the third image is merged, the transparency of the first merged image and the third image is adjusted according to the proportion of the third image in the heatmap of the merged image, and then the third image is merged into the first merged image to obtain the second merged image, and so on.

[0100] Optionally, in the step of determining that there are people staying in the monitoring area if the consecutive valid frames meet the preset conditions, it is also possible to obtain the historical valid frames before the current valid frame, then determine the consecutive valid frames based on the historical valid frames and the current valid frame, then calculate the number of consecutive valid frames, and then determine the frame extraction time of the consecutive valid frames based on the number of frames. Finally, when the frame extraction time of the consecutive valid frames is greater than or equal to the preset frame extraction time, it is determined that there are people staying in the target area.

[0101] In this embodiment of the invention, a preset frame-skipping interval exists between the last frame of the aforementioned historical valid frames and the current valid frame. The current valid frame is consecutively extended to the first frame of the historical valid frames as consecutive valid frames, and the number of consecutive valid frames is recorded. Based on the preset frame-skipping interval, the number of frames is calculated to obtain the frame-skipping duration of the consecutive valid frames, and this duration is compared with the preset frame-skipping duration. If the frame-skipping duration of the consecutive valid frames is greater than or equal to the preset frame-skipping duration, it is determined that there are people lingering in the target area. For example, based on a preset frame-skipping interval of 10 seconds, divided into one frame per second, if the consecutive valid frames are greater than or equal to 10 seconds, it indicates that there are people lingering in the target area; that is, in ten consecutive fused images, the number of people exceeds a preset number, thus confirming the presence of people lingering in the target area.

[0102] like Figure 2 As shown in the figure, this embodiment of the invention also provides a personnel stay monitoring process, which includes:

[0103] Initially, alarm thresholds and frame extraction intervals are set based on the scene factors of the current public area. Then, video frames are extracted from all devices in the public area according to the frame extraction interval to obtain frame images. After coarse deduplication using an image hashing algorithm, the similarity of these deduplicated images is calculated using a convolutional neural network. Based on the similarity calculation results, a corresponding regional heatmap is generated. The transparency of the corresponding region is obtained by weighting the corresponding colors of the regional heatmap. The frame images are then fused based on the transparency of the corresponding region.

[0104] After image fusion, the number of people in the fused image is determined. If the number of people in the fused image does not reach the aforementioned alarm threshold, the currently fused image and all images between this fused image are discarded, and the process returns to the step of performing video frame extraction on all devices in the public area according to the frame extraction interval. If the number of people in the fused image reaches the aforementioned alarm threshold, the currently fused image is recorded as the current valid frame, and it is determined whether the total duration of all frame extraction intervals from the current valid frame to the previous historical valid frames is greater than the aforementioned frame extraction interval. If the total duration of all frame extraction intervals from the current valid frame to the previous historical valid frames is less than the aforementioned frame extraction interval, the process returns to the step of performing video frame extraction on all devices in the public area according to the frame extraction interval. If the total duration of all frame extraction intervals from the current valid frame to the previous historical valid frames is greater than or equal to the aforementioned frame extraction interval, an early warning is generated to notify city management personnel, and the method for monitoring the presence of people is terminated.

[0105] like Figure 3 As shown, this embodiment of the invention also provides a personnel stay monitoring device, characterized in that it includes:

[0106] The first acquisition module 301 is used to acquire the current monitoring frame of the monitoring area. The current monitoring frame includes multiple images captured by multiple monitoring devices at the current time. The monitoring area is monitored by the multiple monitoring devices.

[0107] The deduplication module 302 is used to perform coarse deduplication on multiple captured images based on image similarity to obtain multiple images to be fused, wherein the number of images to be fused is less than or equal to the number of captured images;

[0108] The calculation module 303 is used to calculate the transparency of the overlapping areas between the images to be fused, and to perform image fusion on the images to be fused based on the transparency of the overlapping areas to obtain a fused image;

[0109] The detection module 304 is used to detect the number of people in the fused image to obtain the number of people in the fused image;

[0110] The first determining module 305 is used to determine the current monitoring frame as the current valid frame if the number of people in the fused image meets the preset number.

[0111] The second determining module 306 is used to determine whether consecutive valid frames meet preset conditions. The consecutive valid frames include current valid frames and historical valid frames. Two adjacent valid frames in the consecutive valid frames have a preset frame extraction interval.

[0112] The third determining module 307 is used to determine that there are people staying in the monitoring area if the consecutive valid frames meet the preset conditions.

[0113] Optionally, the first acquisition module 301 mentioned above includes:

[0114] The first acquisition submodule is used to acquire the area and the speed of personnel flow in the monitoring area;

[0115] The first determining submodule is used to determine the preset frame extraction interval based on the area of ​​the monitoring area and the speed of personnel flow;

[0116] The second determining submodule is used to determine the current monitoring frame of the monitoring area based on the preset frame extraction interval.

[0117] Optionally, the deduplication module 302 mentioned above includes:

[0118] The first calculation submodule is used to perform hash calculations on the multiple captured images to obtain the hash code of each captured image;

[0119] The second calculation submodule is used to obtain the Hamming distance between each of the captured images based on the hash code of the captured images;

[0120] The third determining submodule is used to compare the Hamming distance between each of the captured images with a preset Hamming distance threshold, determine the captured images that need to be deduplicated among the captured images whose Hamming distance is greater than the preset Hamming distance threshold, and remove the captured images that need to be deduplicated to obtain multiple images to be fused.

[0121] Optionally, the above-mentioned calculation module 303 includes:

[0122] The first extraction submodule is used to extract feature values ​​of the overlapping regions between the images to be fused.

[0123] The third calculation submodule is used to calculate the similarity of the overlapping regions between the images to be fused based on the feature values ​​of the overlapping regions between the images to be fused.

[0124] The fourth determining submodule is used to determine the transparency of the overlapping regions between the images to be fused based on the similarity of the overlapping regions between the images to be fused.

[0125] The first fusion submodule is used to fuse the images to be fused based on the transparency of the overlapping areas between the images to be fused, so as to obtain a fused image.

[0126] Optionally, the above-mentioned device further includes:

[0127] The fourth determining module is used to determine the heatmap distribution of the overlapping regions between the images to be fused based on the similarity of the overlapping regions between the images to be fused.

[0128] The fifth determining module is used to determine the transparency of the overlapping areas between the images to be fused based on the heat map distribution of the overlapping areas between the images to be fused.

[0129] Optionally, the above-mentioned device further includes:

[0130] The sixth determining module is used to determine the thermal proportion of the overlapping areas between the images to be fused based on the thermal map distribution of the overlapping areas between the images to be fused.

[0131] The seventh determining module is used to determine the transparency of the overlapping areas between the images to be fused based on the thermal proportion of the overlapping areas between the images to be fused.

[0132] Optionally, the third determining module 307 mentioned above includes:

[0133] The second acquisition submodule is used to acquire historical valid frames before the current valid frame, wherein there is a preset frame extraction interval between the last frame of the historical valid frames and the current valid frame.

[0134] The fifth determining submodule is used to determine consecutive valid frames based on the historical valid frames and the current valid frames;

[0135] The fourth calculation submodule is used to calculate the number of consecutive valid frames;

[0136] The sixth determining submodule is used to determine the frame extraction duration of consecutive valid frames based on the number of frames.

[0137] The seventh determination submodule is used to determine that there are people staying in the target area when the frame extraction time of the consecutive valid frames is greater than or equal to the preset frame extraction time.

[0138] like Figure 4 As shown, this embodiment of the invention also provides an electronic device, characterized in that it includes a processor, which can execute any of the above-described methods for monitoring personnel stay.

[0139] Specifically, it includes a processor 401 and a memory 402, as well as a computer program for implementing a personnel stay monitoring method stored in the memory 402 and capable of running on the processor 401, wherein:

[0140] The processor 401 executes the calculator program for the personnel stay monitoring method stored in the memory 402, and performs the following steps:

[0141] The current monitoring frame of the monitoring area is obtained. The current monitoring frame includes multiple images captured by multiple monitoring devices at the current time. The monitoring area is monitored by multiple monitoring devices.

[0142] Based on image similarity, coarse deduplication is performed on multiple captured images to obtain multiple images to be fused, wherein the number of images to be fused is less than or equal to the number of captured images;

[0143] Calculate the transparency of the overlapping regions between the images to be fused, and perform image fusion on the images to be fused based on the transparency of the overlapping regions to obtain a fused image;

[0144] The number of people in the fused image is obtained by performing people detection on the fused image;

[0145] If the number of people in the fused image meets the preset number, then the current monitoring frame is determined as the current valid frame;

[0146] Determine whether consecutive valid frames meet preset conditions. The consecutive valid frames include current valid frames and historical valid frames. Two adjacent valid frames in the consecutive valid frames have a preset frame extraction interval.

[0147] If the consecutive valid frames meet the preset conditions, it is determined that there are people staying in the monitoring area.

[0148] Optionally, in the aforementioned method for monitoring personnel presence, the step of processor 401 executing the step of acquiring the current monitoring frame of the monitoring area includes:

[0149] Obtain the area and the speed of personnel flow in the monitored area;

[0150] Based on the area of ​​the monitoring area and the speed of personnel flow, a preset frame extraction interval is determined;

[0151] Based on the preset frame extraction interval, the current monitoring frame of the monitoring area is determined.

[0152] Optionally, in the aforementioned method for monitoring personnel stay, the step of processor 401 performing coarse deduplication processing on multiple captured images based on image similarity to obtain multiple images to be fused includes:

[0153] A hash calculation is performed on the multiple captured images to obtain the hash code of each captured image;

[0154] Based on the hash codes of the captured images, the Hamming distance between each captured image is obtained;

[0155] The Hamming distance between each of the captured images is compared with a preset Hamming distance threshold. Among the captured images whose Hamming distance is greater than the preset Hamming distance threshold, the captured images that need to be deduplicated are identified and removed to obtain multiple images to be fused.

[0156] Optionally, in the aforementioned method for monitoring personnel stay, the step of processor 401 performing the calculation of the transparency of the overlapping regions between the various images to be fused, and performing image fusion on the images to be fused based on the transparency of the overlapping regions to obtain a fused image, includes:

[0157] Extract feature values ​​of the overlapping regions between the images to be fused;

[0158] Based on the feature values ​​of the overlapping regions between the images to be fused, the similarity of the overlapping regions between the images to be fused is calculated.

[0159] The transparency of the overlapping regions between the images to be fused is determined based on the similarity of the overlapping regions between the images to be fused.

[0160] Based on the transparency of the overlapping areas between the images to be fused, the images to be fused are fused to obtain a fused image.

[0161] Optionally, in the aforementioned method for monitoring personnel lingering, the step of determining the transparency of the overlapping regions between the images to be fused based on the similarity of the overlapping regions between the images to be fused by the processor 401 includes:

[0162] Based on the similarity of the overlapping regions between the images to be fused, the heatmap distribution of the overlapping regions between the images to be fused is determined;

[0163] The transparency of the overlapping regions between the images to be fused is determined based on the heatmap distribution of the overlapping regions between the images to be fused.

[0164] Optionally, in the aforementioned method for monitoring personnel lingering, the step of processor 401 determining the transparency of the overlapping regions between the images to be fused based on the heatmap distribution of the overlapping regions between the images to be fused includes:

[0165] Based on the heat map distribution of the overlapping regions between the images to be fused, the thermal proportion of the overlapping regions between the images to be fused is determined.

[0166] The transparency of the overlapping regions between the images to be fused is determined based on the thermal proportion of the overlapping regions between the images to be fused.

[0167] Optionally, in the aforementioned method for monitoring personnel presence, the step of processor 401 determining that personnel are present in the monitoring area if the consecutive valid frames satisfy the preset condition includes:

[0168] Obtain historical valid frames preceding the current valid frame, wherein there is a preset frame extraction interval between the last frame of the historical valid frames and the current valid frame;

[0169] Based on the historical valid frames and the current valid frames, consecutive valid frames are determined;

[0170] Calculate the number of consecutive valid frames;

[0171] Based on the number of frames, determine the frame extraction duration for consecutive valid frames;

[0172] If the frame extraction duration of the consecutive valid frames is greater than or equal to the preset frame extraction duration, it is determined that there are people staying in the target area.

[0173] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the personnel stay monitoring method or the application-side personnel stay monitoring method provided in this invention, and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0174] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0175] The above description discloses only preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A method of monitoring a person's stay, characterized by, The method comprises the following steps: obtaining a current monitoring frame of a monitoring area, the current monitoring frame comprising a plurality of photographed images photographed by a plurality of monitoring devices at a current time, the monitoring area being monitored by the plurality of monitoring devices; performing coarse deduplication processing on the plurality of photographed images based on image similarity to obtain a plurality of to-be-fused images, the number of to-be-fused images being less than or equal to the number of photographed images; calculating the transparency of overlapping areas between each of the to-be-fused images, and performing image fusion on the to-be-fused images based on the transparency of the overlapping areas to obtain a fused image; detecting the number of people in the fused image to obtain the number of people in the fused image; if the number of people in the fused image meets a preset number of people, determining the current monitoring frame as a current valid frame; determining whether a continuous valid frame meets a preset condition, the continuous valid frame comprising a current valid frame and a historical valid frame, adjacent two valid frames in the continuous valid frame having a preset frame interval; if the continuous valid frame meets the preset condition, determining that there is personnel retention in the monitoring area.

2. The method of monitoring personnel presence according to claim 1, wherein, The method comprises the following steps: obtaining the area and the personnel flow speed of the monitoring area; based on the area and the personnel flow speed of the monitoring area, determining a preset frame interval; based on the preset frame interval, determining the current monitoring frame of the monitoring area.

3. The person stay monitoring method according to claim 1 or 2, characterized by, The method comprises the following steps: performing hash calculation on the plurality of photographed images to obtain a hash code of each photographed image; based on the hash code of the photographed image, obtaining the Hamming distance between each photographed image; comparing the Hamming distance between each photographed image with a preset Hamming distance threshold value, determining the photographed images that need to be deduplicated in the photographed images with a Hamming distance greater than the preset Hamming distance threshold value, and removing the photographed images that need to be deduplicated to obtain a plurality of to-be-fused images.

4. The method of monitoring personnel presence according to claim 1, wherein, The method comprises the following steps: extracting feature values of overlapping areas between each of the to-be-fused images; based on the feature values of the overlapping areas between each of the to-be-fused images, calculating the similarity of the overlapping areas between each of the to-be-fused images; based on the similarity of the overlapping areas between each of the to-be-fused images, determining the transparency of the overlapping areas between each of the to-be-fused images; based on the transparency of the overlapping areas between each of the to-be-fused images, fusing the to-be-fused images to obtain a fused image.

5. The method for monitoring the presence of personnel according to claim 4, characterized in that, The method comprises the following steps: based on the similarity of the overlapping areas between each of the to-be-fused images, determining the heat map distribution of the overlapping areas between each of the to-be-fused images; based on the heat map distribution of the overlapping areas between each of the to-be-fused images, determining the transparency of the overlapping areas between each of the to-be-fused images.

6. The method of monitoring personnel presence according to claim 5, wherein, The transparency of the overlapping area between each of the to-be-fused images is determined based on a heat map distribution of the overlapping area between each of the to-be-fused images, and the image fusion is performed on the to-be-fused images based on the transparency of the overlapping area between each of the to-be-fused images. The transparency of the overlapping area between each of the to-be-fused images is determined based on a heat map distribution of the overlapping area between each of the to-be-fused images. The transparency of the overlapping area between each of the to-be-fused images is determined based on a heat map distribution of the overlapping area between each of the to-be-fused images.

7. The method of monitoring personnel presence according to any one of claims 1-6, wherein, If the continuous effective frames meet the preset condition, it is determined that there is personnel retention in the monitoring area. A historical effective frame before a current effective frame is acquired, and the historical effective frame and the current effective frame have a preset frame interval. Based on the historical effective frame and the current effective frame, a continuous effective frame is determined. The number of frames of the continuous effective frame is calculated. Based on the number of frames, the frame extraction duration of the continuous effective frame is determined. If the frame extraction duration of the continuous effective frame is greater than or equal to a preset frame extraction duration, it is determined that there is a personnel retention situation in the target area.

8. A people stay monitoring apparatus characterized by, The personnel retention monitoring device comprises: A first acquisition module is configured to acquire a current monitoring frame of a monitoring area, wherein the current monitoring frame comprises a plurality of shooting images captured by a plurality of monitoring devices at a current time, and the monitoring area is monitored by the plurality of monitoring devices. A de-duplication module is configured to perform coarse de-duplication on the plurality of shooting images based on image similarity to obtain a plurality of to-be-fused images, wherein the number of to-be-fused images is less than or equal to the number of shooting images. A calculation module is configured to calculate the transparency of the overlapping area between each of the to-be-fused images, and perform image fusion on the to-be-fused images based on the transparency of the overlapping area to obtain a fused image. A detection module is configured to perform person number detection on the fused image to obtain the person number of the fused image. A first determination module is configured to determine the current monitoring frame as a current effective frame if the person number of the fused image meets a preset person number. A second determination module is configured to determine whether a continuous effective frame meets a preset condition, wherein the continuous effective frame comprises a current effective frame and a historical effective frame, and adjacent two effective frames in the continuous effective frame have a preset frame interval. A third determination module is configured to determine that there is personnel retention in the monitoring area if the continuous effective frame meets the preset condition.

9. An electronic device, comprising: The memory, the processor, and the computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the personnel retention monitoring method according to any one of claims 1 to 7 when executing the computer program. The computer program is stored on the computer readable storage medium, and the computer program is executable on the processor to implement the steps of the personnel retention monitoring method according to any one of claims 1 to 7.

10. A computer readable storage medium characterized by, ​

Images