Abnormal behavior monitoring method and system suitable for smart city and program product

Abstract

The invention discloses an abnormal behavior monitoring method and system suitable for a smart city and a program product, and the method comprises the steps: obtaining continuous multi-frame sampling images based on a preset sampling moment, and calculating a corresponding disparity map; performing target segmentation on any frame of calculated disparity map by using a segmentation algorithm, and calculating spatial information of each divided target based on the disparity map, the spatial information at least comprising a target position; based on the spatial information, dividing each target in the continuous multi-frame disparity map in sequence to generate a corresponding target group; for any target group, calculating a moving track based on the target position and a preset sampling moment; and when it is judged that the moving tracks of the at least two target groups conform to the abnormality judgment rule, generating an abnormal behavior alarm. According to the technical scheme, the moving track of each target is directly calculated in the disparity map, image recognition is not needed, and the universality of abnormal behavior detection in the smart city is improved.

Description

Technical Field

[0001] The present application relates to the technical field of smart cities, and in particular to an abnormal behavior monitoring method applicable to smart cities, an abnormal behavior monitoring system applicable to smart cities, and a computer program product. Background Art

[0002] As an emerging urban management method, smart city can be widely used in urban planning and construction, urban security monitoring and other aspects. In the process of realizing smart city, real-time perception is the foundation of building smart city. Therefore, smart light poles came into being, that is, sensing devices are installed on the light poles, so that various types of information in the city can be obtained in real time through the sensing devices on the smart light poles, such as binocular cameras. Among them, in terms of urban security monitoring, it is necessary to analyze pedestrians and vehicles on various sections of the city in real time to determine whether there is abnormal group behavior, such as people fleeing, serious traffic accidents, etc.

[0003] In the prior art, taking abnormal behavior detection of pedestrians as an example, the image features of pedestrians are usually processed using images and neural network technology, such as the Chinese patent application "An Abnormal Behavior Detection Method, System, Device, Medium and Program Product" with publication number CN117994859A.

[0004] This method usually requires the deployment of a network model in the server, which not only requires advance training and labeling of a large number of sample images, increasing the hardware and time costs in the implementation process; but also has the problem of a single detection target and lack of universality, and there is even the possibility of missing or misdetecting some targets. Summary of the invention

[0005] The purpose of this application is to improve the universality of abnormal behavior detection in smart cities and reduce the implementation cost.

[0006] The technical solution of the first aspect of the present application is: a method for abnormal behavior monitoring suitable for smart cities is provided, the method comprising: step 1: based on a preset sampling time, obtaining multiple frames of continuous sampling images and calculating the corresponding disparity map; step 2: using a segmentation algorithm to perform target segmentation on any calculated frame disparity map, and based on the disparity map, calculating the spatial information of each divided target, wherein the spatial information at least includes the target position; step 3: based on the spatial information, dividing each target in the continuous multiple frames of disparity map in turn to generate corresponding target groups; step 4: for any target group, calculating the movement trajectory based on the target position and the preset sampling time; step 5: when it is determined that the movement trajectory of at least two target groups meets the abnormal judgment rule, generating an abnormal behavior alarm.

[0007] In any of the above technical solutions, further, step 3 specifically includes: establishing a spatial coordinate system with a preset origin, and determining the first intermediate coordinates of the optical center coordinates of each binocular camera in the spatial coordinate system; for the targets segmented out of the multi-frame disparity map acquired at the same preset sampling time, based on the corresponding binocular camera internal parameters and the corresponding first intermediate coordinates, coordinate transformation is performed to determine the second intermediate coordinates corresponding to the target position of any target in any frame disparity map in the spatial coordinate system; based on the second intermediate coordinates, each target in the continuous multi-frame disparity map is divided in turn to generate a corresponding target group.

[0008] In any of the above technical solutions, further, the method also includes: labeling the multiple frames of disparity maps obtained at the same preset sampling time, wherein the label is determined by the number of the smart lamp pole equipped with the binocular camera; for the multiple frames of disparity maps obtained at the same preset sampling time, respectively selecting the second intermediate coordinates in the two frames of disparity maps with adjacent labels, and when it is determined that the difference between the two second intermediate coordinates is less than or equal to the filtering threshold, filtering out the target in the disparity map with a larger label and the corresponding second intermediate coordinates; based on the filtered second intermediate coordinates, dividing each target in the continuous multiple frames of disparity maps in turn to generate a corresponding target group.

[0009] In any of the above technical solutions, the method further includes: determining in turn whether the second intermediate coordinates of each target in the continuous multi-frame disparity map with the same label overlap; if so, filtering out the corresponding target, if not, retaining the corresponding target, and based on the second intermediate coordinates of the retained target, dividing each target in the continuous multi-frame disparity map in turn to generate a corresponding target group.

[0010] In any of the above technical solutions, further, the spatial information also includes the target size, and the method also includes: judging in turn whether the target size of each target in the disparity map of each frame belongs to the set screening interval, wherein the minimum value of the screening interval is the pedestrian screening threshold, and the maximum value of the screening interval is the vehicle screening threshold; if not, filtering out the corresponding target, and if so, retaining the corresponding target, and based on the second intermediate coordinates of the retained target, dividing each target in the disparity map of multiple consecutive frames in turn to generate a corresponding target group.

[0011] In any of the above technical solutions, further, the spatial information also includes target size, and each target in the continuous multi-frame disparity map is divided in turn to generate a corresponding target group, which specifically includes: labeling the multi-frame disparity map obtained at the same preset sampling time, wherein the label is determined by the number of the smart lamp pole equipped with a binocular camera; based on the target size of each target in any frame disparity map, determine the target search radius corresponding to each target, and record the target position of each target as the path origin; based on the path origin and the target search radius, search for the target in the disparity map of the next frame with the same label in turn, and record the target position of the searched target as a waypoint to generate a path tree diagram; in the path tree diagram, select the target included in the branch with the longest path and no reuse of waypoints to generate a corresponding target group.

[0012] In any of the above technical solutions, the method further includes: determining the category corresponding to each target based on the target size of each target in the disparity map of any frame; based on the path origin and the target search radius, searching for the targets in the disparity map of the next frame with the same label in sequence, and recording the target positions of the searched targets of the same category as waypoints to generate a path tree diagram.

[0013] In any of the above technical solutions, further, when it is determined that the category of the target is a vehicle, the method also includes: determining a trajectory search direction, wherein an angle between the trajectory search direction and the length direction of the target is less than or equal to a preset angle.

[0014] The technical solution of the second aspect of the present application is: providing an abnormal behavior monitoring system suitable for smart cities, the system comprising: a smart lamp pole, a binocular camera is installed on the smart lamp pole, the binocular camera is configured to obtain and send continuous multiple frames of sampled images and calculate corresponding disparity maps based on a preset sampling time; a domain processor, the domain processor is configured to receive the disparity map, use a segmentation algorithm to perform target segmentation on any calculated frame of disparity map, and calculate the spatial information of each divided target based on the disparity map, wherein the spatial information at least includes the target position; and based on the spatial information, each target in the continuous multiple frames of disparity map is divided in turn, and the corresponding target group is generated and sent; a cloud platform, the cloud platform is configured to receive the target group, and for any target group, calculate the movement trajectory based on the target position and the preset sampling time; and when it is determined that the movement trajectory of at least two target groups meets the abnormality determination rule, an abnormal behavior alarm is generated.

[0015] The technical solution of the third aspect of the present application is: a computer program product is provided, including a computer program, which, when executed by a processor, implements the abnormal behavior monitoring method applicable to a smart city as described in any one of the technical solutions of the first aspect of the present application.

[0016] The beneficial effects of this application are:

[0017] The technical solution in the present application directly performs target segmentation based on the disparity values ​​of each target in the disparity map, and determines the corresponding target position and target size, and then obtains the corresponding movement trajectory after completing the target grouping, without the need for image recognition, which helps to improve the universality of abnormal behavior detection in smart cities and reduce implementation costs.

[0018] In this application, based on the uniqueness of the positions of multiple smart light poles, in order to ensure the consistency of the target positions of each target in this section of urban road, the coordinate system of the segmented targets is unified to improve the accuracy of target grouping and avoid false detection. At the same time, considering that there may be overlapping areas of vision of binocular cameras in multiple smart light poles, as well as the cumulative error in the parallax calculation process, the targets after the unified coordinate system are filtered out to further improve the accuracy of target grouping and avoid false detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The advantages of the above and / or additional aspects of the present application will become apparent and easily understood in the description of the embodiments in conjunction with the following drawings, in which:

[0020] Figure 1 is a schematic block diagram of an abnormal behavior monitoring system applicable to a smart city according to an embodiment of the present application;

[0021] Figure 2 is a schematic flow chart of an abnormal behavior monitoring method applicable to a smart city according to an embodiment of the present application;

[0022] Figure 3 is a partial screenshot of a disparity map in a certain experimental scene according to an embodiment of the present application;

[0023] Figure 4 is a schematic diagram of a depth z calculation principle according to an embodiment of the present application;

[0024] Figure 5 is a schematic diagram of a process of filtering out the second intermediate coordinate according to an embodiment of the present application;

[0025] Figure 6 It is a schematic diagram of a path tree diagram according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0027] In the following description, many specific details are elaborated to facilitate a full understanding of the present application. However, the present application may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present application is not limited to the specific embodiments disclosed below.

[0028] Embodiment 1:

[0029] like Figure 1 As shown, the present embodiment provides an abnormal behavior monitoring system 100 suitable for a smart city, the system 100 comprising: a smart lamp pole 101, a binocular camera is mounted on the smart lamp pole 101, the binocular camera is configured to obtain and send a continuous multi-frame sampling image based on a preset sampling time and calculate a corresponding disparity map; a domain processor 102, the domain processor 102 is configured to receive the disparity map, use a segmentation algorithm to perform target segmentation on any calculated frame disparity map, and calculate spatial information of each divided target based on the disparity map, wherein the spatial information at least includes the target position; and based on the spatial information, each target in the continuous multi-frame disparity map is divided in turn, and a corresponding target group is generated and sent; a cloud platform 103, the cloud platform 103 is configured to receive the target group, and for any target group, calculate the moving trajectory based on the target position and the preset sampling time; and when it is determined that the moving trajectory of at least two target groups meets the abnormal determination rule, an abnormal behavior alarm is generated.

[0030] Specifically, in the process of building a smart city, a smart lamp pole 101 equipped with a binocular camera can be used as a sensing device. After the binocular camera is installed and calibrated, it can output a sampled image within the field of view and a corresponding disparity map, wherein the sampled image includes a left image obtained by a left camera and a right image obtained by a right camera, and the disparity map is calculated from the left image and the right image based on the disparity calculation principle.

[0031] In this system 100, based on the "cloud-edge-end" technical framework, multiple smart lamp poles 101 can be used as "ends", domain processors 102 as "edges", and cloud platforms 103 as "clouds". The smart lamp poles 101 will send the calculated disparity map to the corresponding domain processor 102, and the domain processor 102 will process the disparity map to obtain the corresponding target group. Then the domain processor 102 will send the processing results to the cloud platform 103, and finally the cloud platform 103 will monitor abnormal behavior.

[0032] Since the system is directly based on the disparity map, each target in the disparity map can correspond one-to-one to the target in the left image, and based on the parameters of the binocular camera, the disparity can be converted into depth to obtain the position of each target, and then the corresponding trajectory can be fitted. Therefore, when the trajectory meets the abnormal judgment rules, it can be considered that abnormal behavior has occurred, such as people fleeing, people gathering, serious traffic accidents, etc., and abnormal behavior alarms are issued.

[0033] The system 100 in this embodiment directly monitors abnormal behavior based on the target trajectory. There is no need to identify the target in the image through image recognition and then locate it. That is, there is no need to deploy network models, perform model training, image annotation, etc., which helps to improve the universality of abnormal behavior detection in smart cities and reduce implementation costs.

[0034] It should be noted that the left image captured by the binocular camera in the smart lamp pole 101 can be sent to the cloud platform 103 via the domain processor 102 to realize other functions.

[0035] Embodiment 2:

[0036] like Figure 2 As shown, this embodiment provides an abnormal behavior monitoring method applicable to a smart city, the method comprising:

[0037] Step 1: Based on the preset sampling time, obtain multiple consecutive frames of sampled images and calculate the corresponding disparity map.

[0038] Step 2: Use a segmentation algorithm to perform target segmentation on any calculated disparity map of a frame, and calculate spatial information of each segmented target based on the disparity map, wherein the spatial information at least includes the target position.

[0039] Specifically, Figure 3 As shown, since the disparity map represents the disparity values ​​of each target, and different targets are not adjacent in space, that is, the disparity values ​​are different; and for a single target, its outer contour is continuous in space, that is, the disparity value is the same, therefore, different targets can be represented by different colors, and the same target can be represented by the same color.

[0040] Therefore, based on the continuity of the disparity map, segmentation algorithms (such as clustering and connected domain analysis) can be used to process the disparity data to segment different targets.

[0041] It should be noted that the minimum bounding rectangle of the segmented target may be selected as the target frame of the target, and then the spatial information of the corresponding target may be calculated based on the vertex coordinates of the target frame.

[0042] like Figure 4As shown in the figure, for the disparity value of a certain disparity point, combined with the internal parameters of the binocular camera, such as the baseline distance B, focal length f, optical center coordinates, etc., the depth z between the disparity point and the plane where the binocular camera is located (xoy plane) can be calculated. The corresponding calculation formula is:

[0043]

[0044] Where disp is the disparity value.

[0045] It should be noted that when calculating the depth z between any target and the plane where the binocular camera is located, the disparity at the center point of the target can be selected as the basis for calculating the depth z; or the disparity at multiple points in the target area can be selected to obtain the corresponding distances, and then the average of these depths is calculated, and this average is used as the final depth z.

[0046] After obtaining the depth z, based on the pixel coordinates (u, v) of the target in the left image and the corresponding disparity value, the target position (x w ,y w ,z w ), the corresponding calculation formula is:

[0047]

[0048] Where (u0, v0) is the optical center coordinate of the binocular camera.

[0049] Similarly, using the same method as above, the vertex coordinates of the target box corresponding to the target in the image can be converted to a camera coordinate system with the optical center of the binocular camera as the origin, and then the target size of the target, such as width and length, can be calculated by using methods such as the Euclidean algorithm, where the width corresponds to the x-axis direction and the length corresponds to the y-axis direction.

[0050] Step 3: Based on the spatial information, each target in the disparity map of consecutive frames is divided in turn to generate corresponding target groups.

[0051] Step 4: For any target group, calculate the movement trajectory based on the target position and the preset sampling time.

[0052] Those skilled in the art will appreciate that, for any type of target, its moving speed within a certain period of time in a city can be determined, such as about 15 m / s for vehicles and about 1.5 m / s for pedestrians.

[0053] Therefore, the position of each target in the disparity map of the current frame (x w ,y w ,z w), and a preset step length, which can be associated with the target in the next frame disparity map. The preset step length can be set according to the moving speed.

[0054] In this way, based on this correlation, each target in the continuous multi-frame disparity map can be divided to generate multiple corresponding target groups, wherein each target group corresponds to a target, and the target group at least includes the target position of the target at each preset sampling time.

[0055] For each target group, since the position of the target and the corresponding time are known, the corresponding movement trajectory can be obtained, and the corresponding speed, acceleration, movement direction, etc. can also be obtained, and the trajectory prediction can also be performed based on the above information.

[0056] Step 5: When it is determined that the movement trajectories of at least two target groups meet the abnormality determination rules, an abnormal behavior alarm is generated.

[0057] Specifically, after obtaining the movement trajectory of each target, abnormal behavior monitoring in the city can be performed based on the set abnormality determination rules, wherein the abnormality determination rules can be set based on the actual situation, for example:

[0058] When the trajectories of multiple pedestrians show a broken line change and their speed tends to increase, it can be considered that abnormal behavior of people fleeing has occurred;

[0059] When the trajectories of multiple pedestrians converge and some pedestrians are not moving, it can be considered that abnormal behavior of people gathering has occurred;

[0060] When at least two vehicles overlap and are not moving, it can be considered that an abnormal vehicle collision has occurred;

[0061] When the moving trajectories of multiple vehicles in non-intersection areas have different degrees of curved motion, it can be considered that traffic abnormalities have occurred ahead.

[0062] In any of the above embodiments, further, considering that multiple smart lamp poles are set on both sides of a certain section of urban road, in order to ensure the consistency of the target positions of various targets in the section of urban road, it is also necessary to unify the coordinate system. Therefore, the above step 3 specifically includes:

[0063] A spatial coordinate system is established with a preset origin, and the first intermediate coordinates of the optical center coordinates of each binocular camera in the spatial coordinate system are determined; wherein the preset origin can be set manually, and the optical center coordinates of the binocular camera, the installation position of the binocular camera, and the installation position of the smart lamp pole can all be obtained during the installation and calibration of the binocular camera, which are known information. Therefore, the coordinates of the optical center coordinates of each binocular camera in the spatial coordinate system can be determined by coordinate transformation, and the coordinates are marked as the first intermediate coordinates.

[0064] For the targets segmented in the multiple disparity images obtained at the same preset sampling time, coordinate transformation is performed based on the corresponding binocular camera internal parameters and the corresponding first intermediate coordinates to determine the second intermediate coordinates corresponding to the target position of any target in any disparity image in any frame in the spatial coordinate system;

[0065] Based on the second intermediate coordinates, each target in the disparity map of the continuous multiple frames is divided in turn to generate a corresponding target group.

[0066] Specifically, for any frame disparity map acquired by any binocular camera, in the coordinate transformation process of any target therein, the target is first segmented to obtain its corresponding minimum circumscribed rectangle, and then the disparity of its center point is selected as the basis for calculating the depth of the target, and then combined with the binocular camera internal parameters, the target position in the camera coordinate system with the binocular camera optical center as the origin can be calculated, and finally combined with the first intermediate coordinate corresponding to the binocular camera, the coordinate transformation is performed to obtain the corresponding second intermediate coordinate in the spatial coordinate system, so that the targets acquired by multiple binocular cameras can be converted to a unified coordinate system, corresponding to a unique position coordinate, namely the second intermediate coordinate, so as to complete the final target grouping.

[0067] In any of the above embodiments, further, considering that there are usually multiple smart lamp poles on both sides of the road, and the fields of view of the binocular cameras in each smart lamp pole may overlap, the same target may appear in the fields of view of two adjacent binocular cameras. Therefore, the method further includes:

[0068] The multiple frames of disparity images acquired at the same preset sampling time are numbered, where the number is determined by the number of the smart light pole on which the binocular camera is installed. In simple terms, the disparity images acquired by each binocular camera are numbered in sequence according to the number of the smart light pole. For example, the disparity image acquired by the i-th smart light pole (binocular camera) at time t is numbered i. t , and the parallax icon number obtained by the i+1th smart light pole is (i+1) t ; Correspondingly, at time t+1, the labels of the two disparity images are i and t+1 、(i+1) t+1 .

[0069] For multiple disparity maps obtained at the same preset sampling time, the second intermediate coordinates in two disparity maps with adjacent labels are respectively selected, and when it is determined that the difference between the two second intermediate coordinates is less than or equal to the screening threshold, the target in the disparity map with a larger label and the corresponding second intermediate coordinates are filtered out;

[0070] Specifically, Figure 5As shown in the figure, the binocular cameras of two adjacent smart lamp poles will have overlapping fields of view. When there is a target in this area, it will be presented in two disparity maps, such as target 2. After being transformed into the same spatial coordinate system, there will be errors of varying degrees in the disparity calculation and coordinate calculation at each stage. This will result in the existence of two "targets 2" after the coordinate transformation, namely target 2 and target 2'.

[0071] Since the second intermediate coordinate represents the center position coordinate of a target, and any target should have a certain width, when the difference between the two second intermediate coordinates corresponding to target 2 and target 2' (such as the distance difference) is less than or equal to the screening threshold, it can be artificially determined that target 2 and target 2' should be the same target.

[0072] However, in the process of disparity calculation, "the accuracy of near disparity is higher than that of far disparity," therefore, when the field of view of the binocular camera is tilted forward, the disparity value of "target 2" in disparity map i is more accurate than that in disparity map i+1. Therefore, "target 2'" and the corresponding second intermediate coordinates in the disparity map labeled i+1 are filtered out, and "target 2" and the corresponding second intermediate coordinates in the disparity map labeled i are retained to eliminate the target position offset caused by the accumulated error in the calculation process.

[0073] Based on the filtered second intermediate coordinates, each target in the disparity map of the continuous multiple frames is divided in turn to generate a corresponding target group.

[0074] In any of the above embodiments, further, considering that abnormal behaviors in cities are usually directed at dynamic targets that can move, such as pedestrians and vehicles, and the left image and disparity map captured by the binocular camera will inevitably contain static targets, therefore, before performing target group division based on the spatial information of each segmented target, the static targets can also be filtered out to avoid being divided into the target group, thereby increasing the amount of calculation. Therefore, the method further includes:

[0075] Determine in sequence whether the second intermediate coordinates of each target in the disparity images of consecutive multiple frames with the same label overlap;

[0076] If yes, the corresponding target is filtered out; if no, the corresponding target is retained, and based on the second intermediate coordinates of the retained target, each target in the disparity map of the continuous multiple frames is divided in turn to generate a corresponding target group.

[0077] It should be noted that for vehicles, abnormal behaviors are usually caused by collective lane changes due to traffic accidents; for pedestrians, abnormal behaviors usually include fleeing and gathering. Therefore, during these behaviors, there will be changes in the movement trajectory. If the position of a target remains unchanged in multiple consecutive frames of disparity maps, it can be determined as a static target and then filtered out.

[0078] In any of the above embodiments, further, considering that abnormal behaviors in cities are usually directed at pedestrians and vehicles, other targets, such as buildings, trees, railings, signboards, etc., will inevitably be included in the left image and disparity map captured by the binocular camera. Therefore, before performing target group division based on the spatial information of each segmented target, the above other targets can also be filtered out to avoid being divided into the target group. Therefore, the above spatial information also includes the target size, and the method further includes:

[0079] It is sequentially determined whether the target size of each target in the disparity map of each frame belongs to the set screening interval, wherein the minimum value of the screening interval is the pedestrian screening threshold, and the maximum value of the screening interval is the vehicle screening threshold;

[0080] If not, the corresponding target is filtered out; if so, the corresponding target is retained, and based on the second intermediate coordinates of the retained target, each target in the continuous multi-frame disparity map is divided in turn to generate a corresponding target group.

[0081] Specifically, the target size includes the width and length of the target, which correspond to the x-axis and y-axis sizes of the target in the image, respectively. Therefore, when setting the screening threshold, it is necessary to set the screening thresholds in both directions at the same time to meet the conditions of pedestrians and vehicles.

[0082] It should be noted that two screening intervals may be set separately, so as to target pedestrians and vehicles respectively.

[0083] In any of the above embodiments, further, the spatial information also includes the target size, and each target in the disparity map of the continuous multiple frames is divided in turn to generate the corresponding target group, which specifically includes:

[0084] Labeling multiple frames of disparity images acquired at the same preset sampling time, wherein the labels are determined by the numbers of the smart lamp poles on which the binocular cameras are installed;

[0085] Based on the target size of each target in any frame disparity map, the target search radius corresponding to each target is determined, and the target position of each target is recorded as the path origin; wherein the search radius can be determined according to the moving speed corresponding to the target category.

[0086] Based on the path origin and the target search radius, the target in the disparity map of the next frame with the same label is searched in sequence, and the target position of the searched target is recorded as a waypoint to generate a path tree diagram;

[0087] In the path tree diagram, select the targets included in the branch with the longest path and no reused waypoints to generate the corresponding target group.

[0088] Specifically, Figure 6 As shown, target 1 is set as a pedestrian, target 2 is set as a vehicle, and target 3 is set as a pedestrian. In the process of generating the path tree diagram, targets 1 to 3 will each correspond to a tree diagram, among which the tree diagram of target 2 is omitted. At time t1, target 2 passes near target 1. Since this scheme does not distinguish between target types, the target position of target 2 will be incorporated into the tree diagram corresponding to target 1. When target 2 passes near target 3 at time t2, target 3 will be incorporated into the tree diagram corresponding to target 1. Therefore, from time t2, the target position of target 3 is reused in the path tree diagrams of targets 1 and 3. The right branch corresponding to target 1 is an incorrect branch, and each waypoint in the left branch is selected to generate the target group corresponding to target 1.

[0089] In any of the above embodiments, further, since the target sizes of pedestrians and vehicles are different, in order to improve the search efficiency and ensure the accuracy of the path tree diagram, the method further includes:

[0090] Based on the object size of each object in the disparity map of any frame, determining the category corresponding to each object; wherein the category includes at least pedestrians and vehicles;

[0091] Based on the path origin and the target search radius, the targets in the disparity map of the next frame with the same label are searched in sequence, and the target positions of the searched targets with the same category are recorded as waypoints to generate a path tree diagram.

[0092] It should be noted that different search radii can be set for pedestrians and vehicles, so that there is a smaller search range when generating the pedestrian path tree diagram to avoid searching nearby vehicles.

[0093] In any of the above embodiments, further, considering that when a pedestrian has abnormal behavior, the direction of movement is uncontrollable, but the steering angle of the vehicle is predictable, therefore, when the category of the target is determined to be a vehicle, the method further includes:

[0094] A trajectory search direction is determined, wherein an angle between the trajectory search direction and a length direction of the target is less than or equal to a preset angle, and the preset angle is determined by a steering angle of the vehicle, so as to improve the accuracy of a path tree diagram corresponding to the vehicle.

[0095] Embodiment 3:

[0096] Based on the above embodiment, this embodiment further provides a computer program product, the computer program product includes a processor, a storage medium for storing instructions executable by the processor;

[0097] Among them, the processor is configured to implement the abnormal behavior monitoring method applicable to a smart city as described in any one of the above-mentioned embodiments 1 when executing executable instructions.

[0098] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment method can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned control methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated as: HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0099] So far, various embodiments of the present application have been described in detail. In order to avoid obscuring the concept of the present application, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed herein.

[0100] Although some specific embodiments of the present application have been described in detail through examples, those skilled in the art should understand that the above examples are only for illustration rather than for limiting the scope of the present application.

[0101] The steps in this application can be adjusted in order, combined, and deleted according to actual needs.

[0102] Although the present application is disclosed in detail with reference to the accompanying drawings, it should be understood that these descriptions are merely exemplary and are not intended to limit the application of the present application. The scope of protection of the present application is defined by the appended claims and may include various modifications, alterations and equivalents made to the invention without departing from the scope and spirit of the present application.

Claims

1. A method for monitoring abnormal behavior in smart cities, characterized in that: The method comprises: Step 1: Based on the preset sampling time, obtain multiple consecutive frames of sampled images and calculate the corresponding disparity map; Step 2: using a segmentation algorithm to perform target segmentation on the calculated disparity map of any frame, and calculating spatial information of each segmented target based on the disparity map, wherein the spatial information at least includes the target position; Step 3: Based on the spatial information, each target in the disparity map of consecutive frames is divided in turn to generate a corresponding target group; Step 4: For any of the target groups, based on the target position and the preset sampling time, calculate the movement trajectory; Step 5: When it is determined that the movement trajectories of at least two target groups meet the abnormality determination rules, an abnormal behavior alarm is generated.

2. The abnormal behavior monitoring method applicable to smart cities as claimed in claim 1, characterized in that: The step 3 specifically includes: Establishing a spatial coordinate system with a preset origin, and determining the first intermediate coordinates of the optical center coordinates of each binocular camera in the spatial coordinate system; For the targets segmented out of the disparity maps of multiple frames acquired at the same preset sampling time, coordinate transformation is performed based on the corresponding binocular camera internal parameters and the corresponding first intermediate coordinates to determine the second intermediate coordinates corresponding to the target position of any target in the disparity map of any frame in the spatial coordinate system; Based on the second intermediate coordinates, each target in the disparity map of consecutive frames is divided in turn to generate corresponding target groups.

3. The abnormal behavior monitoring method applicable to smart cities as claimed in claim 2, characterized in that: The method further comprises: Labeling multiple frames of the disparity map acquired at the same preset sampling time, wherein the labels are determined by the numbers of the smart lamp poles on which the binocular cameras are installed; For the multiple disparity maps obtained at the same preset sampling time, respectively select the second intermediate coordinates in the two disparity maps with adjacent labels, and when it is determined that the difference between the two second intermediate coordinates is less than or equal to the screening threshold, filter out the second intermediate coordinates in the disparity map with a larger label; Based on the filtered second intermediate coordinates, each target in the disparity map of consecutive frames is divided in turn to generate a corresponding target group.

4. The abnormal behavior monitoring method applicable to smart cities as claimed in claim 3, characterized in that: The method further comprises: sequentially determining whether the second intermediate coordinates of each target in a plurality of consecutive disparity maps with the same label overlap; If yes, the corresponding target is filtered out; if no, the corresponding target is retained, and based on the second intermediate coordinates of the retained target, each target in the disparity map of consecutive frames is divided in turn to generate a corresponding target group.

5. The abnormal behavior monitoring method applicable to smart cities as claimed in claim 3 or 4, characterized in that: The spatial information also includes the target size, and the method further includes: sequentially determining whether the target size of each target in the disparity map of each frame belongs to a set screening interval, wherein the minimum value of the screening interval is a pedestrian screening threshold, and the maximum value of the screening interval is a vehicle screening threshold; If not, the corresponding target is filtered out; if so, the corresponding target is retained, and based on the second intermediate coordinates of the retained target, each target in the disparity map of consecutive frames is divided in turn to generate a corresponding target group.

6. The abnormal behavior monitoring method applicable to smart cities as claimed in claim 1, characterized in that: The spatial information also includes the target size, and the sequentially dividing the targets in the disparity map of the consecutive frames to generate corresponding target groups specifically includes: Labeling multiple frames of the disparity map acquired at the same preset sampling time, wherein the labels are determined by the numbers of the smart lamp poles on which the binocular cameras are installed; Based on the target size of each target in the disparity map of any frame, determine the target search radius corresponding to each target, and record the target position of each target as the path origin; Based on the path origin and the target search radius, sequentially search for targets in the disparity map of the next frame with the same label, and record the target positions of the searched targets as waypoints to generate a path tree diagram; In the path tree diagram, the targets included in the branch with the longest path and no reused waypoints are selected to generate the corresponding target group.

7. The abnormal behavior monitoring method applicable to smart cities as claimed in claim 6, characterized in that: The method further comprises: Determine the category corresponding to each of the objects based on the object size of each of the objects in the disparity map of any frame; Based on the path origin and the target search radius, the targets in the disparity map of the next frame with the same label are searched in sequence, and the target positions of the searched targets with the same category are recorded as the waypoints to generate the path tree diagram.

8. The abnormal behavior monitoring method applicable to smart cities as claimed in claim 7, characterized in that: When it is determined that the category of the target is a vehicle, the method further includes: A trajectory search direction is determined, wherein an angle between the trajectory search direction and a length direction of the target is less than or equal to a preset angle.

9. An abnormal behavior monitoring system suitable for smart cities, characterized in that: The system comprises: A smart lamp pole, wherein a binocular camera is installed on the smart lamp pole, and the binocular camera is configured to obtain and send a plurality of consecutive frames of sampled images and calculate a corresponding disparity map based on a preset sampling time; A domain processor, the domain processor being configured to receive the disparity map, perform target segmentation on the calculated disparity map of any frame using a segmentation algorithm, and calculate spatial information of each segmented target based on the disparity map, wherein the spatial information at least includes a target position; and Based on the spatial information, each target in the disparity map of consecutive frames is divided in sequence, and a corresponding target group is generated and sent; A cloud platform, the cloud platform being configured to receive the target groups, and for any of the target groups, calculate a movement trajectory based on the target position and the preset sampling time; and When it is determined that the movement trajectories of at least two target groups meet the abnormality determination rule, an abnormal behavior alarm is generated.

10. A computer program product, characterized in that It includes a computer program, which, when executed by a processor, implements the abnormal behavior monitoring method suitable for a smart city as described in any one of claims 1 to 7.

Images