Method and system for recognizing dangerous behavior of pedestrians based on dynamic monitoring of unmanned aerial vehicles

Abstract

The application discloses a kind of pedestrian dangerous behavior identification method and system based on unmanned aerial vehicle dynamic monitoring, it is related to image processing technical field.The method comprises: obtaining the multiple-frame pedestrian monitoring image collected by unmanned aerial vehicle;Based on multiple-frame pedestrian monitoring image, the walking track sequence of each pedestrian is generated, and the position of pedestrian in each pedestrian monitoring image is included in walking track sequence;Each walking track sequence is compared, and the collision possibility of each pedestrian in each pedestrian monitoring image is determined;Based on the collision possibility of each pedestrian in each pedestrian monitoring image, the dangerous behavior evaluation value of each pedestrian is determined;Based on dangerous behavior evaluation value, pedestrian monitoring image is adaptively enhanced, so as to identify the dangerous behavior of pedestrian in pedestrian monitoring image.The application can improve the accuracy of dangerous behavior identification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a pedestrian dangerous behavior recognition method and system based on dynamic monitoring of unmanned aerial vehicles. BACKGROUND

[0002] In the current rapid urbanization, pedestrian safety has become a focus in the field of transportation. Pedestrians frequently pass through busy intersections, and the safety risk is high, making pedestrian safety a major concern. Traditional pedestrian safety monitoring methods rely mainly on fixed cameras and ground monitoring equipment, but these devices have obvious limitations and cannot meet the growing demand for pedestrian safety.

[0003] In recent years, unmanned aerial vehicles have emerged in the field of intelligent transportation and safety monitoring due to their unique advantages. They are flexible and mobile, can quickly reach different locations, and have a wide coverage, breaking through the spatial limitations of traditional devices. Currently, by equipping unmanned aerial vehicles with high-definition cameras and intelligent vision systems, pedestrian trajectories can be accurately collected and analyzed in depth. This innovative application opens up a new technical path for identifying dangerous pedestrian behavior and provides protection for pedestrian safety.

[0004] However, during video shooting, the image is prone to shaking due to factors such as flight state, and the resolution is uneven. These problems make the pedestrian area in the image unclear and incomplete, making it difficult to completely obtain the pedestrian motion trajectory, resulting in poor accuracy of dangerous behavior recognition. SUMMARY

[0005] The present application provides a pedestrian dangerous behavior recognition method and system based on dynamic monitoring of unmanned aerial vehicles, which can improve the accuracy of dangerous behavior recognition.

[0006] In a first aspect of the present application, a pedestrian dangerous behavior recognition method based on dynamic monitoring of unmanned aerial vehicles is provided, comprising:

[0007] Obtaining multiple-frame pedestrian monitoring images collected by an unmanned aerial vehicle;

[0008] Based on the multiple-frame pedestrian monitoring images, generating a walking trajectory sequence for each pedestrian, which includes the position of the pedestrian in each pedestrian monitoring image;

[0009] Comparing each walking trajectory sequence to determine the collision probability of each pedestrian in each pedestrian monitoring image;

[0010] Based on the collision probability of each pedestrian in each pedestrian monitoring image, determining the dangerous behavior evaluation value of each pedestrian;

[0011] Based on the dangerous behavior evaluation value, the pedestrian monitoring image is adaptively enhanced to identify the dangerous behavior of the pedestrian in the pedestrian monitoring image.

[0012] Further, the present application also proposes generating a walking trajectory sequence of each pedestrian based on the multi-frame pedestrian monitoring image, comprising:

[0013] The multi-frame pedestrian monitoring image is input into an image segmentation model to obtain a pedestrian area in each pedestrian monitoring image.

[0014] The first position of the target pedestrian in the corresponding pedestrian monitoring image is extracted from each pedestrian area.

[0015] The first positions are sorted in time sequence to construct a walking trajectory sequence of the target pedestrian.

[0016] Further, the present application also proposes extracting the first position of the target pedestrian in the corresponding pedestrian monitoring image from each pedestrian area, comprising:

[0017] In the case that the first pedestrian area does not contain the target pedestrian, the front reference image sequence and the rear reference image sequence are obtained, the front reference image sequence includes each front pedestrian monitoring image between the first pedestrian area and the second pedestrian area, the rear reference image sequence includes each rear pedestrian monitoring image between the first pedestrian area and the third pedestrian area, the second pedestrian area is the front pedestrian area of the first pedestrian area that does not contain the target pedestrian, and the third pedestrian area is the rear pedestrian area of the first pedestrian area that does not contain the target pedestrian.

[0018] Based on the second position of the target pedestrian in each front pedestrian monitoring image, the target walking speed of the target pedestrian in the first pedestrian monitoring image is predicted, and the first pedestrian monitoring image is the pedestrian monitoring image corresponding to the first pedestrian area.

[0019] Based on the second position of the target pedestrian in each front pedestrian monitoring image and the third position of the target pedestrian in each rear pedestrian monitoring image, the target walking direction of the target pedestrian in the first pedestrian monitoring image is predicted.

[0020] Based on the target walking speed and the target walking direction, the target position of the target pedestrian in the first pedestrian monitoring image is determined.

[0021] Further, the present application also proposes predicting the target walking speed of the target pedestrian in the first pedestrian monitoring image based on the second position of the target pedestrian in each front pedestrian monitoring image, comprising:

[0022] Based on the position difference of the target pedestrian in adjacent front pedestrian monitoring images, the front walking speed of the target pedestrian in each front pedestrian monitoring image is determined.

[0023] determine a speed weight of each front pedestrian monitoring image based on a time interval between each front pedestrian monitoring image and the first pedestrian monitoring image;

[0024] determine a target walking speed of the target pedestrian in the first pedestrian monitoring image by using the front side walking speed and the corresponding speed weight.

[0025] Further, the present application also proposes predicting a target walking direction of the target pedestrian in the first pedestrian monitoring image based on a second position of the target pedestrian in each front pedestrian monitoring image and a third position of the target pedestrian in each rear pedestrian monitoring image, comprising:

[0026] performing straight line fitting on the second position of the target pedestrian in each front pedestrian monitoring image to obtain a front side walking fitting straight line, and performing straight line fitting on the third position of the target pedestrian in each rear pedestrian monitoring image to obtain a rear side walking fitting straight line;

[0027] determining a front side walking direction as an included angle between the front side walking fitting straight line and the horizontal direction, and determining a rear side walking direction as an included angle between the rear side walking fitting straight line and the horizontal direction;

[0028] determining the target walking direction of the target pedestrian in the first pedestrian monitoring image based on the front side walking direction and the rear side walking direction.

[0029] Further, the present application also proposes that the walking trajectory sequence further comprises a walking direction and a walking speed of the pedestrian in each row pedestrian monitoring image;

[0030] comparing each walking trajectory sequence to determine a collision possibility of each pedestrian in each pedestrian monitoring image, comprising:

[0031] comparing each walking trajectory sequence to determine a neighboring pedestrian of the target pedestrian in a second pedestrian monitoring image, the second pedestrian monitoring image being any one pedestrian monitoring image;

[0032] determining a first degree of anxious and chaotic running of the target pedestrian based on the walking trajectory sequence of the target pedestrian, and determining a second degree of anxious and chaotic running of the neighboring pedestrian based on the walking trajectory sequence of the neighboring pedestrian;

[0033] determining a relative speed between the target pedestrian and the neighboring pedestrian based on the walking direction and the walking speed of the target pedestrian in the second pedestrian monitoring image, and the walking direction and the walking speed of the neighboring pedestrian in the second pedestrian monitoring image;

[0034] determining the collision possibility of the target pedestrian in the second pedestrian monitoring image by using the first degree of anxious and chaotic running, the second degree of anxious and chaotic running and the relative speed.

[0035] Further, the present application also proposes that the first panic running degree of the target pedestrian is determined based on the walking trajectory sequence of the target pedestrian, comprising:

[0036] The walking acceleration of the target pedestrian in each pedestrian monitoring image is determined based on the walking speed difference between adjacent pedestrian monitoring images in the walking trajectory sequence of the target pedestrian;

[0037] The variance of each walking acceleration is determined as the step frequency panic of the target pedestrian;

[0038] The walking directions of the target pedestrian in each pedestrian monitoring image in the walking trajectory sequence are processed by mean value to obtain the walking mean direction of the target pedestrian;

[0039] The path tortuosity of the target pedestrian is determined by using the walking direction of the target pedestrian in each pedestrian monitoring image and the walking mean direction;

[0040] The first panic running degree of the target pedestrian is determined by using the step frequency panic and the path tortuosity.

[0041] Further, the present application also proposes that the dangerous behavior evaluation value of each pedestrian is determined based on the collision possibility of each pedestrian in each pedestrian monitoring image, comprising:

[0042] The collision time fitting curve is constructed based on the collision possibility of the target pedestrian in each pedestrian monitoring image, and the collision time fitting curve is used to represent the collision possibility of the target pedestrian at each time;

[0043] The high-risk collision points with the collision possibility greater than a preset collision threshold are extracted from the collision time fitting curve;

[0044] The collision danger value of the target pedestrian is determined based on the high-risk collision points;

[0045] The walking trajectory sequence of the target pedestrian is compared with the walking trajectory sequence of other pedestrians except the target pedestrian to obtain the walking trajectory similarity between the target pedestrian and other pedestrians;

[0046] The dangerous behavior evaluation value of the target pedestrian is determined by using the collision danger value and the walking trajectory similarity.

[0047] Further, the present application also proposes that the collision danger value of the target pedestrian is determined based on the high-risk collision points, comprising:

[0048] The instantaneous slope of each high-risk collision point in the collision time fitting curve is processed by mean value to obtain an average slope value;

[0049] The average slope value, the maximum collision possibility in the high-risk collision point, and the number of high-risk collision points are multiplied to obtain a collision danger value of the target pedestrian.

[0050] In a second aspect, the present application provides a pedestrian dangerous behavior identification system based on dynamic monitoring of a UAV, comprising:

[0051] An image acquisition module is configured to acquire multiple frames of pedestrian monitoring images collected by the UAV.

[0052] A trajectory generation module is configured to generate a walking trajectory sequence of each pedestrian based on the multiple frames of pedestrian monitoring images, the walking trajectory sequence including the position of the pedestrian in each frame of pedestrian monitoring image.

[0053] A trajectory comparison module is configured to compare the walking trajectory sequences to determine the collision possibility of each pedestrian in each frame of pedestrian monitoring image.

[0054] A danger evaluation module is configured to determine a dangerous behavior evaluation value of each pedestrian based on the collision possibility of each pedestrian in each frame of pedestrian monitoring image.

[0055] An image enhancement module is configured to adaptively enhance the pedestrian monitoring image based on the dangerous behavior evaluation value, so as to identify the dangerous behavior of the pedestrian in the pedestrian monitoring image.

[0056] The present application has the following advantages:

[0057] In the pedestrian dangerous behavior identification method based on dynamic monitoring of a UAV, multiple frames of pedestrian monitoring images collected by the UAV are acquired first, and the multiple frames of images can make up for the insufficient information of a single frame of image, and to a certain extent, alleviate the information loss caused by image problems. Then, a walking trajectory sequence of each pedestrian is generated, the position of the pedestrian in each frame is determined, the movement trajectory of the pedestrian is presented more completely, and the trajectory interruption problem is overcome. The collision possibility is determined by comparing the walking trajectory sequences, and the potential danger can be captured in advance. The dangerous behavior evaluation value is determined based on the collision possibility, and a quantitative basis is provided for subsequent processing. Finally, the image is adaptively enhanced according to the dangerous behavior evaluation value, the image is optimized in a targeted manner, the features related to the dangerous behavior are highlighted, the image quality is improved, the pedestrian area is clearer, the pedestrian dangerous behavior can be more accurately identified by the system, the deficiencies of traditional methods are effectively solved, and the accuracy of the dangerous behavior identification is improved. BRIEF DESCRIPTION OF DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, and the advantages thereof, the following will briefly introduce the drawings required by the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 A flowchart of a pedestrian dangerous behavior recognition method based on dynamic monitoring of unmanned aerial vehicles provided by an embodiment of the present application is shown in the figure.

[0060] Figure 2 A flowchart of S102 provided by an embodiment of the present application is shown in the figure.

[0061] Figure 3 A flowchart of S202 provided by an embodiment of the present application is shown in the figure.

[0062] Figure 4 A flowchart of S103 provided by an embodiment of the present application is shown in the figure.

[0063] Figure 5 A flowchart of S104 provided by an embodiment of the present application is shown in the figure.

[0064] Figure 6 A structural diagram of a pedestrian dangerous behavior recognition system based on dynamic monitoring of unmanned aerial vehicles provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0065] In order to further illustrate the technical means and effects adopted by the present application to achieve the predetermined purposes, the following describes the pedestrian dangerous behavior recognition method and system based on dynamic monitoring of unmanned aerial vehicles according to the present application, its specific implementation, structure, features and effects in detail in combination with the drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0067] It should be noted that the acquisition, storage, use, processing, etc. of data in the technical solutions of the present application comply with the relevant provisions of laws and regulations.

[0068] It should be noted that in the embodiments of the present application, some software, components, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and the purpose is only to illustrate the feasibility in the implementation of the technical solutions of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0069] In the conventional existing pedestrian safety monitoring system, the fixed camera and the ground monitoring device are limited by static deployment and narrow field of view, and it is difficult to dynamically capture the motion trajectory of a large range of pedestrians. Due to the influence of air flow disturbance and mechanical vibration during the flight of the unmanned aerial vehicle, the video stream collected has inter-frame jitter and resolution fluctuation, resulting in a shift of the pedestrian area coordinates in multiple frames of images. When pedestrians are dense or there are moving obstacles, the pedestrian detection model is prone to region segmentation errors, causing trajectory sequence breakage or coordinate positioning drift. Such data defects directly lead to cumulative errors in the calculation of the collision probability between trajectories, and further affect the input data quality of the dangerous behavior evaluation model.

[0070] For example, in the pedestrian intersection area of a transportation hub, the unmanned aerial vehicle conducts aerial monitoring at a rate of 30 frames per second. When the flight height is reduced to 15 meters, the propeller vibration causes pixel-level displacement of the image, causing the centroid coordinates of the same pedestrian in three consecutive frames of images to shift horizontally by 2-3 pixels. At the same time, two pedestrians moving in opposite directions overlap in the third frame of image, and the target detection algorithm incorrectly merges their bounding boxes, causing the trajectory association in the next five frames of images to be interrupted. At this time, the collision probability predicted based on the broken trajectory will incorrectly determine that the two pedestrians are at a safe distance, while in the actual physical space, the relative speed of the two pedestrians has reached 1.5 meters per second.

[0071] In the face of the above problems, the present application first considers how to eliminate the problem of trajectory breakage caused by image jitter of the unmanned aerial vehicle and pedestrian occlusion. The traditional method directly splices the trajectory based on the single-frame detection result, and when there is a region segmentation error, the trajectory sequence is immediately interrupted. In this regard, the present application attempts to establish a cross-frame association mechanism to predict the position of the pedestrian in the frame where the detection fails through the continuity analysis of the trajectory of the previous and subsequent frames. However, this method requires a large amount of computing resources for trajectory interpolation, making it difficult to meet the real-time monitoring requirements.

[0072] Further research has found that pedestrian motion has the characteristics of speed and direction continuity. The present application explores a trajectory repair method based on kinematic parameters, and infers the reasonable position during the occlusion period through the trend of the walking speed and direction change of adjacent frames. However, relying solely on the kinematic model is prone to sudden direction changes, leading to cumulative prediction errors. In this regard, the present application jointly optimizes the trajectory generation and collision prediction, and uses the spatial constraint relationship between multiple pedestrian trajectories to reversely correct abnormal position points.

[0073] Ultimately, this invention constructs a multi-dimensional trajectory sequence including position, velocity, and direction, and combines it with relative motion analysis between trajectories to dynamically assess collision risk while generating trajectories. When a trajectory break is detected, the potential location of the target pedestrian is deduced in reverse based on the motion state of nearby pedestrians and the calculated collision probability, forming a closed-loop correction mechanism.

[0074] In this regard, such as Figure 1 As shown, this invention proposes a method for identifying dangerous pedestrian behaviors based on UAV dynamic monitoring. This method can be applied to a pedestrian dangerous behavior identification system based on UAV dynamic monitoring, and includes the following steps S101 to S105:

[0075] S101, acquire multiple frames of pedestrian surveillance images collected by the drone;

[0076] S102, Based on multiple frames of pedestrian monitoring images, generate a walking trajectory sequence for each pedestrian, the walking trajectory sequence including the pedestrian's position in each frame of pedestrian monitoring images;

[0077] S103, compare the walking trajectory sequences to determine the collision probability of each pedestrian in each pedestrian monitoring image;

[0078] S104, Based on the collision probability of each pedestrian in each pedestrian monitoring image, determine the dangerous behavior evaluation value of each pedestrian;

[0079] S105, Based on the dangerous behavior evaluation value, adaptive enhancement is performed on the pedestrian monitoring image to enable the identification of dangerous pedestrian behaviors in the pedestrian monitoring image.

[0080] In this embodiment, multi-frame pedestrian monitoring images refer to a sequence of images containing pedestrian movement information continuously collected by a drone. Specifically, this can be achieved by using a high-definition camera mounted on the drone to capture images at fixed time intervals, in order to capture the dynamic changes in the position and posture of pedestrians.

[0081] A pedestrian trajectory sequence refers to spatiotemporal path data generated by analyzing the positional changes of the same pedestrian in multiple frames of images. Specifically, it can be achieved by using an image segmentation model to extract pedestrian regions and track position coordinates, and is used to record the spatial distribution of pedestrians at different times.

[0082] Collision probability refers to the probability of potential collision risk calculated by comparing the motion parameters between different pedestrian trajectories. Specifically, it can be achieved by analyzing relative speed, differences in walking direction, and trajectory intersections, and is used to quantify the degree of danger of interaction between pedestrians.

[0083] The dangerous behavior evaluation value is a risk indicator of pedestrian behavior based on the comprehensive calculation of the collision possibility, which can be realized by using weighted average or a nonlinear function to fuse multiple frames of collision possibility data, and is used for evaluating the dangerous level of pedestrian behavior.

[0084] The adaptive enhancement refers to dynamically adjusting image processing parameters according to the dangerous behavior evaluation value to optimize the recognition effect, which can be realized by using contrast enhancement, deblurring algorithm or regional focusing technology, and is used for improving the visual saliency of dangerous behavior features.

[0085] The core innovation of the present application lies in generating a pedestrian trajectory sequence through multiple images and analyzing the collision possibility, and combining adaptive image enhancement technology to solve the problems of trajectory interruption and low recognition accuracy of dangerous behavior caused by image jitter and occlusion in traditional methods.

[0086] Specifically, the present application evaluates dangerous behavior by analyzing pedestrian trajectories and collision risks. The walking trajectory sequence records the change of pedestrian position over time, which can reflect the motion characteristics of pedestrians. By comparing the walking trajectory sequences of different pedestrians, the relative positions and speeds between them can be calculated, and the collision possibility can be evaluated. Pedestrians with high collision possibility are more likely to have dangerous behavior. The dangerous behavior evaluation value considers the collision risk of pedestrians, which can be used to guide the image enhancement process and highlight high-risk areas.

[0087] The adaptive image enhancement adjusts the enhancement degree according to the dangerous behavior evaluation value, and focuses on processing high-risk areas. This helps to improve the accuracy of dangerous behavior recognition, especially in the case of poor image quality. The whole process forms a closed loop from image acquisition to behavior recognition, through multiple steps to gradually extract and analyze pedestrian motion information, and finally realizes effective recognition of dangerous behavior.

[0088] As an example, first, the unmanned aerial vehicle performs cruising over the designated area, and collects multiple frames of pedestrian monitoring images. The image acquisition frequency is set to 30 frames per second, and the resolution is 1920x1080 pixels.

[0089] Then, the collected images are input into a pedestrian detection algorithm. The algorithm uses a convolutional neural network to process each frame of image, and outputs the bounding box coordinates of each pedestrian. By calculating the center point coordinates of the bounding box, the position of the pedestrian in the image is obtained. The position information of the same pedestrian in consecutive multiple frames of images is arranged in time sequence to generate the walking trajectory sequence of the pedestrian.

[0090] Then, the walking trajectory sequences of different pedestrians are compared with each other. The distance between any two pedestrians in each frame image, and their relative speed and direction are calculated. If the distance between two pedestrians is less than a preset threshold (e.g. 2 meters), and the relative speed is greater than a certain threshold (e.g. 1.5 meters / second), it is considered that there is a collision possibility between the two pedestrians in the frame image.

[0091] Based on the above collision possibility analysis result, a dangerous behavior evaluation value is calculated for each pedestrian. The dangerous behavior evaluation value considers the following factors: the collision possibility frequency with other pedestrians, the severity of the collision possibility (determined by the relative speed and distance), the motion characteristics of the pedestrian itself (such as sudden acceleration or sharp turning), etc. The weighted sum of these factors is obtained to obtain the final dangerous behavior evaluation value.

[0092] Finally, the original image is adaptively enhanced according to the dangerous behavior evaluation value. For the area of the pedestrian with a high dangerous behavior evaluation value, the brightness and contrast of the image are improved, and the edge details are enhanced. At the same time, the position and motion trajectory of the high-risk pedestrian can be marked in the image, and the enhanced pedestrian monitoring image is more conducive to identifying and analyzing dangerous behavior. For example, for the i-th pedestrian region in any one frame pedestrian monitoring image, the dangerous behavior evaluation value of the i-th pedestrian is taken as the sharpening intensity of each pixel point in the i-th pedestrian region; the i-th pedestrian region is sharpened to improve the detail information in the i-th pedestrian region.

[0093] Through the embodiment, a plurality of frame pedestrian monitoring images collected by the unmanned aerial vehicle are first acquired, and the plurality of frame images can make up for the insufficient information of a single frame image, and to a certain extent, alleviate the information loss caused by image problems. Then, the walking trajectory sequence of each pedestrian is generated, the position of the pedestrian in each frame is determined, the motion trajectory of the pedestrian can be more completely presented, and the problem of trajectory interruption is overcome. The collision possibility is determined by comparing the walking trajectory sequences, and the potential danger can be captured in advance. The dangerous behavior evaluation value is determined based on the collision possibility, and a quantitative basis is provided for subsequent processing. Finally, the image is adaptively enhanced according to the dangerous behavior evaluation value, the image is optimized in a targeted manner, the features related to dangerous behavior are highlighted, the image quality is improved, the pedestrian region is clearer, the system can more accurately identify the dangerous behavior of the pedestrian, and the deficiencies of the traditional methods are effectively solved, thereby improving the accuracy of the dangerous behavior identification.

[0094] In some schemes of the present application, when the walking trajectory sequence of each pedestrian is generated, if there is occlusion or image quality fluctuation in the pedestrian region, the position of the target pedestrian may be extracted inaccurately, which affects the integrity and continuity of the trajectory sequence, and finally reduces the precision of the dangerous behavior identification.

[0095] To this end, as Figure 2As shown, the application further proposes that S102 can specifically include S201 to S203:

[0096] S201, inputting each of the multiple frames of pedestrian monitoring images into an image segmentation model to obtain a pedestrian region in each of the pedestrian monitoring images;

[0097] S202, extracting a first position of the target pedestrian in the corresponding pedestrian monitoring image from each of the pedestrian regions;

[0098] S203, sorting each of the first positions in a time sequence to construct a walking trajectory sequence of the target pedestrian.

[0099] In this embodiment, the image segmentation model can adopt an instance segmentation algorithm based on deep learning, such as MaskR-CNN or YOLACT, for segmenting an independent pedestrian region in each frame of pedestrian monitoring image. The extraction of the first position is realized by calculating the geometric center point or the bottom center point of the pedestrian region, ensuring that the position information matches the actual standing point of the pedestrian. The time sorting is arranged by the image acquisition timestamp or the frame number, forming a time-continuous trajectory point sequence.

[0100] Specifically, the image segmentation model performs pixel-level segmentation on each frame of pedestrian monitoring image, and outputs a set of pedestrian regions with pedestrian bounding boxes and masks. For each target pedestrian, the same pedestrian region in different frames is associated through a cross-frame pedestrian re-identification algorithm, ensuring that the walking trajectory sequence corresponds to the same target. The calculation of the first position adopts the bottom center point coordinates of the pedestrian region, which are determined by the horizontal midpoint of the lower boundary and the vertical lower boundary of the bounding box, avoiding the offset of the center point due to the change of the posture of the pedestrian. In the time sorting process, if the target pedestrian is not detected in a certain frame, the missing position data is supplemented through an interpolation algorithm to maintain the continuity of the trajectory sequence. The walking trajectory sequence generated in this way contains accurate position information of the target pedestrian in each frame of image, providing reliable input for subsequent collision possibility calculation.

[0101] As an example, each of the multiple frames of pedestrian monitoring images is input into an image segmentation model to obtain a pedestrian region in each of the pedestrian monitoring images. Specifically, a deep learning semantic segmentation network such as U-Net or DeepLab can be used to perform pixel-level classification on the input pedestrian monitoring image, separating the pedestrian region from the background region in the image.

[0102] Then, a first position of the target pedestrian in the corresponding pedestrian monitoring image is extracted from each of the pedestrian regions. For example, the first position of the pedestrian can be determined by calculating the centroid or the center point coordinates of the bounding box of the pedestrian region.

[0103] Finally, the first positions are sorted in chronological order to construct the walking trajectory sequence of the target pedestrian. In this way, an ordered sequence containing the position information of the target pedestrian at different time points can be obtained.

[0104] Through this embodiment, the position information of pedestrians in consecutive image frames can be accurately extracted, and a complete walking trajectory sequence can be constructed. This provides a reliable data foundation for subsequent pedestrian behavior analysis, which helps to improve the accuracy of dangerous behavior recognition. At the same time, the image segmentation model can effectively deal with the problem of pedestrian detection in complex backgrounds, improving the robustness of trajectory extraction.

[0105] In some of the above schemes of the present application, when generating the walking trajectory sequence of each pedestrian, if the target pedestrian is not detected in the pedestrian area of a certain frame, it will lead to the inability to directly extract the position information of that frame, causing the trajectory sequence to be interrupted, and thus affecting the accurate calculation of the subsequent collision possibility.

[0106] To this end, as shown in the present application, S202 can further include the following S301 to S304: Figure 3

[0107] S301, in the case that the first pedestrian area does not contain the target pedestrian, obtaining a front reference image sequence and a rear reference image sequence, the front reference image sequence including each front pedestrian monitoring image between the first pedestrian area and the second pedestrian area, the rear reference image sequence including each rear pedestrian monitoring image between the first pedestrian area and the third pedestrian area, the second pedestrian area being the front one of the pedestrian area not containing the target pedestrian of the first pedestrian area, and the third pedestrian area being the rear one of the pedestrian area not containing the target pedestrian of the first pedestrian area;

[0108] S302, predicting a target walking speed of the target pedestrian in the first pedestrian monitoring image based on the second position of the target pedestrian in each front pedestrian monitoring image, the first pedestrian monitoring image being the pedestrian monitoring image corresponding to the first pedestrian area;

[0109] S303, predicting a target walking direction of the target pedestrian in the first pedestrian monitoring image based on the second position of the target pedestrian in each front pedestrian monitoring image and the third position of the target pedestrian in each rear pedestrian monitoring image;

[0110] S304, determining the target position of the target pedestrian in the first pedestrian monitoring image based on the target walking speed and the target walking direction.

[0111] ​In the embodiment, the front reference image sequence is composed of the front consecutive frames in which the target pedestrian is not occluded, and the rear reference image sequence is composed of the rear consecutive frames in which the target pedestrian reappears. The target walking speed is calculated by the position difference of the adjacent frames in the front pedestrian monitoring image, and the position change amount is obtained by the weighted average of the time interval. The target walking direction is determined by the position fitting angle of the straight line of the front and rear pedestrian monitoring images. The target position is linearly extrapolated according to the predicted target walking speed and target walking direction to generate the coordinates in the unoccluded frame.

[0112] Specifically, when the target pedestrian is occluded in a frame, the motion trend before being occluded is extracted through the front reference image sequence, and a time decay coefficient is used when calculating the speed weight, so that the weight of the frame closer to the occluded frame is higher. The rear reference image sequence is used to capture the motion trend after the target pedestrian reappears, the straight line before and after is fitted by the least square method, and the weighted average value is taken when calculating the direction angle to balance the motion inertia before and after. The final target position is obtained by vector composition of the speed and direction, and interpolation compensation is performed according to the time span of the occluded frame to ensure the continuity of the trajectory sequence and avoid the trajectory break caused by occlusion, thereby improving the accuracy of subsequent collision possibility analysis.

[0113] As an example, the Kalman filtering algorithm can be used to predict the target walking speed and target walking direction of the target pedestrian in the first pedestrian monitoring image. First, the position information of the target pedestrian in the front reference image sequence is used to establish a motion model to predict the initial state of the target pedestrian in the first pedestrian monitoring image. Then, the position information of the target pedestrian in the rear reference image sequence is used to correct the prediction result to obtain more accurate target walking speed and target walking direction. Finally, based on the predicted target walking speed and target walking direction, the target position coordinates of the target pedestrian in the first pedestrian monitoring image are calculated by extrapolation method.

[0114] As another example, the horizontal axis coordinate of the target pedestrian in the first pedestrian monitoring image can be determined by the following formula 1:

[0115] Formula 1

[0116] In formula 1, is used to represent the horizontal axis coordinate of the i-th pedestrian in the j-th first pedestrian monitoring image, is used to represent the horizontal axis coordinate of the i-th pedestrian in the j-th first pedestrian monitoring image closest to the front pedestrian monitoring image, is used to represent the target walking speed of the i-th pedestrian in the j-th first pedestrian monitoring image, is used to represent the target walking direction of the i-th pedestrian in the j-th first pedestrian monitoring image, A horizontal axis coordinate of the i th pedestrian in a nearest rear pedestrian monitoring image of the j th first pedestrian monitoring image. A time interval between a time corresponding to a nearest rear pedestrian monitoring image of the i th pedestrian in the j th first pedestrian monitoring image and a time corresponding to the j th first pedestrian monitoring image. A time interval between a time corresponding to a nearest rear pedestrian monitoring image of the i th pedestrian in the j th first pedestrian monitoring image and a time corresponding to the j th first pedestrian monitoring image.

[0117] The vertical axis coordinate of the target pedestrian in the first pedestrian monitoring image can be determined by the following formula 2:

[0118] Formula 2

[0119] In formula 2, A vertical axis coordinate of the i th pedestrian in the j th first pedestrian monitoring image, A vertical axis coordinate of the i th pedestrian in a nearest rear pedestrian monitoring image of the j th first pedestrian monitoring image, A target walking speed of the i th pedestrian in the j th first pedestrian monitoring image, A target walking direction of the i th pedestrian in the j th first pedestrian monitoring image, A vertical axis coordinate of the i th pedestrian in a nearest rear pedestrian monitoring image of the j th first pedestrian monitoring image. A time interval between a time corresponding to a nearest rear pedestrian monitoring image of the i th pedestrian in the j th first pedestrian monitoring image and a time corresponding to the j th first pedestrian monitoring image. A time interval between a time corresponding to a nearest rear pedestrian monitoring image of the i th pedestrian in the j th first pedestrian monitoring image and a time corresponding to the j th first pedestrian monitoring image.

[0120] Through the embodiment, the position of the target pedestrian can be accurately predicted when the target pedestrian is blocked or temporarily disappears from the field of view. Thus, the continuity and integrity of the pedestrian trajectory are improved, and more reliable data basis is provided for subsequent dangerous behavior identification. Meanwhile, the method fully utilizes the image information of the front and rear time periods, effectively reduces the positioning error caused by poor quality of a single frame of image, and improves the overall trajectory prediction accuracy.

[0121] In some schemes of the present application, when predicting the target walking speed of the target pedestrian, the influence of the front pedestrian monitoring images of different time intervals on the current speed is different, resulting in low accuracy of the predicted target walking speed.

[0122] To this end, the application further proposes that S302 specifically can include:

[0123] Based on the position difference of the target pedestrian in the adjacent front pedestrian monitoring image, determine the front walking speed of the target pedestrian in each front pedestrian monitoring image;

[0124] Based on the time interval between each front pedestrian monitoring image and the first pedestrian monitoring image, determine the speed weight of each front pedestrian monitoring image;

[0125] Using each front walking speed and the corresponding speed weight, determine the target walking speed of the target pedestrian in the first pedestrian monitoring image.

[0126] In this embodiment, the front walking speed can be obtained by calculating the position coordinate difference of the target pedestrian in two adjacent front pedestrian monitoring images, and dividing by the time interval between the two frames. The speed weight is dynamically adjusted according to the length of the time interval. The shorter the time interval of the front pedestrian monitoring image, the greater the corresponding speed weight. The time interval and the speed weight can be mapped using an exponential decay function. For example, the speed weight can be determined as the result of the inverse of the time interval after normalization processing. The target walking speed can be obtained by weighting and summing each front walking speed and the corresponding speed weight. The weight distribution process uses normalization processing.

[0127] Specifically, when calculating the front walking speed, first obtain the time stamps of two adjacent front pedestrian monitoring images, calculate the time difference as the time interval. Divide the coordinate difference of the target pedestrian in the two images by the time interval to obtain the instantaneous speed in the time interval. For multiple front pedestrian monitoring images, the instantaneous speed between each two adjacent frames is calculated in turn to form a set of front walking speeds.

[0128] In determining the speed weight, we adopt a dynamic weight distribution mechanism based on time interval. Specifically, the time interval between each previous pedestrian monitoring image and the first pedestrian monitoring image is obtained and input into the preset weight calculation logic. Specifically, the sum of the speed weights corresponding to all instantaneous speeds is set to 1, so that the reciprocal of the time interval is normalized to obtain the maximum weight. For example, when there are three previous pedestrian monitoring images, and their time intervals with the first pedestrian monitoring image are 1 second, 2 seconds and 3 seconds, the reciprocals of the time intervals are 1, 0.5 and 1 / 3 respectively. The reciprocals are normalized, i.e. the proportion of the sum of the total reciprocals is calculated, and the total reciprocal sum is 1+0.5+1 / 3=1.833, so the corresponding normalized weights are 1 / 1.833≈0.546, 0.5 / 1.833≈0.273 and 0.333 / 1.833≈0.181 respectively. The target walking speed is the sum of the products of the instantaneous speeds and the corresponding weights. This dynamic weight distribution mechanism can effectively reduce the interference of data with long time intervals on the current speed prediction and improve the accuracy of target position prediction.

[0129] As an example, based on the second position of the target pedestrian in each previous pedestrian monitoring image, the target walking speed of the target pedestrian in the first pedestrian monitoring image is predicted. First, based on the position difference of the target pedestrian in adjacent previous pedestrian monitoring images, the front walking speed of the target pedestrian in each previous pedestrian monitoring image is determined. For example, the instantaneous speed in each frame image can be obtained by calculating the Euclidean distance of the positions of the target pedestrian in two adjacent frames of images and dividing the time interval of the two frames of images.

[0130] Further, based on the time interval between each previous pedestrian monitoring image and the first pedestrian monitoring image, the speed weight of each previous pedestrian monitoring image is determined. Specifically, an exponential decay function can be used to calculate the weight, and the shorter the time interval, the greater the weight, and the longer the time interval, the smaller the weight.

[0131] Finally, the target walking speed of the target pedestrian in the first pedestrian monitoring image is determined by using the front walking speed and the corresponding speed weight. Thus, the speed information in multiple frames of images can be considered comprehensively to obtain a more accurate target walking speed prediction value through weighted summation.

[0132] Specifically, the target walking speed of the target pedestrian in the first pedestrian monitoring image can be determined by the following formula 3:

[0133] Formula 3

[0134] In formula 3, is used to represent the target walking speed of the i-th pedestrian in the j-th first pedestrian monitoring image, a number of front-person monitoring images before the jth first-person monitoring image in which the ith pedestrian appears; a speed weight of the kth front-person monitoring image, which is obtained by normalizing the inverse of the time interval between the jth first-person monitoring image and the kth front-person monitoring image; a front-side walking speed of the ith pedestrian in the kth front-person monitoring image of the jth first-person monitoring image.

[0135] wherein, a target walking speed of the ith pedestrian in the jth first-person monitoring image, which is obtained by normalizing the speed weight of the kth front-person monitoring image, multiplying the normalized speed weight of the kth front-person monitoring image by the corresponding front-side walking speed, and then accumulating the results.

[0136] Through the embodiment, the target walking speed prediction accuracy of the target pedestrian can be effectively improved. By considering the speed information in multiple images and introducing a weight mechanism based on the time interval, the influence of single-image noise can be reduced, and more stable and reliable speed prediction results can be obtained. This helps to more accurately judge the motion state and potential dangerous behavior of the pedestrian, thereby improving the pedestrian safety monitoring capability of the entire system.

[0137] In some schemes of the present application, when the pedestrian area does not contain the target pedestrian, the target walking direction of the target pedestrian needs to be predicted through the front-side reference image sequence and the rear-side reference image sequence. However, in the case of occlusion or trajectory interruption of the pedestrian, relying only on single direction prediction may cause the target walking direction to deviate greatly from the actual motion trajectory, thereby affecting the accuracy of target position prediction and ultimately reducing the reliability of collision possibility judgment.

[0138] To this end, the S303 specifically can include:

[0139] linearly fitting the second positions of the target pedestrian in the front-person monitoring images to obtain a front-side walking fitting straight line, and linearly fitting the third positions of the target pedestrian in the rear-person monitoring images to obtain a rear-side walking fitting straight line;

[0140] determining the angle between the front-side walking fitting straight line and the horizontal direction as the front-side walking direction, and determining the angle between the rear-side walking fitting straight line and the horizontal direction as the rear-side walking direction;

[0141] determining the target walking direction of the target pedestrian in the first-person monitoring image based on the front-side walking direction and the rear-side walking direction.

[0142] In the embodiment, the front-side walking fitting straight line is generated by linear regression of the position coordinates of the target pedestrian in the front-side reference image sequence by the least square method, and the rear-side walking fitting straight line is obtained by processing the position coordinates of the rear-side reference image sequence by the same method. The front-side walking direction is determined by calculating the angle between the front-side walking fitting straight line and the horizontal coordinate axis, and the rear-side walking direction is calculated in the same way. The target walking direction is obtained by weighted summation of the front-side walking direction and the rear-side walking direction, and the weight is dynamically adjusted according to the time span of the front-side reference image sequence and the rear-side reference image sequence.

[0143] Specifically, the target pedestrian coordinate points of three consecutive images in the front-side reference image sequence are selected, and the front-side walking fitting straight line equation is fitted by the least square method, and the angle between the straight line and the horizontal axis is calculated as 35 degrees. The coordinate points of two consecutive images in the rear-side reference image sequence are selected, and the rear-side walking fitting straight line equation is fitted, and the angle between the straight line and the horizontal axis is calculated as 42 degrees. According to the time length of 2 seconds covered by the front-side reference sequence and the time length of 1 second covered by the rear-side reference sequence, the front-side direction weight is set to 0.67, and the rear-side direction weight is set to 0.33. The final target walking direction is calculated as 35*0.67+42*0.33=37.29 degrees. This method effectively eliminates the single-sided prediction error by fusing the motion trend data before and after the occlusion, so that the direction prediction result is closer to the real motion trajectory, and provides an accurate direction reference for subsequent position calculation.

[0144] As an example, the second positions of the target pedestrian in each front-side pedestrian monitoring image and the third positions of the target pedestrian in each rear-side pedestrian monitoring image are extracted. These position information is used to predict the target walking direction of the target pedestrian in the first pedestrian monitoring image.

[0145] Specifically, first, the second positions of the target pedestrian in each front-side pedestrian monitoring image are linearly fitted to obtain a front-side walking fitting straight line. Meanwhile, the third positions of the target pedestrian in each rear-side pedestrian monitoring image are linearly fitted to obtain a rear-side walking fitting straight line.

[0146] Further, the angle between the front-side walking fitting straight line and the horizontal direction is determined as the front-side walking direction, and the angle between the rear-side walking fitting straight line and the horizontal direction is determined as the rear-side walking direction.

[0147] Therefore, based on the front-side walking direction and the rear-side walking direction, the target walking direction of the target pedestrian in the first pedestrian monitoring image is determined. For example, the target walking direction can be obtained by weighted summation of the front-side walking direction and the rear-side walking direction.

[0148] Through this embodiment, the walking direction of the pedestrian in the missing image can be more accurately predicted. By utilizing the position information of the pedestrians on the front and back sides for straight line fitting and combining the walking directions on the front and back sides, the error caused by single direction prediction can be effectively reduced. This method considers the continuity and trend of pedestrian movement, improves the prediction accuracy of the pedestrian movement trajectory, and provides a more reliable data basis for subsequent collision risk assessment.

[0149] In some schemes of the present application, when generating the walking trajectory sequence and comparing the walking trajectory sequence to determine the collision possibility, only the position information of the pedestrian is used for collision prediction, and the influence of the change of the movement state of the pedestrian on the collision risk is not considered, resulting in insufficient accuracy of the collision possibility evaluation result.

[0150] To this end, as shown in Figure 4 the present application further proposes that the walking trajectory sequence further includes the walking direction and the walking speed of the pedestrian in each frame of the pedestrian monitoring image.

[0151] S103 can specifically include the following S401 to S404:

[0152] S401, compare each walking trajectory sequence to determine the adjacent pedestrian of the target pedestrian in the second pedestrian monitoring image, the second pedestrian monitoring image being any one of the pedestrian monitoring images;

[0153] S402, determine the first panic running degree of the target pedestrian based on the walking trajectory sequence of the target pedestrian, and determine the second panic running degree of the adjacent pedestrian based on the walking trajectory sequence of the adjacent pedestrian;

[0154] S403, determine the relative speed between the target pedestrian and the adjacent pedestrian based on the walking direction and the walking speed of the target pedestrian in the second pedestrian monitoring image and the walking direction and the walking speed of the adjacent pedestrian in the second pedestrian monitoring image;

[0155] S404, determine the collision possibility of the target pedestrian in the second pedestrian monitoring image by using the first panic running degree, the second panic running degree, and the relative speed.

[0156] In this embodiment, the determination of the adjacent pedestrian can be achieved by calculating the spatial distance threshold of the target pedestrian and other pedestrians in the same pedestrian monitoring image, and the distance threshold is dynamically adjusted according to the flight height of the unmanned aerial vehicle. The first panic running degree is obtained by weighted calculation of the step frequency panic and the path tortuosity, the step frequency panic is characterized by the step acceleration variance, and the path tortuosity is characterized by the standard deviation of the deviation angle between the walking direction and the mean direction. The relative speed is obtained by calculating the speed vector difference modulus of the target pedestrian and the adjacent pedestrian by the vector composition method.

[0157] Specifically, in the second pedestrian monitoring image, pedestrians with a distance less than three meters from the target pedestrian are screened as proximate pedestrians through coordinate positioning. The walking acceleration variance of the target pedestrian is calculated through the speed change amount in five consecutive images, and the path tortuosity is calculated through the standard deviation of the mean direction from ten historical direction data.

[0158] The collision possibility can be determined by the following formula 4:

[0159] Formula 4

[0160] In formula 4, is used to represent the collision possibility of the i-th pedestrian in the l-th second pedestrian monitoring image, is used to represent the number of proximate pedestrians of the i-th pedestrian in the l-th second pedestrian monitoring image, is used to represent the relative speed of the i-th pedestrian and the d-th proximate pedestrian in the l-th second pedestrian monitoring image, is used to represent the first degree of anxious and chaotic running of the i-th pedestrian, is used to represent the second degree of anxious and chaotic running of the d-th proximate pedestrian of the i-th pedestrian, is used to represent the Euclidean distance between the i-th pedestrian and the d-th proximate pedestrian in the l+1-th second pedestrian monitoring image, and norm is used to represent standardization processing.

[0161] It should be noted that, in order to ensure that the calculation result is meaningful, the present embodiment adds a tuning factor greater than 0 to the denominator is added to prevent the denominator from being 0, and the value of the tuning factor is set by the implementer according to the actual situation, and the present application does not make special restrictions.

[0162] Wherein, when the trajectory feature of the pedestrian is that the speed changes more frequently, the behavior is more anxious, that is, the degree of anxious and chaotic running is greater, and it is easier to be dangerous, that is, the collision possibility is greater; the relative speed between the pedestrian and the proximate pedestrian is greater at the same time, and the distance between the two is smaller at the next moment, and the collision possibility is greater.

[0163] As an example, when comparing each pedestrian trajectory sequence to determine the collision possibility of each pedestrian in each pedestrian monitoring image, first, compare each pedestrian trajectory sequence to determine the proximate pedestrians of the target pedestrian in the second pedestrian monitoring image, and the second pedestrian monitoring image is any pedestrian monitoring image.

[0164] Further, a first degree of anxious and chaotic running of the target pedestrian is determined based on the walking trajectory sequence of the target pedestrian, and a second degree of anxious and chaotic running of the adjacent pedestrian is determined based on the walking trajectory sequence of the adjacent pedestrian. Specifically, the degree of anxious and chaotic running can be calculated by analyzing the position change and speed change of the pedestrian in consecutive frames. For example, the acceleration change rate of the pedestrian in consecutive frames can be calculated, and the greater the acceleration change rate, the higher the degree of anxious and chaotic running.

[0165] Further, based on the walking direction and speed of the target pedestrian in the second pedestrian monitoring image and the walking direction and speed of the adjacent pedestrian in the second pedestrian monitoring image, the relative speed between the target pedestrian and the adjacent pedestrian is determined. Thus, the relative speed can be obtained by calculating the difference between the speed vectors of the two pedestrians.

[0166] Finally, the first degree of anxious and chaotic running, the second degree of anxious and chaotic running, and the relative speed are used to determine the collision probability of the target pedestrian in the second pedestrian monitoring image by the above formula 4.

[0167] Through the embodiment, the collision risk between pedestrians can be more accurately evaluated. By considering the degree of anxious and chaotic running of the pedestrian and the relative motion state, the behavior characteristics and potential dangers of the pedestrian can be more comprehensively analyzed. This method not only can identify pedestrians who may collide, but also can distinguish pedestrians whose collision risk is increased due to anxiety or panic. Therefore, the method can improve the accuracy and reliability of pedestrian dangerous behavior recognition, and provide an important basis for timely taking preventive measures.

[0168] In some of the above schemes of the present application, when determining the degree of anxious and chaotic running of the pedestrian based on the walking trajectory sequence, only a single indicator is used to evaluate the behavior of the pedestrian, which cannot fully reflect the unpredictability of the motion state of the pedestrian, resulting in insufficient accuracy in judging the collision probability, especially in the case of interruption or occlusion of the pedestrian trajectory, it is difficult to effectively identify potential dangerous behavior.

[0169] To this end, the present application further provides that S402 can specifically include:

[0170] Based on the walking speed difference between adjacent pedestrian monitoring images in the walking trajectory sequence of the target pedestrian, the walking acceleration of the target pedestrian in each pedestrian monitoring image is determined;

[0171] The variance of each walking acceleration is determined as the step frequency anxiousness of the target pedestrian;

[0172] The walking directions of the target pedestrian in each pedestrian monitoring image in the walking trajectory sequence are averaged to obtain the average walking direction of the target pedestrian;

[0173] The path tortuosity of the target pedestrian is determined using the walking direction of the target pedestrian in each pedestrian monitoring image and the average walking direction.

[0174] The first focus panic degree of the target pedestrian is determined by using the step frequency focus acuity and the path tortuosity.

[0175] In this embodiment, the walking acceleration is calculated by the time interval and the speed difference between adjacent images, for example, the time interval between adjacent images is 0.1 seconds, and the speed difference is 0.5 meters per second, then the acceleration is 5 meters per second. The step frequency focus acuity quantifies the degree of speed change by the variance of the acceleration sequence, the larger the variance, the more significant the acceleration fluctuation. The walking mean direction is calculated by the arithmetic mean or vector composition of the direction angle at each time, for example, the direction angles of the consecutive three frames are 30°, 45°, and 60°, respectively, and the mean direction is 45°. The path tortuosity is measured by the deviation of the direction at each time from the mean direction, for example, the sine value of the mean of the angle difference is used as the path tortuosity.

[0176] To solve the dimensional mismatch problem between the step frequency focus acuity and the path tortuosity, we can use the normalization method when fusing the two indicators. Specifically, first, normalize the step frequency focus acuity indicator and the path tortuosity indicator respectively, and map their numerical ranges to the same interval, for example, [0, 1].

[0177] Specifically, in the adjacent pedestrian monitoring images, the target pedestrian's walking speed difference is divided by the time interval to get the instantaneous acceleration, and the variance of all accelerations reflects the stability of the pedestrian's step frequency. If the variance exceeds the threshold, for example, 0.8 meters per second 4 , then it is determined that the step frequency focus acuity is high. The walking direction mean is calculated by vector composition, for example, convert each frame direction to a unit vector and take the combined direction. The path tortuosity is calculated by the sine value of the mean of the angle difference. Finally, the normalized step frequency focus acuity and the normalized path tortuosity are fused by weighted summation or multiplication, for example, the weights are 0.6 and 0.4, respectively, to get the first focus panic degree. This indicator combines speed fluctuation and direction deviation, improves the evaluation accuracy of pedestrian behavior unpredictability, and optimizes the collision probability calculation.

[0178] As an example, when determining the first focus panic degree of the target pedestrian based on the target pedestrian's walking trajectory sequence, first, determine the walking acceleration of the target pedestrian in each pedestrian monitoring image based on the walking speed difference of the target pedestrian between adjacent pedestrian monitoring images in the walking trajectory sequence. Specifically, the speed can be obtained by calculating the change in the position of the pedestrian between two adjacent frames of images divided by the time interval, and then the acceleration can be obtained by calculating the difference between the adjacent speeds.

[0179] Further, the variance of each walking acceleration is determined as the step frequency focus acuity of the target pedestrian. Thus, the degree of speed change of the pedestrian can be quantified, reflecting the abnormality of the pedestrian's behavior.

[0180] Further, the walking directions of the target pedestrian in each pedestrian monitoring image in the walking trajectory sequence are averaged to obtain a walking average direction of the target pedestrian. For example, the angle values of the pedestrian movement directions in each frame image can be arithmetically averaged to obtain the overall movement trend.

[0181] Further, the walking directions of the target pedestrian in each pedestrian monitoring image and the walking average direction are used to determine the path tortuosity of the target pedestrian. Specifically, the deviations of the actual directions from the average direction in each frame can be calculated, and then the sine values of the average of the deviations are calculated to obtain the path tortuosity.

[0182] Finally, the step frequency urgency index and the path tortuosity index are normalized, respectively, and the normalized step frequency urgency and the normalized path tortuosity are used to determine the first panic running degree of the target pedestrian. As an optional implementation, the two can be multiplied by a weight to obtain a comprehensive score, and the weight can be adjusted according to the actual application scenario.

[0183] Through the embodiment, the speed change and direction change characteristics of pedestrian movement can be comprehensively considered, and the panic running degree of the pedestrian can be accurately quantified. Thus, abnormal behaviors can be effectively identified, and reliable basis can be provided for subsequent collision risk assessment, thereby improving the accuracy and reliability of pedestrian dangerous behavior identification. Further, the method can adapt to the pedestrian movement characteristics in different scenarios, and has strong universality and robustness.

[0184] In some of the above schemes of the present application, a method for evaluating pedestrian dangerous behavior based on collision probability is proposed. However, when determining the dangerous behavior evaluation value, only the collision probability data of a single frame image is relied on, the dynamic characteristics of the change of collision probability over time are ignored, and the potential influence of the similarity between different pedestrian trajectories on group dangerous behavior is not considered, resulting in insufficient accuracy of dangerous behavior evaluation.

[0185] To this end, as shown in Figure 5 The present application further proposes that S104 specifically can include the following S501 to S505:

[0186] S501, based on the collision probability of the target pedestrian in each pedestrian monitoring image, a collision time fitting curve is constructed, and the collision time fitting curve is used to represent the collision probability of the target pedestrian at each time;

[0187] S502, from the collision time fitting curve, a high-risk collision point with a collision probability greater than a preset collision threshold is extracted;

[0188] S503, based on the high-risk collision point, a collision danger value of the target pedestrian is determined;

[0189] S504, compare the walking trajectory sequence of the target pedestrian with the walking trajectory sequence of other pedestrians except the target pedestrian to obtain walking trajectory similarity of the target pedestrian and other pedestrians;

[0190] S505, determine a dangerous behavior evaluation value of the target pedestrian by using the collision danger value and the walking trajectory similarity.

[0191] In the embodiment, the collision time fitting curve can be constructed by a cubic polynomial fitting algorithm, and the time dimension is the timestamp sequence of the pedestrian monitoring image; the high-risk collision point extraction adopts a threshold detection mechanism, and when the collision possibility is greater than a preset collision threshold, the high-risk collision point is determined; the walking trajectory similarity comparison can adopt a dynamic time warping algorithm.

[0192] Specifically, the construction of the collision time fitting curve adopts a time series analysis method, taking the timestamp of each pedestrian monitoring image as the horizontal coordinate and the collision possibility at the corresponding moment as the vertical coordinate, and a continuous curve is obtained by least square fitting. When a high-risk collision point exceeding a preset threshold is detected in the curve, a quantitative collision danger value is generated by calculating the average slope change of each high-risk collision point, combined with the maximum collision possibility and the occurrence frequency of the high-risk collision point. At the same time, the dynamic time warping algorithm is used to compare the trajectory sequences of the target pedestrian and other pedestrians to obtain the walking trajectory similarity of the target pedestrian and other pedestrians. Finally, the collision danger value is divided by the walking trajectory similarity to generate a comprehensive dangerous behavior evaluation value. This scheme effectively solves the limitations of single-frame evaluation by capturing the time dynamic characteristics of the collision possibility and the group behavior correlation.

[0193] Specifically, the walking trajectory similarity can be determined by the following formula 5:

[0194] Formula 5

[0195] In formula 5, is used to represent the walking trajectory similarity between the i th pedestrian and the a th pedestrian, is used to represent the Euclidean distance between the position coordinates of the i th pedestrian and the a th pedestrian in the l th pedestrian monitoring image, is used to represent the Euclidean distance between the position coordinates of the i th pedestrian and the a th pedestrian in the l+1 th pedestrian monitoring image, is used to represent the total number of pedestrian monitoring images, and exp is used to represent the exponential function operation with the natural constant e as the base.

[0196] Wherein, the greater the Euclidean distance between the position coordinates of the i th pedestrian and the a th pedestrian in each pedestrian monitoring image, the smaller the walking trajectory similarity between the i th pedestrian and the a th pedestrian.

[0197] The dangerous behavior evaluation value can be determined by the following formula 6:

[0198] Formula 6

[0199] In formula 6, a dangerous behavior evaluation value of the i-th pedestrian, a collision danger value of the i-th pedestrian, an average value of the walking track similarity of the i-th pedestrian and other pedestrians.

[0200] As an example, in the process of pedestrian monitoring image sequence processing, the collision possibility data of the target pedestrian is introduced into the three times spline interpolation algorithm to generate a continuous collision time fitting curve. The horizontal axis of the curve corresponds to the normalized timestamp, and the vertical axis maps the collision possibility value. By setting the collision possibility threshold of 0.7 as a screening condition, the high-risk collision points exceeding the threshold in the curve are automatically identified.

[0201] For each high-risk collision point, the average slope value thereof in the collision time fitting curve is calculated, and the maximum collision possibility is recorded. The average slope value, the maximum collision possibility and the number of high-risk collision points are multiplied to obtain the collision danger value. In the walking track similarity calculation link, the walking track similarity of the target pedestrian and other pedestrians is calculated by the above formula 5 to obtain the average value of the walking track similarity. Based on the collision danger value and the average value of the walking track similarity, the dangerous behavior evaluation value of the target pedestrian can be calculated by the above formula 6.

[0202] Through the embodiment, the problem of misjudgment of dangerous behavior caused by interruption of pedestrian track is effectively solved. By constructing a time-continuous collision time fitting curve, the potential risk period in the movement process of the pedestrian can be accurately captured. Combined with the track similarity analysis, the misjudgment caused by group regular movement can be excluded, and the accurate identification of individual abnormal behavior is realized. The double evaluation mechanism significantly improves the reliability of the dangerous behavior detection, and performs excellently especially in the pedestrian dense area and complex motion scene.

[0203] In some schemes of the above-mentioned embodiments of the application, a dangerous behavior evaluation value is determined based on the collision possibility of pedestrians in each pedestrian monitoring image. However, it is difficult to comprehensively reflect the dynamic risk change trend of pedestrians by relying only on the collision possibility of a single image, and the influence of the track similarity between different pedestrians on the dangerous behavior is not considered, resulting in insufficient accuracy of the dangerous behavior evaluation value.

[0204] To this end, the S503 can further include:

[0205] The instantaneous slopes of each high-risk collision point in the collision time fitting curve are averaged to obtain an average slope value;

[0206] The average slope value, the maximum collision possibility value in the high-risk collision point, and the number of high-risk collision points are multiplied to obtain a collision danger value of the target pedestrian.

[0207] In the embodiment, the collision time fitting curve is constructed by using a cubic spline interpolation method, the preset collision threshold is set to 0.7, and the time axis accuracy is 0.5 seconds; when the high-risk collision point is extracted, the corresponding time stamp and the collision possibility value are recorded synchronously.

[0208] Specifically, the collision time fitting curve constructed by using the cubic spline interpolation method can smooth the discrete collision possibility data and accurately reflect the change trend of the pedestrian risk with time. When the collision possibility exceeding the threshold of 0.7 is detected in the fitting curve, the collision possibility point is extracted as the high-risk collision point, and the time stamp and the collision possibility value are recorded. For each high-risk collision point, the first derivative of the position curve where the high-risk collision point is located is calculated as the instantaneous slope, and the arithmetic average of all instantaneous slopes is taken. After the average slope is multiplied by the maximum collision possibility value, the number of high-risk collision points is multiplied, and finally the collision danger value with unified dimensions is obtained.

[0209] As an example, when the target pedestrian appears three high-risk collision points in the continuous pedestrian monitoring image, first, the collision time fitting curve corresponding to each high-risk point is differentiated to obtain the instantaneous slope values of the points, which are 0.45, 0.62 and 0.53 respectively. Then the three instantaneous slope values are summed and divided by three to obtain the average slope value 0.533. Then the maximum collision possibility value 0.92 in the three high-risk collision points is extracted, and the value is multiplied by the average slope value 0.533 and the number of high-risk points three to obtain the collision danger value 0.533*0.92*3=1.471.

[0210] Through the embodiment, the collision risk misjudgment problem caused by incomplete trajectory data in the traditional method is effectively solved. By dynamically capturing the change trend of the collision possibility curve and combining the comprehensive calculation of multi-dimensional parameters, the cumulative effect of the potential collision risk in the pedestrian movement process can be accurately identified. In the case that the pedestrian is temporarily blocked or the resolution of the monitoring image fluctuates, the calculation stability of the collision danger value can be maintained, and the reliability and system anti-interference ability of the dangerous behavior evaluation are significantly improved.

[0211] According to the pedestrian dangerous behavior identification method based on dynamic monitoring of the unmanned aerial vehicle, correspondingly, the present application also provides specific embodiments of the pedestrian dangerous behavior identification system based on dynamic monitoring of the unmanned aerial vehicle.

[0212] As Figure 6As shown, the pedestrian dangerous behavior recognition system 600 based on dynamic monitoring of the unmanned aerial vehicle provided by the embodiment of the application comprises an image acquisition module 610, a trajectory generation module 620, a trajectory comparison module 630, a danger evaluation module 640 and an image enhancement module 650.

[0213] The image acquisition module 610 is used for acquiring multiple-frame pedestrian monitoring images collected by the unmanned aerial vehicle.

[0214] The trajectory generation module 620 is used for generating a walking trajectory sequence of each pedestrian based on the multiple-frame pedestrian monitoring images, wherein the walking trajectory sequence comprises the position of the pedestrian in each pedestrian monitoring image.

[0215] The trajectory comparison module 630 is used for comparing the walking trajectory sequences to determine the collision possibility of each pedestrian in each pedestrian monitoring image.

[0216] The danger evaluation module 640 is used for determining the dangerous behavior evaluation value of each pedestrian based on the collision possibility of each pedestrian in each pedestrian monitoring image.

[0217] The image enhancement module 650 is used for adaptively enhancing the pedestrian monitoring image based on the dangerous behavior evaluation value, so as to identify the dangerous behavior of the pedestrian in the pedestrian monitoring image.

[0218] In the pedestrian dangerous behavior recognition system based on dynamic monitoring of the unmanned aerial vehicle provided by the embodiment of the application, the multiple-frame pedestrian monitoring images collected by the unmanned aerial vehicle are acquired first, the multiple-frame images can make up for the insufficient information of the single-frame image, and the information loss caused by the image problem is relieved to a certain extent. Then, the walking trajectory sequence of each pedestrian is generated, the position of the pedestrian in each frame is determined, the motion trajectory of the pedestrian is presented more completely, and the trajectory interruption problem is overcome. The collision possibility is determined by comparing the walking trajectory sequences, and the potential danger can be captured in advance. The dangerous behavior evaluation value is determined based on the collision possibility, and a quantitative basis is provided for subsequent processing. Finally, the image is adaptively enhanced according to the dangerous behavior evaluation value, the image is optimized in a targeted manner, the features related to the dangerous behavior are highlighted, the image quality is improved, the pedestrian area is clearer, the pedestrian dangerous behavior can be more accurately recognized by the system, the deficiencies of the traditional methods are effectively solved, and the accuracy of the dangerous behavior recognition is improved.

[0219] It should be noted that the present application is not limited to the specific configurations and processes described above and shown in the drawings. For the sake of brevity, detailed descriptions of well-known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order of the steps, after understanding the spirit of the present application.

[0220] It should also be noted that the exemplary embodiments mentioned in the present application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiments, or different from the order in the embodiments, or several steps are performed simultaneously.

[0221] The above merely illustrates the specific implementation of the present application. For the convenience and brevity of description, the specific working processes of the above-described system, module and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be described herein. It should be understood that the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements shall be encompassed within the protection scope of the present application.

Claims

1. A method for identifying dangerous behavior of pedestrians based on dynamic monitoring of unmanned aerial vehicles, characterized in that, The method comprises: acquiring multiple frames of pedestrian monitoring images collected by a UAV; generating a walking trajectory sequence of each pedestrian based on the multiple frames of pedestrian monitoring images, the walking trajectory sequence including a position of the pedestrian in each frame of pedestrian monitoring image; comparing each walking trajectory sequence to determine a collision possibility of each pedestrian in each pedestrian monitoring image; determining a dangerous behavior evaluation value of each pedestrian based on the collision possibility of each pedestrian in each pedestrian monitoring image; performing adaptive enhancement on the pedestrian monitoring images based on the dangerous behavior evaluation value to identify a dangerous behavior of a pedestrian in the pedestrian monitoring images; determining a dangerous behavior evaluation value of each pedestrian based on the collision possibility of each pedestrian in each pedestrian monitoring image comprises: constructing a collision time fitting curve based on the collision possibility of a target pedestrian in each pedestrian monitoring image, the collision time fitting curve being used to represent the collision possibility of the target pedestrian at each time; extracting a high-risk collision point with a collision possibility greater than a preset collision threshold from the collision time fitting curve; determining a collision danger value of the target pedestrian based on the high-risk collision point; comparing the walking trajectory sequence of the target pedestrian with the walking trajectory sequence of other pedestrians except the target pedestrian to obtain walking trajectory similarity between the target pedestrian and the other pedestrians; determining a dangerous behavior evaluation value of the target pedestrian by using the collision danger value and the walking trajectory similarity. 2.The method of claim 1, wherein, The method comprises: inputting the multiple frames of pedestrian monitoring images into an image segmentation model respectively to obtain a pedestrian region in each pedestrian monitoring image; extracting a first position of a target pedestrian in the corresponding pedestrian monitoring image from each pedestrian region; sorting each first position in time sequence to construct a walking trajectory sequence of the target pedestrian. 3.The method of claim 2, wherein, The method comprises: when the first pedestrian region does not contain the target pedestrian, acquiring a front reference image sequence and a rear reference image sequence, the front reference image sequence including each front pedestrian monitoring image between the first pedestrian region and a second pedestrian region, the rear reference image sequence including each rear pedestrian monitoring image between the first pedestrian region and a third pedestrian region, the second pedestrian region being a front pedestrian region of the first pedestrian region that does not contain the target pedestrian, and the third pedestrian region being a rear pedestrian region of the first pedestrian region that does not contain the target pedestrian; predicting a target walking speed of the target pedestrian in a first pedestrian monitoring image based on a second position of the target pedestrian in each front pedestrian monitoring image, the first pedestrian monitoring image being the pedestrian monitoring image corresponding to the first pedestrian region; predict a target walking direction of the target pedestrian in the first pedestrian monitoring image based on the second positions of the target pedestrian in each of the front pedestrian monitoring images and third positions of the target pedestrian in each of the rear pedestrian monitoring images; determine a target position of the target pedestrian in the first pedestrian monitoring image based on the target walking speed and the target walking direction.

4. The method of claim 3, wherein the method further comprises: The prediction of the target walking speed of the target pedestrian in the first pedestrian monitoring image based on the second positions of the target pedestrian in each of the front pedestrian monitoring images comprises: determine front side walking speeds of the target pedestrian in each of the front pedestrian monitoring images based on position differences of the target pedestrian in adjacent front pedestrian monitoring images; determine speed weights of each of the front pedestrian monitoring images based on time intervals between each of the front pedestrian monitoring images and the first pedestrian monitoring image; determine the target walking speed of the target pedestrian in the first pedestrian monitoring image by using each of the front side walking speeds and the corresponding speed weight. 5.The method of claim 3, wherein, The prediction of the target walking direction of the target pedestrian in the first pedestrian monitoring image based on the second positions of the target pedestrian in each of the front pedestrian monitoring images and the third positions of the target pedestrian in each of the rear pedestrian monitoring images comprises: perform straight line fitting on the second positions of the target pedestrian in each of the front pedestrian monitoring images to obtain a front side walking fitting straight line, and perform straight line fitting on the third positions of the target pedestrian in each of the rear pedestrian monitoring images to obtain a rear side walking fitting straight line; determine a front side walking direction as an included angle between the front side walking fitting straight line and a horizontal direction, and determine a rear side walking direction as an included angle between the rear side walking fitting straight line and the horizontal direction; determine the target walking direction of the target pedestrian in the first pedestrian monitoring image based on the front side walking direction and the rear side walking direction. 6.The method of identifying dangerous behavior of pedestrians based on dynamic monitoring of UAVs according to claim 1, characterized in that, The walking trajectory sequence further comprises a walking direction and a walking speed of the pedestrian in each row of pedestrian monitoring images; The comparison between each of the walking trajectory sequences to determine the collision possibility of each pedestrian in each of the pedestrian monitoring images comprises: compare each of the walking trajectory sequences to determine a neighboring pedestrian of a target pedestrian in a second pedestrian monitoring image, the second pedestrian monitoring image being any one of the pedestrian monitoring images; determine a first degree of anxious and chaotic running of the target pedestrian based on the walking trajectory sequence of the target pedestrian, and determine a second degree of anxious and chaotic running of the neighboring pedestrian based on the walking trajectory sequence of the neighboring pedestrian; determine a relative speed between the target pedestrian and the neighboring pedestrian based on a walking direction and a walking speed of the target pedestrian in the second pedestrian monitoring image and a walking direction and a walking speed of the neighboring pedestrian in the second pedestrian monitoring image; determine the collision possibility of the target pedestrian in the second pedestrian monitoring image by using the first degree of anxious and chaotic running, the second degree of anxious and chaotic running, and the relative speed.

7. The method of claim 6, wherein the method further comprises: The first degree of anxiety of the target pedestrian is determined based on the sequence of walking trajectories of the target pedestrian, including: The walking acceleration of the target pedestrian in each frame of the pedestrian monitoring image is determined based on the walking speed difference between adjacent pedestrian monitoring images in the sequence of walking trajectories of the target pedestrian; The variance of each walking acceleration is determined as the step frequency degree of anxiety of the target pedestrian; The walking direction of each pedestrian monitoring image in the sequence of walking trajectories of the target pedestrian is processed by mean value processing to obtain the mean walking direction of the target pedestrian; The path tortuosity of the target pedestrian is determined using the walking direction of each pedestrian monitoring image and the mean walking direction of the target pedestrian; The first degree of anxiety of the target pedestrian is determined using the step frequency degree of anxiety and the path tortuosity. 8.The method of claim 1, wherein, The collision danger value of the target pedestrian is determined based on the high-risk collision points, including: The average slope value is obtained by mean value processing of the instantaneous slope of each high-risk collision point in the collision time fitting curve; The collision danger value of the target pedestrian is obtained by multiplying the average slope value, the maximum collision possibility in the high-risk collision points, and the number of high-risk collision points. 9.A pedestrian dangerous behavior identification system based on dynamic monitoring of a UAV, characterized in that, The system includes: An image acquisition module for acquiring multiple frames of pedestrian monitoring images collected by a UAV; A trajectory generation module for generating a walking trajectory sequence of each pedestrian based on the multiple frames of pedestrian monitoring images, the walking trajectory sequence including the position of the pedestrian in each pedestrian monitoring image; A trajectory comparison module for comparing each walking trajectory sequence to determine the collision possibility of each pedestrian in each pedestrian monitoring image; A danger evaluation module for determining the danger behavior evaluation value of each pedestrian based on the collision possibility of each pedestrian in each pedestrian monitoring image, including: A collision time fitting curve is constructed based on the collision possibility of the target pedestrian in each pedestrian monitoring image, which is used to represent the collision possibility of the target pedestrian at each time; High-risk collision points with a collision possibility greater than a preset collision threshold are extracted from the collision time fitting curve; The collision danger value of the target pedestrian is determined based on the high-risk collision points; The walking trajectory similarity between the target pedestrian and other pedestrians is obtained by comparing the walking trajectory sequence of the target pedestrian with the walking trajectory sequence of the other pedestrians; The danger behavior evaluation value of the target pedestrian is determined using the collision danger value and the walking trajectory similarity; An image enhancement module for adaptively enhancing the pedestrian monitoring image based on the danger behavior evaluation value to identify the dangerous behavior of the pedestrian in the pedestrian monitoring image.

Images