Highway toll station area pedestrian behavior intelligent identification early warning system and method

Abstract

The invention provides an intelligent recognition and early warning system and method for pedestrian behaviors in a highway toll station area, and relates to the technical field of artificial intelligence, and the system comprises a video collection module which is used for collecting station area video frames through video collection equipment disposed at a preset position; the recognition result module is used for determining a pedestrian recognition result and a non-safe area recognition result according to the station area video frame; the video type module is used for determining the video frame type of the station area according to the pedestrian recognition result and the non-safe area recognition result; the behavior data module is used for acquiring behavior characteristic data according to the station area video frame; the danger scoring module is used for determining a pedestrian behavior danger score according to the station area video frame type and the behavior characteristic data; and the early warning determining module is used for determining whether to give out early warning or not according to the pedestrian behavior risk score. According to the invention, the efficiency and accuracy of intelligent recognition and early warning of pedestrian behaviors in the highway toll station area can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence, and in particular, to an intelligent recognition and early warning system and method for pedestrian behavior in a highway toll station area. Background Art

[0002] In the related art, CN110443977A discloses a dynamic early warning method for human behavior. The early warning method includes the steps of: S101, acquiring a dynamic image of human behavior; S102, analyzing the dynamic image and classifying human behavior into normal behavior and abnormal behavior; S103, notifying the abnormal behavior. Correspondingly, the solution also provides a dynamic early warning system for this dynamic early warning method. The dynamic early warning method provided by this solution can achieve fast and accurate dynamic early warning.

[0003] CN109509327B discloses an abnormal behavior early warning method and device. The method includes: obtaining real-time data of a target user; searching a clustering library to obtain each target clustering center point corresponding to the target user; performing an outlier detection on the real-time data based on each target clustering center point. If an outlier is detected, generating an early warning message based on the outlier; determining the target abnormal behavior category to which the outlier belongs according to the distance between the outlier and each target clustering center point; searching a preset first relationship table to obtain a target early warning processing terminal corresponding to the target user; searching a preset second relationship table to obtain a target early warning plan corresponding to the target abnormal behavior category; and sending the early warning message and the target early warning plan to the target early warning processing terminal, so that a target early warning processing person holding the target early warning processing terminal performs early warning processing according to the target early warning plan. By applying the embodiment of this solution, early warning processing for abnormal behavior is realized in a timely manner, and social security is improved.

[0004] Based on the above related technologies, fast and accurate dynamic early warning can be achieved. However, the related technologies do not consider the influence of the environment where the human body is located on the judgment of the danger of human behavior, that is, the danger of behavior cannot be judged according to the environmental conditions and human behavior.

[0005] The information disclosed in the background art section of the present application is only intended to deepen the understanding of the general background art of the present application, and should not be regarded as an admission or any form of suggestion that this information constitutes the prior art known to those skilled in the art. Summary of the Invention

[0006] The present invention provides an intelligent recognition and early warning system and method for pedestrian behavior in a highway toll station area, which can solve the technical problem that the related technologies cannot judge the danger of behavior according to the environmental conditions and human behavior.

[0007] According to a first aspect of the present invention, there is provided an intelligent recognition and early warning system for pedestrian behavior in a highway toll station area, including:

[0008] A video acquisition module, configured to acquire station area video frames at multiple moments in multiple toll station area regions through video acquisition devices arranged at preset positions;

[0009] An identification result module, configured to determine a pedestrian identification result and a non-safe area identification result according to the station area video frames;

[0010] A video type module, configured to determine the type of the station area video frames according to the pedestrian identification result and the non-safe area identification result;

[0011] A behavior data module, configured to obtain behavior feature data according to the station area video frames, wherein the behavior feature data includes: pedestrian posture, pedestrian speed, pedestrian acceleration, and pedestrian movement trajectory;

[0012] A danger scoring module, configured to determine a pedestrian behavior danger score according to the type of the station area video frames and the behavior feature data;

[0013] A determination and early warning module, configured to determine whether to issue an early warning according to the pedestrian behavior danger score.

[0014] According to a second aspect of the present invention, there is provided an intelligent recognition and early warning method for pedestrian behavior in a highway toll station area, including:

[0015] Acquire station area video frames at multiple moments in multiple toll station area regions through video acquisition devices arranged at preset positions;

[0016] Determine a pedestrian identification result and a non-safe area identification result according to the station area video frames;

[0017] Determine the type of the station area video frames according to the pedestrian identification result and the non-safe area identification result;

[0018] Obtain behavior feature data according to the station area video frames, wherein the behavior feature data includes: pedestrian posture, pedestrian speed, pedestrian acceleration, and pedestrian movement trajectory;

[0019] Determine a pedestrian behavior danger score according to the type of the station area video frames and the behavior feature data;

[0020] Determine whether to issue an early warning according to the pedestrian behavior danger score.

[0021] Technical effects: According to the present invention, the station area video frames can be recognized based on the image detection model to determine the pedestrian recognition result and the non-safe area recognition result, and based on the pedestrian recognition result and the non-safe area recognition result, the station area video frames can be classified. Further, according to the video frame classification result and the behavioral feature data of the pedestrians extracted from the video frames, the overall behavioral risk of the pedestrians in the video frames is evaluated, which improves the efficiency and accuracy of the intelligent recognition and early warning of pedestrian behaviors in the highway toll station area. When determining the type of the station area video frames, the type of the station area video frames can be determined according to the pedestrian recognition result and the non-safe area recognition result. In the calculation process, the station area video frames can be classified according to two aspects: whether there are pedestrians in the station area video frames and whether the area corresponding to the station area video frames is safe, which is convenient for subsequent judgment on whether it is necessary to give early warning to the behaviors of the pedestrians in the station area video frames. When determining the pedestrian behavioral risk score, the pedestrian behavioral risk score of the station area video frame at the i-th moment at the k-th video acquisition device can be determined according to the second non-safe area recognition result, the pedestrian speed, and the pedestrian acceleration. In the calculation process, the pedestrian behavioral risk score can be evaluated according to the speed and acceleration of the pedestrians and whether the predicted pedestrian movement area is safe, which improves the comprehensiveness and accuracy of the pedestrian behavioral risk score. When determining the pedestrian behavioral risk score, the pedestrian behavioral risk score of the station area video frame at the i-th moment at the k-th video acquisition device can be determined according to the pedestrian posture, the pedestrian speed, and the pedestrian acceleration of the station area video frame at the i-th moment at the k-th video acquisition device. In the calculation process, according to the behavioral risks of each pedestrian in the station area video frame, the most serious situation of the overall pedestrian risk of the video frame can be determined, which improves the comprehensiveness and accuracy of the pedestrian behavioral risk score.

[0022] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and do not limit the present invention. According to the following detailed description of the exemplary embodiments with reference to the accompanying drawings, other features and aspects of the present invention will be clearer. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other embodiments can be obtained based on these drawings without creative efforts.

[0024] Figure 1 Exemplarily shows a block diagram of an intelligent recognition and early warning system for pedestrian behaviors in a highway toll station area according to an embodiment of the present invention;

[0025] Figure 2A schematic flowchart of a method for intelligent recognition and early warning of pedestrian behavior in a highway toll station area according to an embodiment of the present invention is exemplarily shown. Detailed implementation manners

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are only a part rather than all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0027] The technical solutions of the present invention will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0028] Figure 1 A block diagram of a system for intelligent recognition and early warning of pedestrian behavior in a highway toll station area according to an embodiment of the present invention is exemplarily shown. The system includes:

[0029] A video acquisition module, configured to acquire station area video frames at multiple moments of multiple toll station area regions through a video acquisition device arranged at a preset position;

[0030] An identification result module, configured to determine a pedestrian identification result and a non-safe area identification result according to the station area video frames;

[0031] A video type module, configured to determine the type of the station area video frames according to the pedestrian identification result and the non-safe area identification result;

[0032] A behavior data module, configured to obtain behavior feature data according to the station area video frames, where the behavior feature data includes: pedestrian posture, pedestrian speed, pedestrian acceleration, and pedestrian movement trajectory;

[0033] A danger scoring module, configured to determine a pedestrian behavior danger score according to the type of the station area video frames and the behavior feature data;

[0034] A determination and early warning module, configured to determine whether to issue an early warning according to the pedestrian behavior danger score.

[0035] The intelligent recognition and early warning system for pedestrian behavior in highway toll station areas according to an embodiment of the present invention can identify the station area video frames according to an image detection model, determine the pedestrian recognition result and the non-safe area recognition result, and classify the station area video frames based on the pedestrian recognition result and the non-safe area recognition result. Further, according to the video frame classification result and the behavioral feature data of the pedestrians extracted from the video frames, the overall behavioral risk of the pedestrians in the video frames is evaluated, which improves the efficiency and accuracy of the intelligent recognition and early warning of pedestrian behavior in highway toll station areas.

[0036] According to an embodiment of the present invention, in the video acquisition module, it is used to acquire the station area video frames at multiple moments of multiple toll station areas through video acquisition devices arranged at preset positions.

[0037] For example, high-definition cameras are arranged at multiple preset positions (such as, toll service areas, toll plazas, toll station buildings, and toll station affiliated areas), and the high-definition cameras continuously acquire the station area videos in the station area. The shooting area corresponding to each high-definition camera remains fixed. The time interval between two adjacent moments is greater than the duration of 3 consecutive frames. For example, if the frame rate of the station area video acquired by the current high-definition camera is 10 FPS, then the duration of a video frame is 0.1 s, and the time interval between two adjacent moments is greater than 0.3 s. The time interval between two adjacent moments can be 0.5 s, 1 s, etc.

[0038] According to an embodiment of the present invention, in the recognition result module, it is used to determine the pedestrian recognition result and the non-safe area recognition result according to the station area video frames.

[0039] According to an embodiment of the present invention, determining the pedestrian recognition result and the non-safe area recognition result according to the station area video frames includes:

[0040] In the station area video frame, identify whether the area corresponding to the station area video frame is a non-safe area through an image detection model to determine the non-safe area recognition result;

[0041] In the station area video frame, identify whether there are pedestrians in the station area video frame through an image detection model to determine the pedestrian recognition result.

[0042] For example, an image detection model belongs to a type of deep learning model. The image detection model is trained with historical data so that it can identify information in video frames. By using the image detection model to identify the station area video frames, it is determined whether the area in the station area video frame is a non-safe area (such as the fast lane in front of the toll station, the random parking area in the toll station square, and the toll station exit). If the area in the station area video frame is a non-safe area, the non-safe area recognition result corresponding to the station area video frame is 1; otherwise, the non-safe area recognition result is 0. By using the image detection model to identify the station area video frames, it is determined whether there are pedestrians in the station area video frame. If there are pedestrians in the station area video frame, the pedestrian recognition result corresponding to the station area video frame is 1; otherwise, the pedestrian recognition result is 0.

[0043] According to an embodiment of the present invention, in the video type module, it is used to determine the type of the station area video frame according to the pedestrian recognition result and the non-safe area recognition result.

[0044] According to an embodiment of the present invention, determining the type of the station area video frame according to the pedestrian recognition result and the non-safe area recognition result includes: determining the recognition result Vft of the type of the station area video frame at the i-th moment at the k-th video acquisition device according to formula (1) k,i ,

[0045] Vft k,i = if{Ped k,i = 0, 1, if{(Nsa k = 1) and (Ped k,i = 1), 3, 2}}(1)

[0046] where if is a conditional function, and is a logical operator for "and", Nsa k,i is the non-safe area recognition result at the i-th moment at the k-th video acquisition device, Ped k,i is the pedestrian recognition result at the i-th moment at the k-th video acquisition device,

[0047] When the recognition result of the type of the station area video frame at the i-th moment at the k-th video acquisition device is 1, the type of the station area video frame at the i-th moment at the k-th video acquisition device is the first type of video frame;

[0048] When the recognition result of the type of the station area video frame at the i-th moment at the k-th video acquisition device is 2, the type of the station area video frame at the i-th moment at the k-th video acquisition device is the second type of video frame;

[0049] When the recognition result of the station area video frame type at the \(i\)-th moment at the \(k\)-th video acquisition device is 3, the station area video frame type at the \(i\)-th moment at the \(k\)-th video acquisition device is the third type of video frame.

[0050] According to an embodiment of the present invention, in formula (1), the value of the conditional function if{Ped k,i =0, 1, if{(Nsa k =1) and (Ped k,i =1), 3, 2}} includes the following two cases. When the condition Ped k,i =0 is satisfied, it means that there is no pedestrian in the station area video frame at the \(i\)-th moment at the \(k\)-th video acquisition device, and the value of this conditional function is 1. When the condition Ped k,i =0 is not satisfied, the value of this conditional function is the value of the inner conditional function if{(Nsa k =1) and (Ped k,i =1), 3, 2}.

[0051] According to an embodiment of the present invention, the value of the inner conditional function if{(Nsa k =1) and (Ped k,i =1), 3, 2} includes the following two cases. When the condition (Nsa k =1) and (Ped k,i =1) is satisfied, it means that there is a pedestrian in the station area video frame at the \(i\)-th moment at the \(k\)-th video acquisition device and the area corresponding to this video frame is a non-safe area, and the value of this inner conditional function is 3. When the condition (Nsa k =1) and (Ped k,i =1) is not satisfied, there is a pedestrian in the station area video frame at the \(i\)-th moment at the \(k\)-th video acquisition device, and the area corresponding to this video frame is a safe area, and the value of this inner conditional function is 2.

[0052] According to an embodiment of the present invention, when the recognition result of the station area video frame type at the i-th moment at the k-th video acquisition device is 1, it indicates that there is no pedestrian in the station area video frame at the i-th moment at the k-th video acquisition device, and there is no need to determine whether to give a warning about the pedestrian behavior in this station area video frame. It is determined that this station area video frame is a first type of video frame; when the recognition result of the station area video frame type at the i-th moment at the k-th video acquisition device is 2, there is a pedestrian in the station area video frame at the i-th moment at the k-th video acquisition device, and the area corresponding to this station area video frame is a safe area. It is determined that this station area video frame is a second type of video frame; when the recognition result of the station area video frame type at the i-th moment at the k-th video acquisition device is 3, it indicates that there is a pedestrian in the station area video frame at the i-th moment at the k-th video acquisition device and the area corresponding to this station area video frame is a non-safe area. It is determined that this station area video frame is a third type of video frame.

[0053] In this way, the type of the station area video frame can be determined according to the pedestrian recognition result and the non-safe area recognition result. During the calculation process, the station area video frame can be classified according to two aspects: whether there is a pedestrian in the station area video frame and whether the area corresponding to the station area video frame is safe, which is convenient for subsequent judgment of whether to give a warning about the behavior of the pedestrian in the station area video frame.

[0054] According to an embodiment of the present invention, in the behavior data module, it is used to obtain behavior feature data according to the station area video frame, where the behavior feature data includes: pedestrian posture, pedestrian speed, pedestrian acceleration, and pedestrian movement trajectory.

[0055] For example, analyze the station area video frame through a deep learning model (such as a convolutional neural network), and identify the positions of the joint points of the pedestrian (such as the head, shoulders, knees, etc.), so as to infer the pedestrian posture. Analyze the station area video frame at the i-th moment and multiple consecutive video frames between the i-th moment and the (i - 1)-th moment through a target detection or tracking algorithm to determine the pedestrian speed, pedestrian acceleration, and pedestrian movement trajectory of the pedestrian in the station area video frame at the i-th moment. For example, identify the positions of the pedestrian in the consecutive video frames, and based on the change in the position of the pedestrian and the time interval between the consecutive video frames, determine the speed of the pedestrian between adjacent video frames. Average the speeds of the pedestrian between multiple adjacent video frames to obtain the pedestrian speed. Further, based on the change in the speeds of the pedestrian between multiple adjacent video frames, determine the pedestrian acceleration.

[0056] According to an embodiment of the present invention, in the danger scoring module, it is used to determine the danger score of the pedestrian behavior according to the station area video frame type and the behavior feature data.

[0057] According to an embodiment of the present invention, determining the pedestrian behavior risk score based on the station area video frame type and the behavior feature data includes:

[0058] When the station area video frame type at the i-th moment at the k-th video acquisition device is the first type of video frame, the pedestrian behavior risk score of the station area video frame at the i-th moment at the k-th video acquisition device is 0;

[0059] When the station area video frame type at the i-th moment at the k-th video acquisition device is the second type of video frame, determine the pedestrian behavior risk score of the station area video frame at the i-th moment at the k-th video acquisition device according to the pedestrian speed, pedestrian acceleration, and pedestrian movement trajectory of the station area video frame at the i-th moment at the k-th video acquisition device;

[0060] When the station area video frame type at the i-th moment at the k-th video acquisition device is the third type of video frame, determine the pedestrian behavior risk score of the station area video frame at the i-th moment at the k-th video acquisition device according to the pedestrian posture, pedestrian speed, and pedestrian acceleration of the station area video frame at the i-th moment at the k-th video acquisition device.

[0061] For example, when the station area video frame type at the i-th moment at the k-th video acquisition device is the first type of video frame and there is no pedestrian in the station area video frame, the corresponding pedestrian behavior risk score is 0; when the station area video frame type at the i-th moment at the k-th video acquisition device is the second type of video frame, indicating that there is a pedestrian in the station area video frame and the area corresponding to the station area video frame is a safe area, then evaluate the overall behavior risk according to the pedestrian speed, pedestrian acceleration, and pedestrian movement trajectory of the pedestrian in the station area video frame; when the station area video frame type at the i-th moment at the k-th video acquisition device is the third type of video frame, indicating that there is a pedestrian in the station area video frame and the area corresponding to the station area video frame is a non-safe area, then evaluate the overall behavior risk according to the pedestrian posture, pedestrian speed, and pedestrian acceleration of the pedestrian in the station area video frame.

[0062] According to an embodiment of the present invention, when the station area video frame type at the i-th moment at the k-th video acquisition device is the second type of video frame, determining the pedestrian behavior risk score of the station area video frame at the i-th moment at the k-th video acquisition device according to the pedestrian speed, pedestrian acceleration, and pedestrian movement trajectory of the station area video frame at the i-th moment at the k-th video acquisition device includes:

[0063] Determine the pedestrian movement direction according to the pedestrian movement trajectory;

[0064] Determine the toll station area corresponding to the k-th video acquisition device;

[0065] Determine a predicted pedestrian movement area according to the toll station area corresponding to the k-th video acquisition device and the pedestrian movement direction.

[0066] Determine a second non-safe area recognition result according to the predicted pedestrian movement area.

[0067] Determine the pedestrian behavior risk score of the station area video frame at the i-th moment at the k-th video acquisition device according to the second non-safe area recognition result, the pedestrian speed, and the pedestrian acceleration.

[0068] For example, determine the pedestrian movement direction according to the pedestrian movement trajectory (for example, the pedestrian movement direction is moving to the left side of the video screen); determine the specific area where the k-th video acquisition device is installed at the toll station (for example, the toll station platform, the toll plaza); obtain the layout plan of the toll station, and determine the predicted pedestrian movement area according to the layout plan of the toll station, the toll station area corresponding to the k-th video acquisition device, and the pedestrian movement direction; determine the second non-safe area recognition result according to the predicted pedestrian movement area. When the predicted pedestrian movement area is a non-safe area, the second non-safe area recognition result is 2, and when the second non-safe area recognition result is a safe area, the second non-safe area recognition result is 1; evaluate the risk of the pedestrian's action according to the second non-safe area recognition result, the pedestrian speed, and the pedestrian acceleration, and determine the pedestrian behavior risk score of the station area video frame at the i-th moment at the k-th video acquisition device.

[0069] According to an embodiment of the present invention, determining the pedestrian behavior risk score of the station area video frame at the i-th moment at the k-th video acquisition device according to the second non-safe area recognition result, the pedestrian speed, and the pedestrian acceleration includes: determining the pedestrian behavior risk score Pbd of the station area video frame at the i-th moment at the k-th video acquisition device according to formula (2) k,i ,

[0070]

[0071] where Nsa k,i,j,2 is the second non-safe area recognition result of the predicted pedestrian movement area of the j-th pedestrian in the station area video frame at the i-th moment at the k-th video acquisition device, PV k,i,j is the pedestrian speed of the j-th pedestrian in the station area video frame at the i-th moment at the k-th video acquisition device, V T is a preset speed threshold, Pa k,i,j is the pedestrian acceleration of the j-th pedestrian in the station area video frame at the i-th moment at the k-th video acquisition device, a Tis the preset acceleration threshold, max is the function of taking the maximum value, J is the number of pedestrians in the station area video frame at the i-th moment at the k-th video acquisition device, j ≤ J, and both j and J are positive integers.

[0072] According to an embodiment of the present invention, is the relative difference between the pedestrian speed of the j-th pedestrian in the station area video frame at the i-th moment at the k-th video acquisition device and the preset speed threshold. The larger this ratio, the relatively greater the pedestrian speed of the j-th pedestrian in the station area video frame at the i-th moment at the k-th video acquisition device, the greater the possibility that the j-th pedestrian in the station area video frame at the i-th moment at the k-th video acquisition device is running or playing, and the greater the behavioral risk of the j-th pedestrian. is the relative difference between the pedestrian acceleration of the j-th pedestrian in the station area video frame at the i-th moment at the k-th video acquisition device and the preset acceleration threshold. The larger this ratio, the greater the possibility that the j-th pedestrian in the station area video frame at the i-th moment at the k-th video acquisition device is running or playing, and the greater the behavioral risk of the j-th pedestrian. represents the behavioral risk determined according to the speed and acceleration of the j-th pedestrian. represents the weighting according to the second non-safe area recognition result of the predicted pedestrian movement area of the j-th pedestrian in the station area video frame at the i-th moment at the k-th video acquisition device. When the predicted pedestrian movement area of the j-th pedestrian is a non-safe area, Nsa k,i,j,2 = 2, the j-th pedestrian may be running towards the non-safe area of the toll station area, and the behavioral risk is relatively greater. When the predicted pedestrian movement area of the j-th pedestrian is a non-safe area, Nsa k,i,j,2 = 1, and the behavioral risk of the j-th pedestrian remains relatively unchanged. is to take the maximum value of the behavioral risks of J pedestrians in the station area video frame at the i-th moment at the k-th video acquisition device, which can be used to determine whether a pedestrian performs a dangerous behavior at the i-th moment in this area and determine the overall behavioral risk of the pedestrians in the station area video frame at the i-th moment at the k-th video acquisition device.

[0073] In this way, the pedestrian behavior risk score of the station area video frame at the i-th moment at the k-th video acquisition device can be determined according to the second non-safe area recognition result, pedestrian speed and pedestrian acceleration. During the calculation process, the pedestrian behavior risk score can be evaluated according to the speed, acceleration of the pedestrian and whether the predicted pedestrian movement area is safe, improving the comprehensiveness and accuracy of the pedestrian behavior risk score.

[0074] According to an embodiment of the present invention, when the station area video frame type at the i-th moment at the k-th video acquisition device is the third type of video frame, the pedestrian behavior risk score of the station area video frame at the i-th moment at the k-th video acquisition device is determined according to the pedestrian posture, pedestrian speed, and pedestrian acceleration of the station area video frame at the i-th moment at the k-th video acquisition device, including:

[0075] Determine the pedestrian posture feature vector according to the pedestrian posture;

[0076] Obtain a plurality of preset dangerous posture feature vectors;

[0077] Determine the pedestrian behavior risk score Pbd of the station area video frame at the i-th moment at the k-th video acquisition device according to formula (3) k,i ,

[0078]

[0079] where β j is a preset weight, max is the maximum value function, PV k,i,j is the pedestrian speed of the j-th pedestrian in the station area video frame at the i-th moment at the k-th video acquisition device, V T is the preset speed threshold, Pa k,i,j is the pedestrian acceleration of the j-th pedestrian in the station area video frame at the i-th moment at the k-th video acquisition device, a T is the preset acceleration threshold, Z k,i,j is the pedestrian posture feature vector of the j-th pedestrian in the station area video frame at the i-th moment at the k-th video acquisition device, Z k,i,j T is the transposed vector of Z k,i,j is Z T,e is the e-th preset dangerous posture feature vector, E is the number of preset dangerous posture feature vectors, e ≤ E, J is the number of pedestrians in the station area video frame at the i-th moment at the k-th video acquisition device, j ≤ J, and e, E, j, and J are all positive integers.

[0080] For example, represent the pedestrian posture in the form of a rotation matrix, and convert the rotation matrix into a rotation vector through the Rodriguez formula, that is, the pedestrian posture vector; obtain the dangerous posture feature vectors of a plurality of preset dangerous postures (climbing over the guardrail, running).

[0081] According to an embodiment of the present invention, is the relative difference between the pedestrian speed of the j-th pedestrian in the station area video frame at the i-th moment at the k-th video acquisition device and the preset speed threshold. The larger this ratio, the relatively greater the pedestrian speed of the j-th pedestrian in the station area video frame at the i-th moment at the k-th video acquisition device, the greater the likelihood that the j-th pedestrian in the station area video frame at the i-th moment at the k-th video acquisition device is running or playing, and the greater the behavioral risk of the j-th pedestrian. is the relative difference between the pedestrian acceleration of the j-th pedestrian in the station area video frame at the i-th moment at the k-th video acquisition device and the preset acceleration threshold. The larger this ratio, the greater the likelihood that the j-th pedestrian in the station area video frame at the i-th moment at the k-th video acquisition device is running or playing, and the greater the behavioral risk of the j-th pedestrian. is the cosine similarity between the pedestrian pose feature vector of the j-th pedestrian in the station area video frame at the i-th moment at the k-th video acquisition device and the e-th preset dangerous pose feature vector. The larger this cosine similarity, the closer the pedestrian pose of the j-th pedestrian in the station area video frame at the i-th moment at the k-th video acquisition device is to the e-th preset dangerous pose, and the greater the behavioral risk of the j-th pedestrian. is to take the maximum value of the cosine similarities between the pedestrian pose feature vector of the j-th pedestrian in the station area video frame at the i-th moment at the k-th video acquisition device and E preset dangerous pose feature vectors. The above processing of taking the maximum value can be used to determine whether the pedestrian pose of the j-th pedestrian is a preset dangerous pose. represents the behavioral risk of the j-th pedestrian determined based on speed, acceleration, and pose.

[0082] According to an embodiment of the present invention,

[0083] represents taking the maximum value of the sum of the products of the behavioral risks of J pedestrians and their corresponding weights. For example, when the value corresponding to the 1st pedestrian is 0.5, the value corresponding to the 2nd pedestrian is 0.4, the value corresponding to the 2nd pedestrian is 0.3, the preset weights are β1, β2, and β3 respectively, and β1 < β2 < β3, then

[0084]

[0085] The value is 0.5β3 + 0.4β2 + 0.3β1, indicating that when there are more pedestrians with high behavioral risk in the station area video frame at the i-th moment at the k-th video acquisition device, the overall pedestrian risk of the station area video frame at the i-th moment at this video acquisition device becomes more serious as the number of pedestrians with high behavioral risk increases. Among them, the preset weight β j changes in a form combining an exponential function and a linear function. The part of the change of β j in the form of a linear function indicates that the increase in the behavioral risk of a single pedestrian causes the overall pedestrian risk of the station area video frame at the i-th moment at the k-th video acquisition device to increase uniformly. The part of the change of β j in the form of an exponential function indicates that as the number of pedestrians with dangerous behaviors in the station area video frame at the i-th moment at the k-th video acquisition device increases, the overall pedestrian risk of the station area video frame at the i-th moment at the k-th video acquisition device increases rapidly and non-uniformly. The above method of finding the maximum value using the allocated weight values can be used to determine the situation where the overall pedestrian risk of the station area video frame is the most serious.

[0086] In this way, based on the pedestrian posture, pedestrian speed, and pedestrian acceleration of the station area video frame at the i-th moment at the k-th video acquisition device, the pedestrian behavior risk score of the station area video frame at the i-th moment at the k-th video acquisition device can be determined. During the calculation process, based on the behavioral risks of each pedestrian in the station area video frame, the situation where the overall pedestrian risk of the video frame is the most serious can be determined, improving the comprehensiveness and accuracy of the pedestrian behavior risk score.

[0087] According to an embodiment of the present invention, in the determination and warning module, it is used to determine whether to issue a warning according to the pedestrian behavior risk score.

[0088] For example, when the pedestrian behavior risk score of the station area video frame at the i-th moment at the k-th video acquisition device is greater than the set pedestrian behavior risk score threshold, the staff will be immediately reminded that there may be danger in the area corresponding to the k-th video acquisition device.

[0089] The intelligent recognition and early warning system for pedestrian behavior in the highway toll station area according to an embodiment of the present invention can identify the station area video frames according to an image detection model, determine the pedestrian recognition result and the non-safe area recognition result, and classify the station area video frames based on the pedestrian recognition result and the non-safe area recognition result. Further, according to the video frame classification result and the behavioral feature data of the pedestrians extracted from the video frames, the overall behavioral risk of the pedestrians in the video frames is evaluated, which improves the efficiency and accuracy of the intelligent recognition and early warning of pedestrian behavior in the highway toll station area. When determining the type of the station area video frames, the type of the station area video frames can be determined according to the pedestrian recognition result and the non-safe area recognition result. During the calculation process, the station area video frames can be classified according to two aspects: whether there are pedestrians in the station area video frames and whether the area corresponding to the station area video frames is safe, which is convenient for subsequent judgment of whether it is necessary to give early warning to the behavior of the pedestrians in the station area video frames. When determining the pedestrian behavior risk score, the pedestrian behavior risk score of the station area video frame at the i-th moment at the k-th video acquisition device can be determined according to the second non-safe area recognition result, the pedestrian speed and the pedestrian acceleration. During the calculation process, the pedestrian behavior risk score can be evaluated according to the speed, acceleration of the pedestrians and whether the predicted pedestrian movement area is safe, which improves the comprehensiveness and accuracy of the pedestrian behavior risk score. When determining the pedestrian behavior risk score, the pedestrian behavior risk score of the station area video frame at the i-th moment at the k-th video acquisition device can be determined according to the pedestrian posture, the pedestrian speed and the pedestrian acceleration of the station area video frame at the i-th moment at the k-th video acquisition device. During the calculation process, according to the behavioral risks of each pedestrian in the station area video frame, the most serious situation of the overall pedestrian risk of the video frame can be determined, which improves the comprehensiveness and accuracy of the pedestrian behavior risk score.

[0090] Figure 2 Exemplarily shown is a schematic flow chart of a method for intelligent recognition and early warning of pedestrian behavior in a highway toll station area according to an embodiment of the present invention. The method includes:

[0091] Step S101, collecting station area video frames at multiple moments in multiple toll station area regions through video acquisition devices arranged at preset positions;

[0092] Step S102, determining a pedestrian recognition result and a non-safe area recognition result according to the station area video frames;

[0093] Step S103, determining the type of the station area video frames according to the pedestrian recognition result and the non-safe area recognition result;

[0094] Step S104, obtaining behavioral feature data according to the station area video frames, where the behavioral feature data includes: pedestrian posture, pedestrian speed, pedestrian acceleration, and pedestrian movement trajectory;

[0095] Step S105: Determine the pedestrian behavior risk score according to the station area video frame type and the behavior feature data.

[0096] Step S106: Determine whether to issue a warning according to the pedestrian behavior risk score.

[0097] The present invention may be a method, an apparatus, a system, and / or a computer program product. The computer program product may include a computer-readable storage medium having thereon computer-readable program instructions for performing various aspects of the present invention.

[0098] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the drawings are only examples and do not limit the present invention. The object of the present invention has been fully and effectively achieved. The functions and structural principles of the present invention have been demonstrated and explained in the embodiments, and the embodiments of the present invention may have any deformation or modification without departing from the principle.

Claims

1. An intelligent recognition and early warning system for pedestrian behavior in a highway toll station area, characterized in that, Including: A video acquisition module, configured to acquire station area video frames at multiple moments in multiple toll station areas through a video acquisition device arranged at a preset position; An identification result module, configured to determine a pedestrian identification result and a non-safe area identification result based on the station area video frames; A video type module, configured to determine the type of the station area video frames based on the pedestrian identification result and the non-safe area identification result; A behavior data module, configured to obtain behavior feature data based on the station area video frames, where the behavior feature data includes: pedestrian posture, pedestrian speed, pedestrian acceleration, and pedestrian movement trajectory; A danger score module, configured to determine a pedestrian behavior danger score based on the type of the station area video frames and the behavior feature data; A determination warning module, configured to determine whether to issue a warning based on the pedestrian behavior danger score.

2. The intelligent recognition and early warning system for pedestrian behavior in highway toll station areas according to claim 1, characterized in that, Determining the pedestrian identification result and the non-safe area identification result based on the station area video frames includes: In the station area video frames, identifying whether the corresponding area of the station area video frames is a non-safe area through an image detection model to determine the non-safe area identification result; In the station area video frames, identifying whether there are pedestrians in the station area video frames through an image detection model to determine the pedestrian identification result.

3. The intelligent recognition and early warning system for pedestrian behavior in highway toll station areas according to claim 2, characterized in that, Determining the type of the station area video frames based on the pedestrian identification result and the non-safe area identification result includes: According to the formula Vft k,i = if{Ped k,i = 0, 1, if{(Nsa k = 1) and (Ped k,i = 1), 3, 2}} Determine the station area video frame type recognition result Vft at the i-th moment of the k-th video acquisition device k,i , where if is a conditional function, and is the logical operator for "and", Nsa k,i is the non-safe area recognition result at the i-th moment of the k-th video acquisition device, Ped k,i is the pedestrian recognition result at the i-th moment of the k-th video acquisition device When the type identification result of the station area video frame at the i-th moment at the k-th video acquisition device is 1, the type of the station area video frame at the i-th moment at the k-th video acquisition device is the first type of video frame; When the type identification result of the station area video frame at the i-th moment at the k-th video acquisition device is 2, the type of the station area video frame at the i-th moment at the k-th video acquisition device is the second type of video frame; When the type identification result of the station area video frame at the i-th moment at the k-th video acquisition device is 3, the type of the station area video frame at the i-th moment at the k-th video acquisition device is the third type of video frame.

4. The intelligent recognition and early warning system for pedestrian behavior in highway toll station areas according to claim 1, characterized in that, Determining the pedestrian behavior danger score based on the type of the station area video frames and the behavior feature data includes: When the type of the station area video frame at the i-th moment at the k-th video acquisition device is the first type of video frame, the pedestrian behavior danger score of the station area video frame at the i-th moment at the k-th video acquisition device is 0; When the type of the station area video frame at the i-th moment at the k-th video acquisition device is the second type of video frame, determining the pedestrian behavior danger score of the station area video frame at the i-th moment at the k-th video acquisition device according to the pedestrian speed, pedestrian acceleration, and pedestrian movement trajectory of the station area video frame at the i-th moment at the k-th video acquisition device; When the type of the station area video frame at the i-th moment at the k-th video acquisition device is the third type of video frame, determining the pedestrian behavior danger score of the station area video frame at the i-th moment at the k-th video acquisition device according to the pedestrian posture, pedestrian speed, and pedestrian acceleration of the station area video frame at the i-th moment at the k-th video acquisition device.

5. The intelligent recognition and early warning system for pedestrian behavior in highway toll station areas according to claim 4, characterized in that, When the station area video frame type at the \(i\)-th moment at the \(k\)-th video acquisition device is a second type of video frame, determine the pedestrian behavior risk score of the station area video frame at the \(i\)-th moment at the \(k\)-th video acquisition device according to the pedestrian speed, pedestrian acceleration, and pedestrian movement trajectory of the station area video frame at the \(i\)-th moment at the \(k\)-th video acquisition device, including: Determine the pedestrian movement direction according to the pedestrian movement trajectory; Determine the toll station area corresponding to the \(k\)-th video acquisition device; Determine the predicted pedestrian movement area according to the toll station area corresponding to the \(k\)-th video acquisition device and the pedestrian movement direction; Determine the second non-safe area recognition result according to the predicted pedestrian movement area; Determine the pedestrian behavior risk score of the station area video frame at the \(i\)-th moment at the \(k\)-th video acquisition device according to the second non-safe area recognition result, the pedestrian speed, and the pedestrian acceleration.

6. The intelligent recognition and early warning system for pedestrian behavior in highway toll station area according to claim 5, characterized in that, Determine the pedestrian behavior risk score of the station area video frame at the \(i\)-th moment at the \(k\)-th video acquisition device according to the second non-safe area recognition result, the pedestrian speed, and the pedestrian acceleration, including: According to the formula Determine the pedestrian behavior danger score Pbd of the station area video frame at the i-th moment at the k-th video acquisition device k,i , where Nsa k,i,j,2 is the second non-safe area recognition result of the predicted pedestrian movement area of the j-th pedestrian in the station area video frame at the i-th moment at the k-th video acquisition device PV k,i,j is the pedestrian speed of the j-th pedestrian in the station area video frame at the i-th moment at the k-th video acquisition device, V + is the preset speed threshold, Pa k,i,j is the pedestrian acceleration of the j-th pedestrian in the station area video frame at the i-th moment at the k-th video acquisition device, a + is the preset acceleration threshold, max is the function of taking the maximum value, J is the number of pedestrians in the station area video frame at the i-th moment at the k-th video acquisition device, j ≤ J, both j and J are positive integers.

7. The intelligent recognition and early warning system for pedestrian behavior in highway toll station areas according to claim 4, characterized in that, When the station area video frame type at the \(i\)-th moment at the \(k\)-th video acquisition device is a third type of video frame, determine the pedestrian behavior risk score of the station area video frame at the \(i\)-th moment at the \(k\)-th video acquisition device according to the pedestrian posture, pedestrian speed, and pedestrian acceleration of the station area video frame at the \(i\)-th moment at the \(k\)-th video acquisition device, including: Determine the pedestrian posture feature vector according to the pedestrian posture; Obtain multiple preset dangerous posture feature vectors; According to the formula Determine the pedestrian behavior danger score Pbd of the station area video frame at the i-th moment at the k-th video acquisition device k,i , where β j is a preset weight, max is the maximum value function, PV k,i,j is the pedestrian speed of the j-th pedestrian in the station area video frame at the i-th moment at the k-th video acquisition device, V + is a preset speed threshold, Pa k,i,j is the pedestrian acceleration of the j-th pedestrian in the station area video frame at the i-th moment at the k-th video acquisition device, a + is a preset acceleration threshold, Z k,i,j is the pedestrian pose feature vector of the j-th pedestrian in the station area video frame at the i-th moment at the k-th video acquisition device, Z k,i,j + is the transpose vector of Z k,i,j of Z +,e is the e-th preset dangerous pose feature vector, E is the number of preset dangerous pose feature vectors, e ≤ E, J is the number of pedestrians in the station area video frame at the i-th moment at the k-th video acquisition device, j ≤ J, and e, E, j, and J are all positive integers.

8. An intelligent recognition and early warning method for pedestrian behavior in a highway toll station area, characterized in that, Including: Collect station area video frames at multiple moments in multiple toll station areas through video acquisition devices set at preset positions; Determine the pedestrian recognition result and the non-safe area recognition result according to the station area video frames; Determine the station area video frame type according to the pedestrian recognition result and the non-safe area recognition result; Obtain behavior feature data according to the station area video frames, where the behavior feature data includes: pedestrian posture, pedestrian speed, pedestrian acceleration, and pedestrian movement trajectory; Determine the pedestrian behavior risk score according to the station area video frame type and the behavior feature data; Determine whether to issue a warning according to the pedestrian behavior risk score.

Images