Highway toll station non-motor vehicle and pedestrian on-highway intelligent early warning system

Abstract

The invention relates to the technical field of road early warning, in particular to a highway toll station non-motor vehicle and pedestrian on-highway intelligent early warning system, which is characterized in that a toll station image associated road is divided into a plurality of subspaces according to a topological connection sequence, and non-motor vehicles and pedestrians in each subspace are regarded as target objects; moving tracks of the target objects in the subspaces are monitored, an intention label of the moving track in each subspace is deduced according to the track range of the target objects in the subspaces, and the deduced intention labels and the moving tracks are jointly combined into a track map of each target object; judging a trajectory vector of any target object under trajectory fitting, guiding the fitted trajectory vector by a spatial position, and setting an intention vector of each target object; performing cross identification on the intention vectors of the target objects, and obtaining an output early warning result according to the average error of the interaction points of the intention vectors of the target objects and the moving tracks; the accuracy and efficiency of pedestrian intention recognition are realized.

Description

Technical Field

[0001] This invention relates to the field of road early warning technology, specifically an intelligent early warning system for non-motorized vehicles and pedestrians entering highways at highway toll stations. Background Technology

[0002] Among road traffic participants, besides vehicles, there are also vulnerable road users such as pedestrians and non-motorized vehicles. These vulnerable road users, especially pedestrians, have greater freedom of movement and are subject to fewer constraints, making them highly susceptible to traffic accidents. Once an accident occurs, these vulnerable road users often suffer more severe injuries or fatalities due to a lack of protective measures.

[0003] For example, Chinese Patent Publication No. CN114639245A discloses a vehicle-pedestrian collision warning method and device. The method includes: constructing a vehicle entity model and a pedestrian entity model; obtaining the vehicle heading angle, the pedestrian heading angle, the vehicle speed, and the pedestrian speed in the northeast-northeast coordinate system; determining at least one collision critical point based on the vehicle entity model and the pedestrian entity model, and determining the collision critical point coordinate value based on the vehicle heading angle, the pedestrian heading angle, the vehicle speed, and the pedestrian speed; and issuing a collision warning to the vehicle and the pedestrian based on the collision critical point coordinate value and a preset warning judgment rule.

[0004] For example, Chinese Patent Publication No. CN114267181A discloses a pedestrian warning method, device, and electronic device for urban road intersections. This invention uses four aspects as the basis for judgment: pedestrian speed, vehicle speed, pedestrian area, and driver status, to warn of the risks when pedestrians cross the intersection, thereby outputting different collision warning levels to issue collision warning prompts to vehicle drivers, and combining luminous signs, voice warning devices, and zebra crossing warning lights at the intersection to issue warning prompts to pedestrians.

[0005] Existing technologies use the circular coordinates of pedestrian entities and the rectangular coordinates of vehicle entities to obtain the predicted collision point coordinates and perform collision analysis based on velocity components; pedestrian warnings are also achieved by detecting the pixel distance between pedestrians and vehicles. However, existing technologies neglect the movement intentions of vulnerable road users, resulting in the inability to provide timely warnings based on the predicted collision point and pixel distance after identification. In dangerous scenarios such as highway toll stations, it is necessary to analyze pedestrian movement intentions and trajectories to determine the direction of pedestrian intentions and set warning strategies to prevent vulnerable road users from engaging in dangerous behaviors such as entering highways. Summary of the Invention

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: an intelligent early warning system for non-motorized vehicles and pedestrians entering the highway at a toll station, comprising: a target acquisition module, used to divide the toll station image associated with the road into multiple subspaces in a topological connectivity order, wherein non-motorized vehicles and pedestrians in each subspace are regarded as target objects.

[0007] The trajectory extraction module is used to monitor the movement trajectory of the target object in the subspace. Based on the trajectory range of the target object in the subspace, it derives the intent label of the movement trajectory in each subspace, and combines the derived intent label with the movement trajectory to form the trajectory map of each target object.

[0008] The trajectory fitting module is used to determine the trajectory vector of any target object under trajectory fitting based on the intent label of the trajectory map, and to set the intent vector of each target object by guiding the fitted trajectory vector with spatial position.

[0009] The early warning recognition module is used to cross-recognize the intent vector of the target object. By analyzing the interaction points and average error of the movement trajectory of the intent vectors of each target object under cross-analysis, the early warning result is output.

[0010] The beneficial effects of this invention are as follows: First, this invention divides the road associated with toll station images into multiple subspaces in a topologically connected order, treats non-motorized vehicles and pedestrians in each subspace as target objects, monitors the movement trajectory of the target objects in the subspace, derives intent labels based on the trajectory range, combines them into a trajectory map, then determines the trajectory vector based on the intent labels of the trajectory map, and sets the intent vector; finally, cross-recognition is performed based on the intent vector, and the warning result is obtained according to the interaction points and average error. This invention can accurately obtain relevant information about non-motorized vehicles and pedestrians around the toll station, and provide warnings based on pedestrian movement intent, thereby reducing the safety risks brought by non-motorized vehicles and pedestrians entering the highway.

[0011] Second, this invention dynamically segments non-motorized vehicles and pedestrians within each subspace based on the topological relationships of each subspace using a continuous time axis. It sets verification anchor points, including time anchor points marked on the time axis, location anchor points associated with positions, and speed anchor points indicating movement speed. By cross-validating these verification anchor points, target objects are identified. A trajectory extraction mechanism is then embedded into the acquired target objects, categorizing them by subspace type based on consistent trajectories. This allows for more accurate selection of target objects, providing a reliable foundation for subsequent trajectory extraction and analysis, and reducing motion recognition errors caused by foliage obstruction or birds.

[0012] Third, this invention maps the movement trajectory to all involved subspaces, performs stepwise processing on a single subspace and centralized processing on multiple subspaces. Through these two processing methods, the movement trajectory of the target object in different subspaces is determined, and its intent label is accurately derived to comprehensively reflect the movement intent of the target object and provide sufficient data information for constructing a trajectory map.

[0013] Fourth, this invention extracts intent labels from the trajectory map, segments the movement trajectories associated with the intent labels at least once to form multiple segmented local movement trajectories; applies perturbation to the local movement trajectories and linearly fits the trajectory vectors; performs a vector dot product between the trajectory vectors and the potential intent direction to extract the intent vector, thereby improving the accuracy of analyzing the movement intent of the target object; finally, it detects the intersection point of the intent vectors, and constructs trigger conditions based on the subspace to which the intersection point belongs, combined with the average error between the movement trajectories in the trajectory map and the fitted movement trajectories. The warning level corresponding to the trigger condition is used as the current output warning result, which can accurately identify potential dangerous situations and provide corresponding warning levels according to the trigger conditions, thereby achieving timely and effective warnings and reducing safety risks. Attached Figure Description

[0014] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0015] Figure 1 This is a system framework diagram of an intelligent early warning system for non-motorized vehicles and pedestrians entering highways at highway toll stations.

[0016] Figure 2 This is a flowchart illustrating the target acquisition module of an intelligent early warning system for non-motorized vehicles and pedestrians entering highways at highway toll stations.

[0017] Figure 3 This is a flowchart illustrating the trajectory extraction module of an intelligent early warning system for non-motorized vehicles and pedestrians entering highways at highway toll stations.

[0018] Figure 4 This is a flowchart illustrating the trajectory fitting module of an intelligent early warning system for non-motorized vehicles and pedestrians entering highways at highway toll stations.

[0019] Figure 5 This is a flowchart illustrating the early warning and identification module of an intelligent early warning system for non-motorized vehicles and pedestrians entering highways at highway toll stations. Detailed Implementation

[0020] The embodiments of the present invention are described in detail below. The embodiments described below are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. Where specific techniques or conditions are not specified in the embodiments, they shall be performed in accordance with the techniques or conditions described in the literature in the art or in accordance with the product manual.

[0021] See Figure 1 A smart early warning system for non-motorized vehicles and pedestrians entering the highway at a toll station includes: a target acquisition module, a trajectory extraction module, a trajectory fitting module, and an early warning recognition module; wherein, the output end of the target acquisition module is connected to the trajectory extraction module, the output end of the trajectory extraction module is connected to the trajectory fitting module, and the output end of the trajectory fitting module is connected to the early warning recognition module.

[0022] The target acquisition module is used to divide the toll station image into multiple subspaces based on topological connectivity, with non-motorized vehicles and pedestrians in each subspace being regarded as target objects.

[0023] The trajectory extraction module is used to monitor the movement trajectory of the target object in the subspace. Based on the trajectory range of the target object in the subspace, it derives the intent label of the movement trajectory in each subspace, and combines the derived intent label with the movement trajectory to form the trajectory map of each target object.

[0024] The trajectory fitting module is used to determine the trajectory vector of any target object under trajectory fitting based on the intent label of the trajectory map, and to set the intent vector of each target object by guiding the fitted trajectory vector with spatial position.

[0025] The early warning recognition module is used to cross-recognize the intent vector of the target object. By analyzing the interaction points and average error of the movement trajectory of the intent vectors of each target object under cross-analysis, the early warning result is output.

[0026] The aforementioned subspaces include the toll plaza space, the downstream space of the toll plaza, and the upstream space of the toll plaza. The surrounding environment of the toll plaza is divided into multiple spatial regions according to the toll plaza's location and entrance / exit ramps to describe the current regional status around the toll plaza. This forms several types of subspace regions, such as warning zones (outer plaza of the toll plaza and surrounding auxiliary roads), danger zones (toll lanes, ETC lanes, highway ramps), and handling zones (toll booths). Parameters such as movement trajectory and average speed are determined in each subspace. The divided subspaces will be defined according to the actual shapes of toll lanes, ETC lanes, toll booths, and the outer plaza of the toll plaza, establishing multiple regional boundaries and extending the areas corresponding to these lanes to multiple locations at the front and rear plazas of the toll plaza.

[0027] By deploying multiple sets of cameras, radar, and lidar equipment at toll stations, images of pedestrians and non-motorized vehicles at multiple locations are extracted. Image recognition algorithms such as YOLOv7 are then used to identify the current trajectories of pedestrians and non-motorized vehicles for subsequent processing.

[0028] The subspace division is based on GB5768.3-2009 "Road Traffic Signs and Markings". The toll plaza space is within 30m in front of and 20m behind the toll booth (physical coordinates X∈[-20, 30]m, Y∈[-10, 10]m, with the toll booth as the origin); the downstream space is 20-100m behind the toll booth (Y∈[-100, -20]m); the upstream space is 30-100m in front of the toll booth (Y∈[30, 100]m). The topological connectivity order is determined by the vehicle traffic direction of upstream space → plaza space → downstream space.

[0029] like Figure 2 As shown, the implementation of the target acquisition module includes: dynamically segmenting non-motorized vehicles and pedestrians in each subspace according to the topological relationship of each subspace using a continuous time axis, and setting verification anchor points for the dynamically segmented non-motorized vehicles and pedestrians, including time anchor points marked on the time axis, location anchor points associated with the location, and speed anchor points for the movement speed.

[0030] The verification anchors are verified sequentially using a cross-validation method, with time anchors, location anchors, and speed anchors being used as target objects. The non-motorized vehicles and pedestrians corresponding to the verified verification anchors are then used as target objects.

[0031] A trajectory extraction mechanism is embedded in the acquired target object. Target objects with consistent trajectories are classified according to subspace type, and the classified target objects are used as the output target objects.

[0032] The subspace type represents the warning zone, danger zone, and handling zone as described above, to complete the division of the area where the current toll station is located.

[0033] When classifying, the target objects under the three anchor points of time, position, and velocity are used as the main content for identification. The target images with consistent trajectories are identified. These represent multiple target objects extracted through the time anchor points, position anchor points, and velocity anchor points within a unit time period. At least two of these target objects have consistent trajectories. The target objects with consistent trajectories will be used as the output target objects to prevent identification errors due to a single dimension.

[0034] Preferably, target objects with trajectory consistency ≥ 90% are classified according to subspace type, and the classified target objects are used as the final output target objects. In this case, trajectory consistency is achieved by comparing the movement trajectory to which each verified anchor point belongs, comparing these movement trajectories one by one. Trajectory consistency represents the percentage value of the trajectory being consistent after comparison. This percentage is calculated by comparing the proportion of consistent coordinates of each trajectory point on the movement trajectory.

[0035] Preferably, when setting the verification anchor points as described above, the implementation method includes: setting time anchor points for target objects in each subspace in the order of their appearance, setting position anchor points based on the length of the target object's movement path when the time period changes, and using the average speed when setting the position anchor points as the speed anchor points of the corresponding target objects.

[0036] When cross-validating verification anchors, the three anchors are verified step by step. This is achieved by checking the temporal continuity of the target object based on the temporal anchors. For each subspace's tollbooth image, if a target object is not detected in a frame of continuous frame recognition, the tollbooth image of the adjacent subspace is retrieved, and the temporal anchor of that target object is updated as the verified temporal anchor. This indicates that a pedestrian or non-motorized vehicle was not detected in a single frame. It emphasizes that a frame disappeared within the images acquired in a subspace, meaning the pedestrian has moved away from the tollbooth or moved to another subspace. These pedestrians and non-motorized vehicles are the primary targets for current identification, preventing them from forcibly passing through the tollbooth.

[0037] When it is detected that a non-motorized vehicle or pedestrian is not detected in a frame of continuous frame recognition, the toll station image in the adjacent subspace will be retrieved, the time point corresponding to the pedestrian or non-motorized vehicle will be found, and this time point will be updated to the currently labeled time anchor point to complete the verification of the time anchor point.

[0038] Based on location anchors, the directional consistency of the target object is determined, and location anchors that conform to directional consistency are considered verified location anchors. Location anchors are used to emphasize whether the movement of a pedestrian maintains directional consistency across multiple subspace boundaries. Directional consistency is used to filter changes in the pedestrian's movement direction in consecutive frames, removing obviously unreasonable trajectories. For example, if adjacent time points in a continuous time period show a 180° difference in direction angle, this is considered an unreasonable trajectories and needs to be removed to ensure that the currently identified movement trajectory is correct data, and the corresponding pedestrians and non-motorized vehicles are identified as the target objects.

[0039] During directional consistency verification, the coordinates of multiple consecutive trajectory points under the current position anchor point are extracted. The coordinates of the consecutive trajectory points are combined to form a direction vector. The average change angle of the direction vector in consecutive frames is judged. When the average change angle is less than or equal to 30 degrees, it indicates that it conforms to the normal movement pattern. When the average change angle is greater than 30 degrees, it is considered that the current pedestrian movement is in an abnormal movement situation. It is necessary to continue to extract data and update the position anchor point to determine the pedestrian's movement direction relative to the lane and other positions.

[0040] It should be noted that the currently set subspace will divide the area before and after the toll station into multiple small subspaces according to the images captured by the camera, to illustrate the movement of pedestrians and non-motorized vehicles in each lane, auxiliary lane, toll booth, etc. If it is regarded as a verified position anchor point, it means that non-motorized vehicles and pedestrians are moving along the edge of the lane or moving to other lanes and toll booths, etc. In this case, it is necessary to track the position of pedestrians and non-motorized vehicles, treat them as target objects, and determine the movement trajectory of pedestrians and non-motorized vehicles.

[0041] Based on speed anchor points, the average speed range of the target object is determined. Data exceeding the upper limit of the range is filtered, and the corresponding speed anchor points are considered validated. Speed ​​anchor points are biased towards validating excessively fast-moving segments. For example, the average speed range of pedestrians and non-motorized vehicles under normal road conditions is used as the main part of the current validation. This is achieved by generating a confidence interval, such as a range of average speed ± 3 standard deviations, and filtering out anchor points for pedestrians and non-motorized vehicles that exceed the upper limit of this range to prevent the currently identified trajectory from being erroneous data such as birds in flight.

[0042] In one embodiment of the present invention, the trajectory extraction module emphasizes the movement trajectory of the target object, and combines the target object's intent label and movement trajectory into a global trajectory map according to the subspace to which the movement trajectory belongs.

[0043] Preferably, the trajectory map is constructed by: based on the movement trajectory of the target object, selecting the center point of the movement trajectory as the center of a circle, using the smallest circle enclosing the movement trajectory as the trajectory range of the current target object, determining the trajectory range within each subspace of the target object, and constructing the trajectory map of each target object based on the position of the trajectory range within each subspace. When constructing the trajectory map, the acquired images are arranged according to the shooting position, using the X and Y axes, with the center of the current toll station as the origin, the direction perpendicular to the toll station as the Y-axis, and the direction horizontal to the toll station as the X-axis, to obtain a horizontal coordinate system for the trajectory map.

[0044] The center point of the aforementioned movement trajectory can be selected by averaging the coordinates of multiple points along the X and Y axes. Then, the smallest enclosing circle encompassing the movement trajectory is calculated based on this center point. The trajectory range within multiple subspaces is calculated separately for each subspace containing the movement trajectory.

[0045] like Figure 3As shown, the trajectory extraction module monitors the movement trajectory of the target object in the subspace. Its implementation method also includes: tracking the movement trajectory of the target object in real time and mapping the movement trajectory to all involved subspaces, and performing step-by-step processing for a single subspace and centralized processing for multiple subspaces respectively.

[0046] When the target object involves a subspace, the movement trajectory of the target object is judged to determine the trajectory state of the target object, and the decision data of the target object is derived according to the trajectory state to obtain the intent label corresponding to the decision data.

[0047] When the target object involves a subspace, it means that the movement path of pedestrians and non-motorized vehicles is relatively unique, such as when a pedestrian takes a card, a non-motorized vehicle accidentally enters a toll station, or when pedestrians and non-motorized vehicles are gradually approaching the toll station without taking any further action.

[0048] When the target object involves multiple subspaces, for any subspace type, the area ratio of the current target object's trajectory range in each subspace is used as the input parameter. At the same time, the decision data of the target object is derived by combining the weights assigned by the subspace type, and the intent label corresponding to the decision data is obtained.

[0049] When multiple subspaces are involved, it means that the currently identified pedestrian is walking in multiple areas, such as passing through the areas corresponding to multiple subspaces such as the toll plaza outside the toll station, the surrounding auxiliary roads, the toll lanes, the ETC lanes, the highway ramps, and the toll booths. In this case, it is necessary to determine whether the pedestrian is walking normally based on the situation of walking in multiple spaces, and further determine the form of their intention. The same applies to non-motorized vehicles. The trajectory range under multiple areas is used to further identify the behavior status, in order to determine whether there is any behavior of entering the highway.

[0050] The decision data includes the average speed, dwell time, subspace identifier, and coordinates and orientation angles corresponding to the trajectory shape of each target object during trajectory tracking.

[0051] After completing the derivation of decision data for multiple subspaces and a single subspace, all decision data are summarized, and the movement trajectories corresponding to the summarized decision data are combined according to the derivation intent labels to form the current output trajectory map.

[0052] At this point, the decision data will normalize the current data and combine the corresponding movement trajectory with the intent label to form the current trajectory map.

[0053] Preferably, when the target object involves a subspace, the processing method includes: identifying the trajectory state of the target object based on its dwell time, trajectory shape, average speed, and orientation angle relative to the subspace boundary; the dwell time is used to mark the dwell point of pedestrians, etc., during movement, as well as the length of time they stay; the orientation angle indicates the angle with the subspace boundary, and indicates the relative position with respect to the lane or toll plaza.

[0054] Based on the trajectory shape, the subspace type pointed to by the current target object is determined. The dwell point, average speed and direction angle under each subspace type are used as derivation rules. The decision rules labeled with historical data are applied to determine the trajectory state of the current user's movement. The intent label corresponding to the trajectory state is output as decision data.

[0055] For example, the decision rule can be expressed as: IF Average speed = V1 (representing any speed) AND Direction pointing to the lane (described using the subspace type pointed to by the direction angle) THEN Intent = Potential intrusion; This potential intrusion refers to the current user moving towards a relatively dangerous area in a single subspace. If their stop point is a toll booth and the stop time is greater than a certain time, then the intent can be considered to be to take a card; and in some scenarios, when the average speed value is small and the shape represented by the movement trajectory is a hesitant shape, then the intent can be directly pointed to as hesitant or observing; at this time, by taking the stop point, average speed and direction angle as the main derivation targets, the decision rule corresponding to the data is found, and the intent label derived from the decision rule is added to the output decision data.

[0056] The trajectory status here is used to illustrate a preliminary intent derived from multiple values ​​to identify the intent bias of pedestrians identified at the current highway toll station. This intent bias data will be used as output decision data to find the intent label that is closest to the current movement trajectory from the database.

[0057] The intent labeling involves a subspace, and the main data for obtaining the intent label is the dwell point, average speed, and direction angle. The intent labels that conform to the current inference rules are used as the output decision data.

[0058] Preferably, when the target object involves multiple subspaces, the implementation method includes: calculating the area ratio of the trajectory range under each subspace, performing weighted summation based on the weight of the subspace type to obtain a risk score, using the risk score, the area ratio of the trajectory range, and the subspace type as derivation rules, applying decision rules labeled with historical data, and determining the decision data of the target object using the intent label corresponding to the decision rules.

[0059] The weights of the aforementioned subspace types will be set according to the specific type of the subspace. A preset weight will be set for each type of warning zone, danger zone, and response zone. This weight will be configured based on historical data, such as setting weights of 0.35, 0.5, and 0.15 in sequence. Then, the weight division will be refined according to the specific content of the warning zone, danger zone, and response zone, such as lanes and ramps.

[0060] The decision rules used at this time will identify the pedestrian's complete movement trajectory in multiple subspaces. Unlike the processing method for a single subspace, the comprehensive explanation under the analysis of multiple subspaces is the comprehensive intent, which emphasizes the summarization of the pedestrian's movement trajectory status in multiple subspaces. The output decision data will summarize the data in these two cases to illustrate a more complete pedestrian intent recognition.

[0061] The rules derived here are based on the risk score, combined with the area ratio of the trajectory range and the subspace type, to explain whether the current pedestrian's intention belongs to a high-risk intrusion, an emergency situation, or other trajectory states.

[0062] For example, there is a decision rule that can be expressed as follows: risk score ≥ 0.7; area ratio of the danger zone trajectory range ≥ 0.6; subspace type shows that the pedestrian quickly and directly enters the danger zone from the warning zone without obvious deceleration or observation behavior. In this case, the inferred intention is a high-risk intrusion intention.

[0063] At this point, the risk score reaches 0.7 or higher, indicating a relatively dangerous situation overall. Subsequently, if there is a large proportion of the trajectory area in the toll lanes and other locations marked in the danger zone, it indicates that the pedestrian is moving around a lot in the danger zone and has entered the danger zone directly through the warning zone. At this point, the target may pose a threat to the normal operation of the toll station, and it is necessary to record their intentions and generate a corresponding trajectory map.

[0064] Other decision-making rules will be predefined in the database to assist in the subsequent analysis of the behavioral intentions of pedestrians and non-motorized vehicles.

[0065] When multiple subspaces are involved, the output decision data format is the same as the output data format when a single subspace is involved. The difference is the intent described under the decision rule. If the target object is the same when analyzing one subspace at a time and analyzing multiple subspaces together, that is, when the movement trajectory of a pedestrian at a toll station is divided into two processing methods, one analysis of one subspace at a time and analysis of multiple subspaces together, the intent identified by both methods will be marked in the trajectory map.

[0066] The subsequent method for aggregating the decision data from the two approaches involves tracking the trajectory of each target object in real time and combining the intent labels marked under separate subspace analysis with the intent labels under centralized analysis of multiple subspaces as the intent labels for each target object. Each movement trajectory will be labeled with an intent label derived through decision rules to illustrate the movement trajectory of pedestrians and non-motorized vehicles.

[0067] In one embodiment of the present invention, a trajectory fitting module is used to receive a trajectory map from a trajectory extraction module and extract intent labels set according to decision rules. Then, these intent labels are fitted to the movement trajectory. The fitting method includes, but is not limited to, trajectory fitting for the same intent label, trajectory fitting for similar intent labels, and combining multiple intent labels identified in the current trajectory map to describe the movement intent of pedestrians and non-motorized vehicles near the toll station. Based on the movement intent, warnings or other types of intent recognition are then given to pedestrians.

[0068] like Figure 4 As shown, the implementation of the trajectory fitting module includes: extracting the intent labels marked in the trajectory map, segmenting the movement trajectory associated with the intent labels at least once to form multiple segmented local movement trajectories; when segmenting, the segmentation is based on the stationary time and hovering time of the target object during movement, or based on the length of the target object's movement trajectory for equal-sized segmentation.

[0069] A perturbation is applied to each local movement trajectory, and a random error is added to the coordinate values ​​of the local movement trajectory to form multiple perturbation trajectory points.

[0070] The purpose of applying perturbation is to simulate situations where the target is briefly occluded or detection fails. This causes the path trajectory to depend on the overall movement trend when it is being identified and processed in situations where there are obstructions such as leaves, various noises, and uncertainties. At the same time, this method also simulates the effects of non-uniform motion of pedestrians and non-motorized vehicles or changes in camera shooting angle, so that subsequent trajectory processing can cope with these changes and improve the accuracy of intention recognition of movement trajectory as much as possible.

[0071] When applying a disturbance, random errors are introduced into the coordinate values ​​of the moving trajectory, subtly altering the coordinates of points on the trajectory while maintaining the overall shape of the trajectory. The added random error can be selected from normally distributed random numbers with a mean of 0 and a standard deviation of σ. σ represents the intensity of the added noise and can be set based on the current error level of the camera device in recognizing pedestrians and non-motorized vehicle trajectories. For ordinary CMOS cameras, σ typically ranges from 0.2 to 0.4 meters, while for LiDAR, it ranges from 0.05 to 0.1 meters. The dimension of the random error being added is adjusted based on these values.

[0072] When applying the above-mentioned perturbation, in addition to directly perturbing according to the random error of the normal distribution, a certain proportion of trajectory points can be randomly discarded to form a perturbed curve or multiple trajectory points. Then, elastic deformation perturbation can be used to change the position of the entire local movement trajectory by stretching the curve.

[0073] The perturbed trajectory points are linearly fitted to the segmented local movement trajectories until all local movement trajectories are fitted, resulting in the fitted trajectory vector of the overall movement trajectory. In other words, after applying perturbation to all segmented local movement trajectories, they are combined with the perturbed trajectory points to reassemble the overall movement trajectory.

[0074] When fitting, linear least squares fitting can be used to fit the perturbation trajectory points and the trajectory points of the segmented local movement trajectory into a straight line. This fitting method is suitable when the curvature of the local fitting curve is low and the movement trajectory is monotonous, such as a straight movement trajectory. For more complex movement trajectories, such as a movement trajectory that moves back and forth while hovering, polynomial fitting can be used to identify the perturbation at the turning point and fit it into a curve. Then, principal component analysis is performed on the fitted curve to determine the orientation of the trajectory vector in space.

[0075] During principal component analysis, the main direction of motion of the local movement trajectory is obtained by treating the points in the segmented local movement trajectory as a point cloud. The trajectory vector that conforms to the perturbation deviation is used as the output trajectory vector, and the intention label corresponding to the movement trajectory is labeled.

[0076] Linear least squares fitting finds a straight line Find its slope and intercept This minimizes the sum of the squares of the perpendicular distances from all points to this line, where x and y represent the coordinates of each trajectory point. The coordinates of the trajectory points described here include the trajectory points of the segmented local movement trajectory and the trajectory points with applied perturbation, and the number of these trajectory points is denoted as N.

[0077] The slope is calculated as follows: The slope calculated at this point represents the direction of the trajectory vector after the current linear least squares fitting. Alternatively, the current trajectory vector can be normalized to form a unit vector, showing the specific form of the current trajectory vector and reflecting the average motion direction of this local moving set.

[0078] The intercept value is expressed as follows: By outputting this data, the trajectory vector processing is completed.

[0079] When fitting a polynomial, the solution is obtained using a quadratic function, such as... In the form of a, b, and c, the coefficients a, b, and c are obtained using the least squares method. The polynomial will then be fitted to a curve. The last point is selected as the key point, and the first derivative of the slope of the curve is used to construct a trajectory vector representing the direction. The fitted trajectory vector emphasizes the instantaneous direction of movement and better reflects the behavior that pedestrians are about to perform.

[0080] In principal component analysis, the coordinates of all points are subtracted from their average value so that the center of the point cloud is located at the origin. Then, a covariance matrix is ​​set for all points, and the eigenvalues ​​and eigenvectors of the covariance matrix are solved. The eigenvector corresponding to the largest eigenvalue is the first principal component, which represents the direction of the largest data variance, i.e., the main extension direction of the trajectory.

[0081] At this point, three methods are used to fit the perturbed trajectory points and the segmented, undisturbed trajectory points to obtain a trajectory vector that satisfies the overall movement trend of the pedestrian.

[0082] It should be noted that after fitting the trajectory points and forming the trajectory vector using the above method, the trajectory vector will be normalized to form a unit vector.

[0083] The fitted trajectory vector is multiplied by the latent intention direction to determine the projection coefficients in the latent intention direction. The latent intention direction with the largest projection coefficient is taken as the output intention vector.

[0084] The potential intent direction is a preset vector. A unit vector is set for each of the following directions: the direction pointing to the highway entrance, the direction pointing to the toll booth window, the direction pointing to the toll station exit or leaving the lane. These processing methods inject perturbation deviations into the direction of the trajectory after the trajectory extraction module processes the trajectory, in order to improve the handling of anomalies in the initial trajectory extraction caused by changes in camera angle and obstructions such as leaves.

[0085] As for the projection coefficient, it is the result of the dot product of the trajectory vector and the potential intention direction. The dot product method is to calculate the cosine similarity between the normalized trajectory vector and the potential intention direction. The larger the cosine value obtained by the dot product, the closer the direction of the trajectory vector is to the preset intention direction, so as to further limit the specific intention of the pedestrian. The potential intention direction at this time is then added to the intention label of the original movement trajectory to complete the analysis and processing in multiple cases.

[0086] In one embodiment of the present invention, the warning recognition module determines the aggregation or potential conflict point of pedestrians among multiple segmented movement trajectories by cross-identifying the intent vectors; at this time, the intent vectors can be extended to determine the location where multiple pedestrians and non-motorized vehicles tend to gather when moving, and this location may lead to traffic risks or group gathering phenomena in different scenarios, which is the main warning monitoring location of the current toll station.

[0087] For example, if the intersection of the intention vectors of two pedestrians in front of the toll station is close to the card dispenser, multiple pedestrians will temporarily gather at the card dispenser. At this time, it is necessary to observe the gathering situation to prevent other potential problems. If there is an intersection of the intention vectors of pedestrians and non-motorized vehicles, it indicates that collisions or interference may occur. If the intersection of the intention vectors of pedestrians and non-motorized vehicles corresponds to a vehicle, there may be a traffic safety hazard. In this case, it is still necessary to monitor the corresponding location point. Then, it is necessary to determine the minimum error value between the intersection of the intention vectors and the current identified movement trajectory to further identify whether there is a risk of intersection between pedestrians and non-motorized vehicles during movement and issue relevant warnings.

[0088] like Figure 5 As shown, the implementation of the early warning recognition module includes: for each group of target objects, detecting the intersection point of intent vectors, and labeling the intersection point according to the subspace to which the intersection point belongs; at this time, the interaction points between intent vectors are obtained by extending the intent vectors, and then the interaction points are labeled according to the type of subspace to which the interaction points fall. If the current interaction point falls into the danger zone, it is marked as a potential conflict.

[0089] By combining the marked intersection points with the average error between the moving trajectory in the trajectory map and the fitted moving trajectory, trigger conditions are constructed, and the warning level corresponding to the trigger conditions is used as the current warning result.

[0090] If there is only one target object at this time, that is, only one intention vector, or if there is no direct intersection point between each group of target objects, the point extended to the subspace by the intention vector is regarded as the intersection point, which indicates the current movement tendency and intersection position of pedestrians and non-motorized vehicles.

[0091] When calculating the average error, the average error is obtained by comparing the actual movement trajectory collected in the trajectory map with the movement trajectory after adding perturbations and fitting. This determines whether the interaction point of the current intent vector can correspond to the actual movement trajectory, so as to set early warnings for the movement of pedestrians and non-motorized vehicles in a timely manner.

[0092] When setting trigger conditions, set a condition based on the subspace type to which the interaction point belongs, and retrieve the average error to check if the average error is less than a certain set threshold. The threshold set here is based on the accuracy of identifying pedestrian movement trajectory points after the current camera captures the image. Under normal conditions, 0.5m, 1.0m and 1.5m can be used as the threshold for setting trigger conditions.

[0093] The triggering conditions can be illustrated by the following example: when the intersection is within the lane and the average error is <0.5m, it is considered a Level 1 warning, and a combination of audible and visual alarms and a linked barrier gate is used; when the intersection is within the warning zone and the average error is <1.0m, it is considered a Level 2 warning, and LED warnings and voice prompts are issued; if the average error is >1.5m, it is considered a Level 3 warning, at which point it is necessary to re-predict and update the pedestrian's movement trajectory in real time to determine the pedestrian's intention.

[0094] It should be noted that the triggering conditions are set in advance based on the situation at the highway toll station, and the warning level and measures corresponding to each triggering condition are explained, so as to provide a real-time warning response based on the movement intentions of pedestrians, etc.

[0095] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention, which are still covered within the protection scope of the present invention.

Claims

1. A smart early warning system for non-motorized vehicles and pedestrians entering highways at highway toll stations, characterized in that, include: The target acquisition module is used to divide the toll station image into multiple subspaces according to the topological connectivity order of the associated road, and non-motorized vehicles and pedestrians in each subspace are regarded as target objects. The trajectory extraction module is used to monitor the movement trajectory of the target object in the subspace. Based on the trajectory range of the target object in the subspace, it derives the intent label of the movement trajectory in each subspace and combines the derived intent label with the movement trajectory to form the trajectory map of each target object. The trajectory fitting module is used to determine the trajectory vector of any target object under trajectory fitting based on the intent label of the trajectory map, and to set the intent vector of each target object by spatial positioning of the fitted trajectory vector. The early warning recognition module is used to cross-recognize the intent vector of the target object. By analyzing the interaction points and average error of the movement trajectory of the intent vectors of each target object under cross-analysis, the early warning result is output.

2. The intelligent early warning system for non-motorized vehicles and pedestrians entering the highway at a toll station according to claim 1, characterized in that, The target acquisition module can be implemented in the following ways: Based on the topological relationship of each subspace, non-motorized vehicles and pedestrians in each subspace are dynamically segmented by a continuous time axis. Verification anchor points are set for the dynamically segmented non-motorized vehicles and pedestrians, including time anchor points marked on the time axis, location anchor points associated with the location, and speed anchor points of the movement speed. The verification anchors are verified sequentially using a cross-validation method, including time anchors, location anchors, and speed anchors. The non-motorized vehicles and pedestrians corresponding to the verified verification anchors are then used as the target objects. A trajectory extraction mechanism is embedded in the acquired target object. Target objects with consistent trajectories are classified according to subspace type, and the classified target objects are used as the output target objects.

3. The intelligent early warning system for non-motorized vehicles and pedestrians entering the highway at a toll station according to claim 2, characterized in that, When setting verification anchors, the implementation methods include: Set time anchors for target objects in each subspace in chronological order of their appearance, and set position anchors based on the length of the target object's movement path during each time period. Use the average speed when setting the position anchor as the speed anchor for the corresponding target object.

4. The intelligent early warning system for non-motorized vehicles and pedestrians entering the highway at a toll station according to claim 2, characterized in that, When cross-validating anchor points, the implementation methods include: The time continuity of the target object is checked based on the time anchor point. For the toll station image in each subspace, if the target object is not detected in one frame of the current toll station image in the continuous frame recognition, the toll station image of the adjacent subspace is retrieved and the time anchor point of the target object is updated as the verified time anchor point. Based on the location anchor points, the orientation consistency of the target object is determined, and the location anchor points that conform to the orientation consistency are regarded as verified location anchor points. Based on the velocity anchor points, the range of the average velocity of the target object is determined. Data exceeding the upper limit of the range is filtered out, and the corresponding velocity anchor points are regarded as verified velocity anchor points.

5. The intelligent early warning system for non-motorized vehicles and pedestrians entering the highway at a toll station according to claim 1, characterized in that, When constructing a trajectory map, the implementation methods include: Based on the movement trajectory of the target object, the center point of the movement trajectory is selected as the center of the circle, and the smallest circle enclosing the movement trajectory is taken as the trajectory range of the current target object. The trajectory range in each subspace of the target object is determined, and the trajectory map of each target object is constructed based on the position of the trajectory range in each subspace.

6. The intelligent early warning system for non-motorized vehicles and pedestrians entering the highway at a toll station according to claim 2, characterized in that, The trajectory extraction module monitors the movement trajectory of a target object in a subspace, and its implementation also includes: The system tracks the movement trajectory of the target object in real time and maps the trajectory to all involved subspaces, processing each subspace step by step or processing multiple subspaces together. When the target object involves a subspace, the movement trajectory of the target object is judged to determine the trajectory state of the target object, and the decision data of the target object is derived according to the trajectory state to obtain the intent label corresponding to the decision data. When the target object involves multiple subspaces, for any subspace type, the area ratio of the current target object's trajectory range in each subspace is used as the input parameter. At the same time, the weights assigned by the subspace type are combined to deduce the target object's decision data and obtain the intent label corresponding to the decision data. After completing the derivation of decision data for multiple subspaces and a single subspace, all decision data are summarized, and the movement trajectories corresponding to the summarized decision data are combined according to the derivation intent labels to form the current output trajectory map.

7. The intelligent early warning system for non-motorized vehicles and pedestrians entering the highway at a toll station according to claim 6, characterized in that, When the target object involves a subspace, the processing methods include: The trajectory state of the target object is identified based on its dwell time, trajectory shape, average speed, and orientation angle relative to the subspace boundary. Based on the trajectory shape, the subspace type pointed to by the current target object is determined. The dwell point, average speed and direction angle under each subspace type are used as derivation rules. The decision rules labeled with historical data are applied to determine the trajectory state of the current user's movement. The intent label corresponding to the trajectory state is output as decision data.

8. The intelligent early warning system for non-motorized vehicles and pedestrians entering the highway at a toll station according to claim 6, characterized in that, When the target object involves multiple subspaces, the implementation methods include: The area ratio of the trajectory range under each subspace is calculated, and a weighted sum is performed according to the weight of the subspace type to obtain the risk score. The risk score, the area ratio of the trajectory range, and the subspace type are used as derivation rules. Decision rules labeled with historical data are applied, and the decision data of the target object is determined by the intent label corresponding to the decision rules.

9. The intelligent early warning system for non-motorized vehicles and pedestrians entering the highway at a toll station according to claim 1, characterized in that, The trajectory fitting module can be implemented in the following ways: Extract the intent labels marked in the trajectory map, and segment the movement trajectory associated with the intent labels at least once to form multiple segmented local movement trajectories; A perturbation is applied to each local movement trajectory, and a random error is added to the coordinate values ​​of the local movement trajectory to form multiple perturbation trajectory points; The perturbated trajectory points are linearly fitted to the trajectory points of the segmented local movement trajectories until the fitting process is completed for all local movement trajectories, and the overall movement trajectory is obtained as the fitted trajectory vector. The fitted trajectory vector is multiplied by the latent intention direction to determine the projection coefficients in the latent intention direction. The latent intention direction with the largest projection coefficient is taken as the output intention vector.

10. The intelligent early warning system for non-motorized vehicles and pedestrians entering the highway at a toll station according to claim 1, characterized in that, The implementation methods of the early warning identification module include: For each group of target objects, detect the intersection point of intent vectors, and label the intersection point according to the subspace to which it belongs; By combining the marked intersection points with the average error between the moving trajectory in the trajectory map and the fitted moving trajectory, trigger conditions are constructed, and the warning level corresponding to the trigger conditions is used as the current warning result.

Images