Pedestrian and vehicle dynamic prediction method, device and storage medium

Abstract

The present invention relates to a dynamic recognition method, and specifically to a method, device, and storage medium for dynamic prediction of pedestrians and vehicles, the method comprising: the terminal generating an original data set based on image information acquired by a capture operation of an image device, the original data set comprising a pedestrian data set and a vehicle data set; inputting the pedestrian data set into a pedestrian prediction model to obtain a pedestrian prediction trajectory corresponding to the pedestrian data set; inputting the vehicle data set into a vehicle prediction model to obtain a vehicle prediction trajectory corresponding to the vehicle data set; performing trajectory model fitting on the pedestrian prediction trajectory and the vehicle prediction trajectory to determine the collision prediction results of pedestrians and vehicles. The present invention improves the accuracy and comprehensiveness of pedestrian and vehicle collision prediction results at special intersections such as schools, parks, and shopping malls through an intersection coverage detection method and the setting of a standard space-time coordinate system, thereby ensuring road public safety on special sections.

Description

Technical Field

[0001] The present invention relates to a dynamic recognition method, and in particular to a pedestrian and vehicle dynamic prejudgment method, device and storage medium. Background Art

[0002] With the continuous development of cities, the number of vehicles on urban roads has increased, raising numerous safety issues for both pedestrians and vehicles. Collisions between vehicles and pedestrians are becoming more frequent at complex intersections and on roads with high pedestrian traffic, such as those near schools, parks, and shopping malls. This has a significant impact on traffic and road safety. To improve road safety and smooth traffic flow at complex intersections and on roads with high pedestrian traffic, collision prediction and preemptive measures should be implemented to prevent pedestrian and vehicle collisions to a certain extent.

[0003] Patent No. CN112487905B discloses a method and system for predicting the danger level of pedestrians around a vehicle. The system uses onboard sensors to collect image information around the vehicle, and estimates the time to collision (TTC) based on the pedestrian's relative position p = (x, y) and relative speed v = (vx, vy). A machine model is then used to fit the pedestrian's trajectory. Cluster analysis is then used to map different danger levels of pedestrians, and the system then identifies the pedestrian's danger level based on characteristic parameters. By identifying the pedestrian's danger level, collisions between pedestrians and vehicles can be avoided.

[0004] In practice, most vehicle and pedestrian prediction models rely on onboard sensors to predict pedestrian trajectories and determine whether there is a collision risk between the vehicle and pedestrian. However, at complex intersections or roads with heavy traffic, the signals received by onboard sensors are complex and cannot be fully analyzed and calculated, resulting in a high risk of misjudgment. Furthermore, onboard sensors are easily obstructed by obstacles such as pedestrians, vehicles, and the environment, resulting in very limited image information. This results in incomplete data extraction, making it inadequate for complex intersections and roads with heavy traffic, and resulting in poor prediction comprehensiveness. Furthermore, due to the specific vehicle load and travel speed, the coordinates of the sensor acquisition equipment are constantly changing during the prediction process, resulting in large systematic errors in the acquired data. Furthermore, onboard prediction systems are limited by the specific vehicle implementation, and their network interactivity and real-time transmission capabilities are limited, making it impossible to update the trained model in real time, which also poses a certain risk to prediction accuracy. Summary of the Invention

[0005] The present invention addresses the technical problems in the existing technology of single pedestrian or vehicle trajectory identification, incomplete data collection, poor real-time performance and low accuracy. It provides a pedestrian and vehicle dynamic prediction method, device and storage medium for adapting to complex intersections and sections with heavy pedestrian flow, thereby improving the real-time prediction and accuracy of special scenarios.

[0006] In a first aspect, the present invention provides a method for predicting pedestrian and vehicle dynamics, which is applied to a terminal and includes:

[0007] The terminal generates an original data set based on image information obtained by a capture operation of an imaging device, wherein the original data set includes a pedestrian data set and a vehicle data set; the pedestrian data set includes pedestrian morphological parameters, pedestrian trajectory parameters, and a first time parameter; the vehicle data set includes vehicle morphological parameters, vehicle trajectory parameters, and a second time parameter;

[0008] The pedestrian dataset is input into the pedestrian prediction model to obtain the pedestrian prediction trajectory corresponding to the pedestrian dataset, and the vehicle dataset is input into the vehicle prediction model to obtain the vehicle prediction trajectory corresponding to the vehicle dataset. Trajectory models are fitted on the pedestrian prediction trajectory and the vehicle prediction trajectory to judge the collision prediction results of the pedestrian and the vehicle, and the collision prediction results include collision or no collision; the pedestrian prediction model and the vehicle prediction model are both models trained based on multiple corresponding dataset samples, corresponding prediction trajectories and corresponding collision prediction results; each dataset sample in the dataset sample includes pedestrian morphological parameters, pedestrian trajectory parameters and first time parameters related to the pedestrian prediction model generated by a single capture operation, or vehicle morphological parameters, vehicle trajectory parameters and second time parameters related to the vehicle prediction model generated by a single capture operation, the pedestrian dataset and the vehicle dataset are feature data, and the collision prediction result is label data.

[0009] Specifically, the aforementioned pedestrian and vehicle datasets are primarily collected and processed using imaging equipment installed at complex intersections or roads with dense pedestrian traffic. By acquiring images from a bird's-eye view, the acquired image information is sufficiently comprehensive, and a data label can be provided for each vehicle and pedestrian passing through the intersection, rather than being limited to the planar and one-sided acquisition methods of on-board sensors. This results in more comprehensive and accurate collision prediction results. For example, while the vehicle is driving, after acquiring pedestrian information in the vehicle's current surrounding directions, the on-board sensors are unable to acquire the trajectory information of pedestrians behind the current pedestrian's blind spot due to obstruction by the surrounding pedestrians. If the pedestrian behind the blind spot accelerates to overtake the current pedestrian and moves in front of the vehicle, the existing technology is unable to predict the collision between the pedestrian behind the blind spot and the vehicle in advance, as there is no relevant trajectory data recorded before the pedestrian overtakes, which can easily lead to the risk of collision between pedestrians and vehicles. It is worth understanding that not only does the current pedestrian have an information acquisition blind spot, but the current vehicles around the vehicle and the current environmental obstacles also cause information acquisition blind spots in existing technologies, such as common "ghosting," "illegal overtaking," and "random weaving." Because these behaviors cause pedestrian trajectories to change quickly and quickly, with many obstructions and discontinuous trajectory capture, existing technologies are unable to identify and predict them in a timely manner, resulting in low accuracy and incomplete collision prediction results. In addition, during the capture process, the vehicle and pedestrian are moving. The vehicle-mounted sensors can only determine the relative position changes of pedestrians and vehicles through the continuously transmitted data information. This has large limitations and system errors, and thus the problem of insufficient judgment accuracy.

[0010] In the embodiment provided by the present invention, since the imaging device is set up at a fixed position at the intersection, a standard spatial coordinate system (x, y) will be established based on the road conditions. Thus, the pedestrian prediction model and the vehicle prediction model can use the same standard spatial coordinate system (x, y) to independently generate pedestrian prediction trajectories and vehicle prediction trajectories. This allows the two models to not interfere with each other in the process of generating the prediction trajectories, and can also perform trajectory fitting through the same standard spatial coordinate system, thereby resolving interference during the relative displacement of pedestrians and vehicles, eliminating system errors, and improving the accuracy and reliability of collision prediction results. Therefore, the terminal has the ability to analyze and process specific application scenarios and has strong applicability.

[0011] It is worth noting that the present invention uses an imaging device to generate the original data set. In theory, the imaging device is a type of sensing device. If radar, laser or other sensors are used to generate the original data set, it should also be suitable for the pedestrian and vehicle dynamic prediction method provided by the present invention.

[0012] In a pedestrian and vehicle dynamic prediction method provided in another embodiment of the first aspect, the pedestrian morphological parameters include gender, age, height and running status, and the running status includes at least one of walking, running, bicycle and electric vehicle and the corresponding standing state, standing state, avoiding state, overtaking state or stable state; the pedestrian trajectory parameters include a first spatial coordinate, a first speed and a first direction.

[0013] Specifically, the trajectory of a pedestrian mainly depends on the speed and direction of travel. However, due to the large number of uncertainties in the speed and direction of pedestrians during their movements, the accuracy of pedestrian trajectory prediction is greatly affected by these uncertainties. The present invention mainly sets three sets of characteristic data in the pedestrian data set, namely pedestrian morphological parameters, pedestrian trajectory parameters and first time parameters, to judge the speed and direction of pedestrians during their movement. Among them, the pedestrian morphological parameters include gender, age, height and running status. In fact, gender differences between men and women will lead to some differences in running speed, different age groups will also determine the differences in their running speeds, and different heights will lead to different strides. The running states include walking, running, bicycles and electric vehicles, as well as the corresponding stopping state, starting state, overtaking state, avoidance state and stable state. Because pedestrians at complex intersections or busy roads are extremely complex and their trajectory changes are influenced by numerous factors, the present invention introduces at least four conventional operating states: walking, running, cycling, and electric vehicles. Each operating state is assigned at least five state predictions: stopping, starting, avoiding, overtaking, and stable. This fully simulates the possible operating modes of pedestrians and makes pedestrian trajectory prediction more accurate. It is worth noting that the present invention classifies self-balancing vehicles and other similar vehicles as electric vehicles. If self-balancing vehicles appear frequently at intersections, a separate self-balancing vehicle motion state can also be added.

[0014] The pedestrian trajectory parameters include a first spatial coordinate, a first speed and a first direction. Since the present invention introduces a standard spatial coordinate system, the pedestrian trajectory parameters are added with time, speed and angle references on the basis of the standard spatial coordinate system, thereby forming (x 人 ,y 人 , a 人 , v 人 , s 人 ) pedestrian trajectory coordinate system, and the changes of the pedestrian's first spatial coordinate, first speed, first direction and first time parameters are used to fit the pedestrian's trajectory in (x 人 ,y 人 , a 人 , v 人， The running trajectory of s people) in the pedestrian trajectory coordinate system.

[0015] Wherein, the first spatial coordinate is (x 人 ,y 人 ), x人 is the pedestrian's X-axis coordinate, y 人 is the Y-axis coordinate of the pedestrian, a 人 is the first direction (the first direction is constructed with the positive direction of the X axis or the positive direction of the Y axis as a reference starting angle of 0°), v 人 is the first speed, s 人 is the first time parameter.

[0016] In another embodiment of the first aspect, the pedestrian data set and the pedestrian data sample set both include at least: a first weight, a second weight and a third weight; the first weight is used to constrain the degree of influence of the pedestrian morphological parameters on the pedestrian predicted trajectory and collision prediction results; the second weight is used to constrain the degree of influence of the pedestrian running state on the pedestrian predicted trajectory and collision prediction results; the third weight is used to constrain the degree of influence of the first time parameter on the pedestrian predicted trajectory and collision prediction results.

[0017] Specifically, the composition of pedestrians varies in different scenarios. For example, the road sections near shopping malls are mostly young people, who are characterized by high speeds and diverse travel modes (mostly on electric vehicles, bicycles, and walking). Around schools, for example, depending on the situation, such as elementary schools, middle schools, and high schools, the pedestrians are mainly a mix of elderly, middle-aged, and young people. Middle schools are mainly a mix of middle-aged and young people, and high schools are mainly teenagers. Near stations, for example, the pedestrian data set and pedestrian data sample set are mainly mixed. Therefore, the pedestrian data set and pedestrian data sample set both include at least: a first weight, a second weight, and a third weight.

[0018] Furthermore, the first weight ratio is initially made the highest. As the pedestrian prediction model is used for a longer time and the pedestrian structure of the road section is gradually used, the first weight ratio will gradually decrease, and it will better adapt to different intersections. By training and updating the pedestrian prediction model through machine learning, a pedestrian prediction model that matches different application scenarios is obtained, thereby improving the accuracy of pedestrian prediction trajectories and collision prediction results.

[0019] In another embodiment of the first aspect, the pedestrian data set and the pedestrian data sample set both include at least: a first correction parameter, a second correction parameter, a third correction parameter and a fourth correction parameter; the first correction parameter is used to constrain the degree of influence of gender on the first speed, the second correction parameter is used to constrain the degree of influence of age on the first speed, the third correction parameter is used to constrain the degree of influence of height on the first speed, and the fourth correction parameter is used to constrain the degree of influence of the operating state on the first speed; the first correction parameter, the second correction parameter, the third correction parameter and the fourth correction parameter are adjusted as the terminal usage time increases.

[0020] Specifically, the present invention sets corresponding speed value intervals according to the running state, such as walking speed of 0m / s-1.5m / s, running speed of 0m / s-2.5m / s, bicycle speed of 0m / s-3.5m / s, and electric vehicle speed of 0m / s-4.3m / s. However, in the process of trajectory prediction, due to information such as the pedestrian's gender, age, height and running state, the speed of each person is not consistent. For example, for a child, under the same walking state, its peak speed cannot reach 1.5m / s, so it needs to be corrected. Similarly, during walking, when stopping to make a phone call or eat, the speed is 0m / s; when starting to prepare to walk, the initial speed is only 0.6m / s, and reaches 1.5m / s when moving steadily; when avoiding other pedestrians, vehicles or other obstacles, the speed will be reduced to 0.8m / s accordingly, or directly reduced to a stopped state. Obviously, the first speed is affected by many factors. In order to ensure the accuracy of the pedestrian's predicted speed, the first correction parameter S is set accordingly. sex The second correction parameter S is used to constrain the influence of gender on the first speed. age The third correction parameter S is used to limit the influence of age on the first speed. h The fourth correction parameter S is used to constrain the influence of height on the first speed and co Used to constrain the degree of influence of the operating state on the first speed, by correcting these correction parameters, the first speed of the pedestrian can be better predicted, thereby improving the accuracy of the pedestrian's predicted trajectory and collision prediction results.

[0021] In another embodiment of the first aspect, the vehicle morphology parameters include steering information, acceleration and deceleration information, and operation information, and the operation information includes one of parking, overtaking, and normal driving; the vehicle trajectory parameters include a second spatial coordinate, a second speed, and a second direction.

[0022] Specifically, compared to the diverse and changeable pedestrians, the mobility and changeability of vehicles are relatively weak. Therefore, when constructing the vehicle prediction model, the present invention mainly considers the influence of the vehicle steering information and acceleration and deceleration information on the second speed and second direction of the vehicle. The vehicle prediction model also includes the vehicle trajectory coordinate system (x 车 y, a, v, s) and calculates the predicted vehicle trajectory using the vehicle trajectory coordinate system. Turning information is primarily acquired based on turn signals and wheel tilt angles, while acceleration and deceleration information is primarily acquired based on brake lights and traffic light information. The vehicle also includes operational information to determine parking, overtaking, and normal driving states.

[0023] Wherein, the second space coordinate is (x 车 ,y 车 ), x 车is the vehicle's X-axis coordinate, y 车 is the Y-axis coordinate of the vehicle, a 车 is the second direction (the second direction is constructed with the positive direction of the X axis or the positive direction of the Y axis as the reference starting angle of 0°, and is consistent with the 0° starting angle of the pedestrian), v 车 is the second speed, s 车 is the second time parameter.

[0024] Furthermore, the terminal has a communication function with the traffic lights installed at the intersection, and can accurately obtain the traffic light information at the intersection.

[0025] In another embodiment of the first aspect, the vehicle data set and the vehicle data sample set both include at least: a fourth weight, a fifth weight and a sixth weight; the fourth weight is used to constrain the degree of influence of the vehicle morphology parameters on the vehicle predicted trajectory and collision prediction results; the fifth weight is used to constrain the degree of influence of the vehicle operating state on the vehicle predicted trajectory and collision prediction results; the sixth weight is used to constrain the degree of influence of the second time parameter on the vehicle predicted trajectory and collision prediction results.

[0026] Specifically, the fourth weight, fifth weight and sixth weight of the vehicle data set and the corresponding data sample set are compared with those of pedestrians. The weights are mainly adjusted according to the usage time of the vehicle prediction model, and the weight coefficients are optimized and adjusted according to the vehicle prediction trajectory and collision prediction results, thereby improving the accuracy of the vehicle prediction trajectory and collision prediction results.

[0027] In another embodiment of the first aspect, the terminal has a communication function with the vehicle's self-load sensor; the vehicle's self-load sensor generates a vehicle self-load data set based on the collection of the vehicle's own information, and transmits it to the terminal. The terminal inputs the vehicle self-load data set into the vehicle prediction model to obtain the vehicle's self-load prediction trajectory, and performs trajectory fusion on the vehicle prediction trajectory and the vehicle self-load prediction trajectory, thereby improving the accuracy of the vehicle prediction trajectory and the collision prediction result.

[0028] Specifically, with the development of intelligent driving technology, more and more vehicles are equipped with intelligent assistance programs. While pedestrian feature information cannot be obtained through intelligent programs, vehicle feature information can be obtained through onboard intelligent assistance programs. Therefore, this method includes a data transmission interface between the terminal and the vehicle, which enables information communication between the vehicle and the terminal.

[0029] Furthermore, the vehicle prediction model also includes original weights and reference weights. The original weights are used to constrain the degree of influence of the vehicle data set generated by the imaging device according to the capture operation on the vehicle prediction trajectory and collision prediction results. The reference weights are used to constrain the degree of influence of the vehicle self-loaded data set generated by the vehicle self-loaded sensor according to the collection of the vehicle's own information on the vehicle prediction trajectory and collision prediction results.

[0030] Furthermore, initially, the weights of the two are the same. As time goes by, the original weight gradually increases as the results obtained by machine training become more and more accurate, thereby improving the accuracy of the vehicle predicted trajectory and risk collision results obtained by the terminal based on the capture operation.

[0031] In another embodiment of the first aspect, when the vehicle self-load sensor generates a vehicle self-load dataset, it can include a richer set of vehicle morphological parameters. In addition to the steering information, acceleration / deceleration information, and operating information contained in the vehicle dataset generated by the imaging device, it also includes tire pressure information, gear information, brake information, and throttle information. The brake information is used to constrain the operating information, and the tire pressure information, gear information, and throttle information are used to constrain the acceleration / deceleration information. The present invention improves the accuracy of the vehicle prediction trajectory and collision prediction results obtained by the vehicle prediction model through real-time data interaction between the terminal and the vehicle.

[0032] Furthermore, the terminal may also send the obtained pedestrian prediction trajectory, vehicle prediction trajectory and collision prediction result to a specific vehicle to assist the vehicle in risk prediction.

[0033] In a second aspect, the present invention provides a pedestrian and vehicle dynamic prediction device, the device comprising:

[0034] A generating unit, configured for the terminal to generate an original data set based on image information captured by an image device, wherein the original data set includes a pedestrian data set and a vehicle data set; the pedestrian data set includes pedestrian morphological parameters, pedestrian trajectory parameters, and a first time parameter; and the vehicle data set includes vehicle morphological parameters, vehicle trajectory parameters, and a second time parameter;

[0035] A determination unit is used to input the pedestrian data set into a pedestrian prediction model to obtain a pedestrian prediction trajectory corresponding to the pedestrian data set, input the vehicle data set into a vehicle prediction model to obtain a vehicle prediction trajectory corresponding to the vehicle data set, perform trajectory model fitting on the pedestrian prediction trajectory and the vehicle prediction trajectory, and judge the collision prediction results of the pedestrian and the vehicle, wherein the collision prediction results include collision or no collision; the pedestrian prediction model and the vehicle prediction model are both models trained based on multiple corresponding data set samples, corresponding prediction trajectories and corresponding collision prediction results; each data set sample in the data set sample includes a pedestrian morphological parameter, a pedestrian trajectory parameter and a first time parameter related to the pedestrian prediction model generated by a single capture operation, or a vehicle morphological parameter, a vehicle trajectory parameter and a second time parameter related to the vehicle prediction model generated by a single capture operation; the pedestrian data set and the vehicle data set are feature data, and the collision prediction result is label data.

[0036] Specifically, the device includes at least a generating unit and a determining unit. The generating unit is used to generate an original data set according to the capture operation of the image device, while the determining unit determines the collision prediction result of the pedestrian and the vehicle based on the original data set and the original data set sample data.

[0037] In yet another embodiment of the second aspect, the apparatus further includes a first acquisition unit and a first training unit.

[0038] The first acquisition unit is used to obtain the multiple pedestrian data sets or pedestrian data set samples, vehicle data sets or vehicle data set samples, and obtain pedestrian prediction trajectories, vehicle prediction trajectories and collision prediction results based on the corresponding data sets or data set samples.

[0039] The first training unit is used to train a pedestrian prediction model and a vehicle prediction model based on multiple pedestrian data sets or pedestrian data set samples, vehicle data sets or vehicle data set samples, pedestrian prediction trajectories, vehicle prediction trajectories and collision prediction results.

[0040] In fact, in this embodiment, the device itself has the ability to train models, and its pedestrian prediction model and vehicle prediction model as well as the simulation of the trajectories of the two models are obtained through self-training.

[0041] In another embodiment of the second aspect, the apparatus further includes a server and a cloud device.

[0042] The server at least includes a second acquiring unit, a second training unit and a sending unit.

[0043] The second acquisition unit is used to obtain the multiple pedestrian data sets or pedestrian data set samples, vehicle data sets or vehicle data set samples, and obtain pedestrian prediction trajectories, vehicle prediction trajectories and collision prediction results based on the corresponding data sets or data set samples.

[0044] The second training unit is used to train a pedestrian prediction model and a vehicle prediction model based on multiple pedestrian data sets or pedestrian data set samples, vehicle data sets or vehicle data set samples, pedestrian prediction trajectories, vehicle prediction trajectories and collision prediction results.

[0045] The sending unit is used to send the pedestrian prediction model and the vehicle prediction model to the terminal.

[0046] Specifically, when these devices are deployed in large numbers across a city, self-training alone will clearly be insufficient to meet actual usage needs. In this case, a server can be used to train initial pedestrian and vehicle prediction models, as well as to obtain collision prediction results through model fitting. These trained initial pedestrian and vehicle prediction models are then sent to intersections where these devices are deployed. Simultaneously, the server updates these initial pedestrian and vehicle prediction models based on data feedback from the different devices, thereby improving the accuracy of the devices.

[0047] The cloud device is used to provide cloud services.

[0048] Specifically, the cloud device primarily supports real-time interaction between the vehicle's own devices and the terminal. When a vehicle reaches an intersection equipped with the terminal, its onboard application or an external mobile device, such as a mobile phone, can package and send vehicle morphological parameters to the terminal via the cloud device. The terminal then transmits the predicted pedestrian and vehicle trajectories, as well as the predicted collision results, to the onboard application or external mobile device, such as a mobile phone, via the cloud device. This intelligently assists vehicle operation, effectively achieving intelligent linkage between the terminal and the vehicle and effectively preventing pedestrian-vehicle collisions.

[0049] In a third aspect, the present invention provides a terminal comprising at least one processor, a communication interface and a memory, wherein the communication interface is used to send and / or receive data, the memory is used to store computer programs, and the at least one processor is used to call a computer program stored in at least one memory. The terminal can execute the method described in the first aspect or any embodiment of the first aspect.

[0050] It should be noted that the processor included in the terminal described in the fifth aspect above may be a processor specifically used to execute these methods (referred to as a dedicated processor for ease of distinction), or may be a processor that executes these methods by calling a computer program, such as a general-purpose processor. Optionally, the at least one processor may include both a dedicated processor and a general-purpose processor.

[0051] Optionally, the computer program may be stored in a memory. Exemplarily, the memory may be a non-transitory memory, such as a read-only memory (ROM), which may be integrated with the processor on the same device or provided on separate devices. The embodiments of the present invention do not limit the type of memory or the configuration of the memory and the processor.

[0052] In a possible implementation manner, the at least one memory is located outside the terminal.

[0053] In another possible implementation, the at least one memory is located within the terminal.

[0054] In another possible implementation, part of the at least one memory is located inside the terminal, and another part of the memory is located outside the terminal.

[0055] In the present invention, the processor and the memory may also be integrated into one device, that is, the processor and the memory may also be integrated together.

[0056] In a fourth aspect, the present invention provides a computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, and when the computer program runs on a processor, the method described in the above-mentioned first aspect or any embodiment of the first aspect is implemented.

[0057] Optionally, the computer program product may be a software installation package. When the aforementioned method is required, the computer program product may be downloaded and executed on a computing device. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The present invention will be described in further detail below with reference to the accompanying drawings and preferred embodiments. However, those skilled in the art will appreciate that these drawings are drawn only for the purpose of explaining the preferred embodiments and should not be construed as limiting the scope of the present invention. Furthermore, unless otherwise specified, the drawings are merely schematic representations of the composition or structure of the depicted objects and may contain exaggerated representations. Furthermore, the drawings are not necessarily drawn to scale.

[0059] Figure 1A schematic diagram of the architecture of a method for predicting pedestrian and vehicle dynamics provided by an embodiment of the present invention;

[0060] Figure 2 A flowchart of a method for predicting pedestrian and vehicle dynamics provided by an embodiment of the present invention;

[0061] Figure 3 A schematic diagram of a server architecture for implementing a method for predicting pedestrian and vehicle dynamics provided by an embodiment of the present invention;

[0062] Figure 4 A schematic diagram of a cloud device architecture for implementing a method for predicting pedestrian and vehicle dynamics provided by an embodiment of the present invention;

[0063] Figure 5 Another flowchart of a method for predicting pedestrian and vehicle dynamics provided by an embodiment of the present invention;

[0064] Figure 6 A schematic diagram of a collision prediction device for implementing a dynamic prediction method for pedestrians and vehicles provided by an embodiment of the present invention;

[0065] Figure 7 A schematic diagram of a collision training device for implementing a pedestrian and vehicle dynamic prediction method provided by an embodiment of the present invention;

[0066] Figure 8 An embodiment of the present invention provides a terminal for implementing a method for predicting pedestrian and vehicle dynamics;

[0067] Figure 9 An embodiment of the present invention provides a server for implementing a method for predicting pedestrian and vehicle dynamics;

[0068] Figure 10 An embodiment of the present invention provides a cloud device for implementing a method for predicting the dynamics of pedestrians and vehicles. DETAILED DESCRIPTION

[0069] The following is combined with Figures 1 to 10 , the present invention is described in detail.

[0070] The terms "first," "second," "third," and "fourth," etc., in the specification, claims, and accompanying drawings of the present invention are used to distinguish between different items, not to describe a specific order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements, but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0071] The following describes the system architecture used in the embodiments of the present invention. It should be noted that the system architecture and business scenarios described in the present invention are intended to more clearly illustrate the technical solutions of the present invention and do not constitute a limitation on the technical solutions provided by the present invention. Persons skilled in the art will appreciate that, as the system architecture evolves and new business scenarios emerge, the technical solutions provided by the present invention will also be applicable to similar technical problems.

[0072] See also Figure 1 As shown, Figure 1 A schematic diagram of an architecture for implementing a method for predicting pedestrian and vehicle dynamics provided by an embodiment of the present invention includes an imaging device 101 and a terminal 102.

[0073] Imaging devices 101 are deployed in a blanket configuration based on the characteristics of the intersection, capturing images of all pedestrians and vehicles passing through the intersection. Specifically, imaging devices 101 are comprised of two or more physical entities, enabling the collection of image information for the applicable road section. It is worth noting that, to accommodate both daytime and nighttime scenarios, imaging devices 101 can be high-definition cameras with infrared cameras, or they can be replaced with other devices capable of capturing pedestrian and vehicle information, such as radar or sensors.

[0074] Terminal 102 is a computing device capable of receiving and processing data. It can receive and process data information transmitted from imaging device 101. Generally speaking, terminal 102 converts the image information captured by imaging device 101 into raw data sets, namely pedestrian and vehicle data sets. It then inputs these data sets into corresponding pedestrian and vehicle prediction models to obtain corresponding pedestrian and vehicle prediction trajectories. Finally, all predicted trajectories are fitted in a standard spatial coordinate system to obtain collision prediction results.

[0075] It is worth noting that, in order to optimize the computing processing capability of the terminal 102 , the processing operation of generating the original data set from the image information can be directly completed in the imaging device 101 , and the imaging device 101 directly inputs the processed original data set to the terminal 102 .

[0076] For pedestrian datasets, including pedestrian morphological parameters {S sex , S age , S h , S co}、Pedestrian trajectory parameters {x 人 ,y 人 , v 人 , a 人} and the first time parameter s 人 , the pedestrian prediction model generates a pedestrian prediction model based on the input of multiple pedestrian datasets or pedestrian dataset samples.人 A series of pedestrian prediction trajectory coordinates (x 人1 ,y 人 1, a 人1 , v person 1, s person 1), (x person 2, y person 2, a person 2, v person 2, s person 2), ..., (x person i , y people i, a people i , v people i ,s people i ), ..., the pedestrian predicted trajectory is obtained through a series of trajectory coordinates.

[0077] Among them, in the pedestrian prediction model, the unit time s 单位 , the difference between adjacent first times is one unit time s 单位 .x 人i For pedestrians on i*s 单位 Predicted x coordinates and y coordinates after time 人 i is the pedestrian at i*s 单位 The predicted y coordinate after time, a 人i For pedestrians on i*s 单位 Prediction angle after time, v 人i For pedestrians on i*s 单位 Predicted first speed after time, s 人i For pedestrians on i*s 单位 The first time parameter after time, s 人i =s 人 +i*s 单位 .

[0078] In addition, the terminal 101 will also store the actual trajectory information of the pedestrian for training and updating after the terminal 101 is used for a long time.

[0079] For the vehicle dataset, including vehicle shape parameters {S ch , S sd , S cc}, vehicle trajectory parameters {x 车 ,y 车 , V2, a 车} and the second time parameter s 车 The vehicle prediction model generates a vehicle prediction model according to the input of multiple vehicle data sets or vehicle data set samples. 车 A series of vehicle prediction trajectory coordinates (x 车1 ,y 车 1, a 车1 , v 车1 , s 车1 )、(x 车2 ,y 车 2, a 车2 , v车2 , s 车2 ),……、(x 车i ,y 车 i, a 车i , v 车i , s 车i ), ..., the pedestrian predicted trajectory is obtained through a series of trajectory coordinates.

[0080] Among them, S ch For steering information, S sd is the acceleration and deceleration information, S cc is the operation information. At the same time, in the vehicle prediction model, the same unit time s as the pedestrian prediction model is included 单位 , the difference between adjacent second times is one unit time s 单位 .x 车i For vehicles in i*s 单位 Predicted x coordinates and y coordinates after time 车 i is the vehicle at i*s 单位 The predicted y coordinate after time, a 车i For vehicles in i*s 单位 Prediction angle after time, v 车i For vehicles in i*s 单位 Predicted second speed after time, s 车i For vehicles in i*s 单位 The second time parameter after time, s 车i =s 车 +i*s 单位 .

[0081] In addition, the terminal 101 will also store the actual trajectory information of the pedestrian for training and updating after the terminal 101 is used for a long time.

[0082] After the terminal 101 is placed at the corresponding intersection, it will generate a standard spatial coordinate system (x, y) according to the environment of the intersection. The pedestrian prediction model and the vehicle prediction model will generate the corresponding pedestrian prediction trajectory and vehicle prediction trajectory according to the standard spatial coordinate system, and then convert a series of (x 人i ,y 人 i, a 人i , s 人i ) and a series of (x 车i ,y 车 i, a car i, s car i ) is subjected to trajectory fitting in the terminal 101, thereby eliminating the systematic errors between pedestrians and vehicles in the trajectory fitting process through the absolute reference of the standard space coordinate system.

[0083] It's worth noting that Terminal 101 simulates more than just collision prediction results for a single pedestrian or vehicle in the standard spatial coordinate system. Rather, it assigns a corresponding data label to all pedestrians and vehicles passing through the intersection where Terminal 101 is located. For example, pedestrians are differentiated by gender, height, age, clothing, and other characteristics, and given different data labels. Vehicles are differentiated by license plate, color, model, and other characteristics, and given different data labels.

[0084] That is, the pedestrian data set or pedestrian data sample set includes all pedestrian data labels {pedestrian 1, pedestrian 2, pedestrian 3, ...} passing through the intersection, and each pedestrian data label includes the corresponding pedestrian morphological parameters {S sex , S age , S h , S co}、Pedestrian trajectory parameters {x 人 ,y 人 , v 人 , a 人} and the first time parameter s 人 , and then generate all pedestrian prediction trajectories under all pedestrian data labels; the vehicle data set or vehicle data sample set includes all vehicle data labels {vehicle 1, vehicle 2, vehicle 3, ...} passing through the intersection, and each vehicle data label includes the corresponding vehicle shape parameters {S ch , S sd , S cc}, vehicle trajectory parameters {x 车 ,y 车 , V2, a 车} and the second time parameter s 车 , and then generate predicted trajectories for all vehicles under all vehicle data tags. By predicting the full trajectories of pedestrians and vehicles passing through the intersection set by terminal 101, all factors affecting the collision prediction results are processed and calculated, thereby making the predicted pedestrian and vehicle trajectories and collision prediction results more accurate and comprehensive.

[0085] See also Figure 2 As shown, Figure 2 A flowchart of a method for predicting pedestrian and vehicle dynamics provided by an embodiment of the present invention.

[0086] Terminal 101 will obtain the pedestrian prediction trajectory corresponding to the pedestrian data set generated under each pedestrian data label according to steps S210 and S211, and obtain the vehicle prediction trajectory corresponding to the vehicle data set generated under each vehicle data label according to steps S220 and S221. Finally, according to step S230, all pedestrian prediction trajectories and vehicle prediction trajectories will be fitted in the standard space coordinate system (x, y) to determine the collision prediction result of the pedestrian or vehicle.

[0087] In another possible embodiment, before obtaining the collision prediction result according to step S230, the terminal 101 assigns a priority for processing each pedestrian data tag and vehicle data tag.

[0088] Specifically, at intersections in complex, special application scenarios such as schools, campuses, or shopping malls, the volume of pedestrian and vehicle data labels acquired by terminal 101 during a certain period will be very large. For example, during peak hours of school arrival and departure or during peak shopping hours at shopping malls on weekends. To optimize the data processing capabilities of terminal 101, a priority system is built into terminal 101. This priority set system includes at least a first priority and a second priority. The first priority is used to promptly process data information on pedestrians and vehicles that are about to collide; the second priority is used to perform routine processing on predicted pedestrian trajectories, predicted vehicle trajectories, and collision prediction results.

[0089] Among them, the first priority is determined according to the distance between the pedestrian and the vehicle. When the distance between the pedestrian and the vehicle is within 10m (including 10m), the pedestrian and vehicle pair is upgraded to the first priority and is preferably processed; when the distance between the pedestrian and the vehicle is greater than 10m, the pedestrian and vehicle pair is the second priority.

[0090] Furthermore, both the first and second priority levels include a priority coefficient, which is used to sort pedestrian and vehicle data pairs within different priority sets. For example, in the first priority level, although both are in a collision-prone state within 10 meters, the risks posed by different vehicle angles and pedestrian states are different.

[0091] Therefore, the terminal 101 includes a priority coefficient calculation formula: N = n a *n b *n c *1. Where N is the final priority coefficient, n a is the relative vehicle angle coefficient, n b is the pedestrian state coefficient, n c is the pedestrian age coefficient. For reference of each coefficient, please see the table below.

[0092] Table 1

[0093]

[0094] Table 2

[0095]

[0096] Table 3

[0097]

[0098] In fact, although the closer the distance between pedestrians and vehicles, the higher the risk, the results vary greatly at different vehicle positions. Taking Table 1 as an example, the risk factor is highest at -50° to 50° directly in front of the vehicle. The first reason is that when directly in front of the vehicle, the driver is prone to nervousness and misjudgment. In addition, the blind spots on both sides of the vehicle will reduce the driver's probability of foreseeing in advance, which can easily lead to collision accidents. At 50° to 150° directly in front of the vehicle; -150° to -50°, that is, at the side of the vehicle, the risk is lower than that at the front and blind spots, and generally only occurs when the driver does not check the rearview mirror or suddenly turns; and at 150° to -150° directly in front of the vehicle, that is, at the rear of the vehicle, the risk is lowest.

[0099] It is worth understanding that the angle is only used to refer to the three states of the front, body and rear of the vehicle, and is adjusted according to the size and shape of the vehicle. For example, for large trucks, the blind spot of vision is larger, and the definition of the front angle should be larger.

[0100] Table 2 mainly defines the impact of different pedestrian movement modes on danger. Due to the development of today's food delivery business, many electric vehicles have large trajectory changes and fast speeds during movement, which can easily affect the driver's judgment and cause collision accidents. Its priority is the highest, while other movement modes are ranked lower according to speed.

[0101] Table 3 mainly defines the risk level coefficient for the stage of life of pedestrians. Children are mentally immature and active, and are prone to accidents if their guardians neglect them. They have the highest priority, followed by slow-moving elderly people and more impulsive teenagers. Therefore, the risk level is preset according to the age status of pedestrians.

[0102] Finally, the product of the three is used to obtain the priority coefficient for the current priority state, thereby better preventing collision accidents. For example, when the distances between the data tag pairs {pedestrian x1, vehicle y}, {pedestrian x2, vehicle y}, and {pedestrian x3, vehicle y} are all within 10 meters, they are given the first priority. The specific status is shown in Table 4 below. The priority coefficient N1 of {pedestrian x1, vehicle y} is N2=2*1*1=2; the priority coefficient N2 of {pedestrian x2, vehicle y} is N2=1.5*2*2=6; the priority coefficient N3 of {pedestrian x3, vehicle y} is N3=1*1.5*1.5=2.25. Then, when they all belong to the first priority, the terminal processes the data tag pair {pedestrian x2, vehicle y} first, followed by the data tag pair {pedestrian x3, vehicle y}, and finally the data tag pair {pedestrian x1, vehicle y}. This can effectively deal with the problem of delay or deviation in the pedestrian and vehicle collision prediction results when the terminal 101 has a large operating load due to a large flow of people, and timely process and warn pedestrian and vehicle data tag pairs with higher priorities and priority coefficients, thereby reducing the occurrence of pedestrian and vehicle collision accidents.

[0103] Table 4

[0104] Angle relative to the front of the vehicle (°) Running status age {pedestrian x1, vehicle y} 30 walk 25 {pedestrian x2, vehicle y} 80 electric vehicles 10 {pedestrian x3, vehicle y} -160 bike 65

[0105] In another possible embodiment, as shown in Table 5, the pedestrian dataset includes a first weight, a second weight, and a third weight, which are used to respectively constrain the influence of the pedestrian's morphological parameters, the pedestrian's trajectory parameters, and the first time parameter on the pedestrian's predicted trajectory and collision prediction result. Initially, the first weight is the highest and gradually decreases over time.

[0106] Table 5

[0107] Duration of use First weight Second weight The third weight Half a year 60％ 20％ 20％ One year 50％ 25％ 25％ Two years 40％ 30％ 30％ Three years 30％ 35％ 35％

[0108] Specifically, the pedestrian prediction model uses a weighted ratio to improve the accuracy of the pedestrian prediction trajectory and collision prediction results during the continuous training and learning process. During initial use, the pedestrian prediction model has few inputs into the morphological parameters of pedestrians passing through the road section, resulting in low judgment accuracy. Initially, a 60% weight is used to constrain its influence on the pedestrian prediction trajectory and collision prediction results, thereby improving the versatility of terminal 101. As terminal 101 is used for a longer period of time, the pedestrian prediction model records sufficient information on pedestrian morphological parameters and gradually adapts to the travel patterns and age structure of pedestrians passing through the road section. Its weight also decreases accordingly, thereby improving the accuracy of terminal 101 for special scenarios, so as to cope with different special intersections such as schools, parks, shopping malls, and complex intersections, making the pedestrian prediction trajectory and collision prediction results of terminal 101 for different special sections more accurate and targeted.

[0109] It is worth mentioning that in another possible embodiment, when the number of collisions on the road section applied by terminal 101 is too many, the second weight will be increased accordingly, thereby increasing the influence of pedestrian trajectory parameters on pedestrian predicted trajectory and collision prediction results, and to a certain extent improving the accuracy of pedestrian predicted trajectory and collision prediction results.

[0110] In another possible embodiment, the correction coefficient of the pedestrian morphological parameters provided by the present invention is used to correct possible speed changes and direction changes in the predicted trajectory of the pedestrian.

[0111] The pedestrian data set or pedestrian data sample set includes a standard stable speed table, where the standard stable speed corresponding to the pedestrian morphological parameters {male, 25, 170} is 1.5m / s. Specifically, pedestrians passing through the intersection are large in size, with many combinations of gender, age, height, and running status. In order to reduce the impact of these uncertain factors on the pedestrian prediction trajectory and collision prediction results, the pedestrian prediction model will be modified according to the formula V 稳i =S sexi *Sagei *S hi *S coi 100% V 标 Get the stable speed of pedestrians under different data labels, where V 稳i is the stable speed of the corrected pedestrian data label {pedestrian i}, S sexi The pedestrian data label is {pedestrian i}’s gender, S agei is the age of the pedestrian data label {pedestrian i}, S hi is the height of the pedestrian data label {pedestrian i}, S coi is the running state of the pedestrian data label {pedestrian i}, V 标 is the standard running speed. Table 6 is a comparison table of some correction parameters. For example, when the pedestrian morphology parameters of {pedestrian i} are {male, 35, 180, walking}, V 稳i =1*0.95*1.1*1*100%*1.5m / s=1.5675m / s; When the pedestrian morphology parameters of {pedestrian j} are {female, 15, 160, electric vehicle}, V 稳j =0.9*0.9*0.9*2*100%*1.5m / s=1.187m / s.

[0112] Table 6

[0113] gender <![CDATA[S sex ]]> age <![CDATA[S age ]]> height <![CDATA[S h ]]> Running status <![CDATA[S co ]]> male 1 35 0.95 180 1.1 walk 1 female 0.9 15 0.9 160 0.9 electric vehicles 2

[0114] Furthermore, S co In addition to walking, running, cycling, etc., it also includes stopping state, avoiding state, etc., that is, S co ={V 方式 , V 状态}, V 状态 We will find V 稳i Speed ​​has an impact. This technical solution is based on V 状态 The influence coefficient is set, where the stop state is 0, the stand state is 0.3, the avoid state is 0.7, the overtaking state is 1.25, and the stable state is 1. For example, when {pedestrian j} is in the avoidance state, its corresponding first speed will be expected to be reduced to V according to the formula 避j =1.187m / s*0.75=0.89025m / s.

[0115] Furthermore, S co ={V 方式 , V 避让 The corresponding speedometer also includes the steering angle a 修 Information is used to constrain the degree of influence of the avoidance state on the first direction.

[0116] It is worth noting that the terminal 101 can also predict the pedestrian's upcoming avoidance state and avoidance angle based on different pedestrian predicted trajectories, thereby making the dynamics of the pedestrian's predicted trajectory more accurate.

[0117] In another possible embodiment, the vehicle dataset includes a fourth weight, a fifth weight, and a sixth weight, respectively constraining the influence of the vehicle's morphological parameters, the vehicle's operating state, and the second time parameter on the predicted vehicle trajectory and collision prediction results. Initially, the weights are 35%, 35%, and 30%, respectively. As the vehicle prediction model is used for an extended period, the proportion of the fifth weight gradually increases to adapt to the driving conditions of the vehicles at the intersection.

[0118] Table 7

[0119] Duration of use First weight Second weight The third weight Half a year 35％ 35％ 30％ One year 30％ 40％ 30％ Two years 25％ 45％ 40％ Three years 25％ 50％ 25％

[0120] In another possible embodiment, the pedestrian prediction model and the vehicle prediction model may adopt basic network models such as convolutional neural network (CNN) and long short-term memory artificial neural network (LSTM). Convolutional neural network usually includes: input layer, convolution layer (Convolution Layer), pooling layer (Pooling layer), fully connected layer (Fully Connected Layer, FC) and output layer. Generally speaking, the first layer of convolutional neural network is the input layer and the last layer is the output layer. Convolution layer (Convolution Layer) refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolution layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layers. A convolution layer usually contains several feature planes, and each feature plane can be composed of some neural units arranged in a rectangular shape. The neural units in the same feature plane share weights, and the shared weights here are the convolution kernels. The pooling layer, typically after the convolutional layer, generates large-dimensional features. It then divides the features into several regions and takes the maximum or average value to generate new, smaller-dimensional features. The fully-connected layer combines all local features into global features, which are then used to calculate the final score for each class. A long-short term memory (LSTM) neural network typically consists of an input layer, a hidden layer, and an output layer. The input layer consists of at least one input node. When the LSTM network is unidirectional, the hidden layer only includes the forward hidden layer; when the LSTM network is bidirectional, the hidden layer includes both the forward hidden layer and the backward hidden layer. Each input node is connected to a node in the forward hidden layer and a node in the backward hidden layer, outputting input data to these nodes, respectively. Each hidden node in the hidden layer is connected to an output node, outputting its own computational results to the output node. The output node performs computations based on the output nodes of the hidden layer and outputs data.

[0121] See also Figure 3 As shown, a schematic diagram of a server architecture for implementing a method for dynamic prediction of pedestrians and vehicles provided by an embodiment of the present invention.

[0122] Specifically, as the number of terminals 101 increases, relying solely on terminals 101 for model training and updating will result in insufficient interactivity and real-time performance between different terminals 101. The present invention also provides a schematic diagram of a server architecture. Initially, server 103 trains corresponding pedestrian and vehicle prediction models based on pedestrian and vehicle dataset samples. The trained models are then distributed to terminals 102a, 102b, and 102c at different locations. After prolonged use, these terminals feed back real-time pedestrian and vehicle datasets to server 103. Server 103 continuously retrains the data transmitted from different terminals to obtain more complete pedestrian and vehicle prediction models, which are then distributed to different terminals 102a, 102b, and 102c. This allows for updates and maintenance of the pedestrian and vehicle prediction models, resulting in higher accuracy in the collision prediction results of terminals 102a, 102b, and 102c. This also optimizes the overall organizational structure and accelerates processing speed.

[0123] See also Figure 4 As shown, a schematic diagram of a cloud device architecture for implementing a method for dynamic prediction of pedestrians and vehicles provided by an embodiment of the present invention.

[0124] Specifically, in order to improve the interactive experience between the vehicle 105 and the terminal 102, the cloud device 104 is introduced. When the terminal 102 is processing the vehicle's predicted trajectory, it will receive the information obtained by the sensors on board the vehicle 105 to generate a vehicle data set. Since the imaging device 101 is limited by the external information acquisition method, it cannot obtain the internal data information of the vehicle 105. The vehicle 105 can upload its own information to the terminal 102 through the cloud device 104. Compared with the previous single information acquisition method, the terminal 102 will be able to obtain more abundant vehicle morphological parameters. The vehicle morphological parameters obtained by the vehicle 105 in the present invention are {S 转向 , S 加减速 , S 运行 , S 胎压 , S 挡位 , S 刹车 , S 油门}, and predict the vehicle trajectory according to the obtained vehicle morphological parameters, and then perform trajectory fusion processing on the vehicle predicted trajectory a obtained by the terminal 102 based on the vehicle data set generated by the imaging device 101 and the vehicle predicted trajectory b obtained based on the vehicle data set uploaded by the vehicle's own sensor through weighting, thereby making the vehicle predicted trajectory more accurate.

[0125] It is worth understanding that the terminal 102 or the server 103 can have two different vehicle prediction models built in. One set of vehicle data sets generated by the image device is placed in the corresponding vehicle prediction model to obtain the vehicle prediction trajectory, and the other set of vehicle data sets generated by the vehicle's own sensors is placed in the corresponding vehicle prediction model to obtain the vehicle prediction trajectory.

[0126] In addition, after the terminal 102 generates the corresponding vehicle prediction trajectory and collision prediction results, it will also be uploaded to the vehicle 105 through the cloud device 104, so that the vehicle can adjust its driving speed and driving angle in time according to the information uploaded by the terminal 102 to avoid collisions between the vehicle and pedestrians, thereby improving the intelligence of the vehicle and reducing the risk of collision.

[0127] See also Figure 5 As shown, another flow chart of a method for realizing dynamic prediction of pedestrians and vehicles provided by an embodiment of the present invention.

[0128] Specifically, after receiving the vehicle dataset generated by the vehicle's own sensors, the cloud device sends it to the terminal through step S301. The terminal sends the pedestrian dataset and vehicle dataset generated by the imaging device and the vehicle dataset generated by the vehicle's own sensors to the server through step S302. The server receives the data through steps S303 and S304, and respectively puts them into the corresponding pedestrian prediction model and vehicle prediction model for training, obtains the updated pedestrian prediction model and vehicle prediction model, and distributes them to the terminal. The terminal accepts the updated pedestrian prediction model and vehicle prediction model trained by the server. Through the above implementation, the terminal's pedestrian prediction model and vehicle prediction model will be linked to the cloud device and server for real-time update, thereby improving the accuracy of the obtained pedestrian prediction trajectory and vehicle prediction trajectory.

[0129] The above describes in detail the method according to the embodiment of the present invention. The following provides an apparatus according to the embodiment of the present invention.

[0130] See also Figure 6 As shown, Figure 6 1 is a structural diagram of a false touch judgment device 40 based on machine learning provided in an embodiment of the present invention. The device 40 may be the terminal mentioned above or a device in the terminal. The device 40 may include a generation unit 401 and a determination unit 402, wherein each unit is described in detail as follows.

[0131] Specifically, the generating unit 401 is used for the terminal to generate an original data set based on the image information captured by the imaging device, wherein the original data set includes a pedestrian data set and a vehicle data set. The generating unit 401 can be integrated into the terminal or directly integrated into the imaging device.

[0132] Determination unit 402 is configured to input the pedestrian dataset into a pedestrian prediction model to obtain a corresponding pedestrian prediction trajectory; input the vehicle dataset into a vehicle prediction model to obtain a corresponding vehicle prediction trajectory; perform trajectory fitting based on the pedestrian and vehicle prediction trajectories obtained by the pedestrian and vehicle prediction models, and determine a collision prediction result for the pedestrian and vehicle, where the collision prediction result includes collision or non-collision; the dataset is feature data, and the collision prediction result is label data. Determination unit 402 is primarily provided in the terminal.

[0133] In another possible embodiment, the device 40 further includes a receiving unit configured to receive the pedestrian prediction model and vehicle prediction model sent by a server. In this device 40, the terminal's false touch judgment model is sent by the server. Specifically, because the terminal itself cannot perform model training, the server sends the trained model to the terminal. Optionally, the terminal returns the original data set generated by its touch screen to the server for updating the false touch judgment model.

[0134] In another possible implementation, the device 40 further includes:

[0135] A first acquisition unit is configured to acquire the plurality of pedestrian data set samples, vehicle data set samples, and pedestrian predicted trajectories, vehicle predicted trajectories, and collision prediction results obtained from the corresponding data set samples;

[0136] The first training unit is used to train a pedestrian prediction model and a vehicle prediction model based on multiple pedestrian data set samples, vehicle data set samples, pedestrian prediction estimates, vehicle prediction trajectories and collision prediction results.

[0137] In the present device 40 , the terminal itself has the capability of training the model.

[0138] See also Figure 7 As shown, Figure 7 1 is a structural diagram of a collision training device 50 based on machine learning provided by an embodiment of the present invention. The device 50 may be the aforementioned server or a device in the server. The device 50 may include a second acquisition unit 501, a second training unit 502 and a sending unit 503, wherein each unit is described in detail as follows.

[0139] The second acquisition unit 501 is used to acquire the multiple pedestrian data set samples, vehicle data set samples, and pedestrian predicted trajectories, vehicle predicted trajectories and collision prediction results obtained from the corresponding data set samples.

[0140] The second training unit 502 is used to train a pedestrian prediction model and a vehicle prediction model based on multiple pedestrian data set samples, vehicle data set samples, pedestrian prediction estimates, vehicle prediction trajectories and collision prediction results.

[0141] The sending unit 503 is configured to send a pedestrian prediction model and a vehicle prediction model to the terminal.

[0142] Specifically, the collision training device is used for training and distributing pedestrian prediction devices and vehicle prediction devices, thereby improving the accuracy of collision prediction results.

[0143] See also Figure 8 As shown, Figure 8 6 is a schematic diagram of the structure of a terminal 60 provided in an embodiment of the present invention. The terminal 60 includes a processor 601, a communication interface 602, and a memory 603. The processor 601, the communication interface 602, and the memory 603 may be connected via a bus or other means. The embodiment of the present invention uses a bus connection as an example.

[0144] The processor 601 is the computing and control core of the terminal 60, capable of parsing various instructions and data within the terminal 60. For example, the processor 601 may be a central processing unit (CPU), capable of transmitting various interactive data between the internal components of the terminal 60, etc. The communication interface 602 may optionally include a standard wired interface or a wireless interface (such as Wi-Fi or a mobile communication interface), which can be used to send and receive data under the control of the processor 601. The communication interface 602 may also be used for the transmission and interaction of signaling or instructions within the terminal 60. The memory 603 (Memory) is a storage device within the terminal 60, used to store programs and data. It is understood that the memory 603 herein may include both the terminal 60's built-in memory and, of course, the terminal 60's supported extended memory. The memory 603 provides storage space, which stores the terminal 60's operating system, the program code or instructions required for the processor to perform corresponding operations, and optionally, the relevant data generated by the processor after performing the corresponding operations.

[0145] See also Figure 9 As shown, Figure 9 7 is a schematic diagram of the structure of a server 70 provided in an embodiment of the present invention. The server 70 includes a processor 701, a communication interface 702, and a memory 703. The processor 701, the communication interface 702, and the memory 703 may be connected via a bus or other means. The embodiment of the present invention uses a bus connection as an example.

[0146] Among them, the processor 701 is the computing core and control core of the server 70, which can parse various instructions in the server 70 and various data of the server 70. For example, the processor 701 can be a central processing unit (CPU), which can transmit various interactive data between the internal structures of the server 70, and so on. The communication interface 702 can optionally include a standard wired interface, a wireless interface (such as WI-FI, a mobile communication interface, etc.), which can be used to send and receive data under the control of the processor 701; the communication interface 702 can also be used for the transmission and interaction of internal signaling or instructions of the server 70. The memory 703 (Memory) is a memory device in the server 70 for storing programs and data. It can be understood that the memory 703 here can include the built-in memory of the server 70, and of course it can also include the extended memory supported by the server 70. The memory 703 provides a storage space that stores the operating system of the server 70, which may include but is not limited to: Android system, iOS system, Windows Phone system, etc. The present invention is not limited to this. The storage space also stores the program code or instructions required for the processor to perform the corresponding operation. Optionally, the storage space can also store related data generated after the processor performs the corresponding operation.

[0147] See also Figure 10 As shown, Figure 10 8 is a schematic diagram of the structure of a cloud device 80 provided in an embodiment of the present invention. The server 80 includes a processor 801, a communication interface 802, and a memory 803. The processor 801, the communication interface 802, and the memory 803 may be connected via a bus or other means. The embodiment of the present invention uses a bus connection as an example.

[0148] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the computer program is executed by a processor, the processor implements Figure 2 and Figure 5 The operations performed by the terminal in the embodiment, or the implementation of Figure 2 and Figure 5 The operations performed by the server in the embodiment described above.

[0149] The embodiment of the present invention further provides a computer program product, which, when executed on a processor, implements Figure 2 and Figure 5 The operations performed by the terminal in the embodiment, or the implementation of Figure 2 and Figure 5 The operations performed by the server in the embodiment described above.

[0150] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a program. The program can be stored in a computer-readable storage medium. When executed, the program can include the processes in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0151] The present invention has been described in detail above. Specific examples have been used in the present invention to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only intended to help understand the present invention and its core concepts. It should be noted that those skilled in the art may make various improvements and modifications to the present invention without departing from the principles of the present invention, and such improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A method for predicting pedestrian and vehicle dynamics, characterized in that: The method is applied to a terminal, and includes: The terminal generates an original data set based on image information obtained by a capture operation of an imaging device, wherein the original data set includes a pedestrian data set and a vehicle data set; the pedestrian data set includes pedestrian morphological parameters, pedestrian trajectory parameters, and a first time parameter; the vehicle data set includes vehicle morphological parameters, vehicle trajectory parameters, and a second time parameter; The pedestrian data set is input into the pedestrian prediction model to obtain the pedestrian prediction trajectory corresponding to the pedestrian data set, and the vehicle data set is input into the vehicle prediction model to obtain the vehicle prediction trajectory corresponding to the vehicle data set. The pedestrian prediction trajectory and the vehicle prediction trajectory are fitted with trajectory models to judge the collision prediction results of pedestrians and vehicles, and the collision prediction results include collision or no collision; the pedestrian prediction model and the vehicle prediction model are both models trained according to multiple corresponding data set samples, corresponding prediction trajectories and corresponding collision prediction results; each data set sample in the data set sample includes a pedestrian morphological parameter, a pedestrian trajectory parameter and a first time parameter related to the pedestrian prediction model generated by a single capture operation, or a vehicle morphological parameter, a vehicle trajectory parameter and a second time parameter related to the vehicle prediction model generated by a single capture operation, the pedestrian data set and the vehicle data set are feature data, and the collision prediction result is label data; The pedestrian data set and the pedestrian data sample set both include: a first weight, a second weight, and a third weight; the first weight is used to constrain the degree of influence of the pedestrian morphological parameters on the pedestrian predicted trajectory and the collision prediction result; the second weight is used to constrain the degree of influence of the pedestrian trajectory parameters on the pedestrian predicted trajectory and the collision prediction result; the third weight is used to constrain the degree of influence of the first time parameter on the pedestrian predicted trajectory and the collision prediction result; Initially, the first weight is the highest and gradually decreases over time; as the number of collisions on the terminal application section increases, the second weight increases accordingly.

2. The pedestrian and vehicle dynamic prediction method according to claim 1, characterized in that: The pedestrian morphological parameters include gender, age, height and running status, and the running status includes walking, running, bicycle and electric vehicle, and one of the corresponding stopping state, standing state, avoiding state, overtaking state or stable state; the pedestrian trajectory parameters include the first spatial coordinate, the first speed and the first direction.

3. The pedestrian and vehicle dynamic prediction method according to claim 2, characterized in that: The pedestrian data set and the pedestrian data sample set both include: a first correction parameter, a second correction parameter, a third correction parameter and a fourth correction parameter; The first correction parameter is used to constrain the influence of gender on the first speed, the second correction parameter is used to constrain the influence of age on the first speed, the third correction parameter is used to constrain the influence of height on the first speed, and the fourth correction parameter is used to constrain the influence of the operating state on the first speed; The first correction parameter, the second correction parameter, the third correction parameter and the fourth correction parameter are adjusted as the usage time of the terminal increases.

4. The pedestrian and vehicle dynamic prediction method according to claim 1, characterized in that: The vehicle morphology parameters include steering information, acceleration / deceleration information, and operation information, wherein the operation information includes one of parking, overtaking, and normal driving; the vehicle trajectory parameters include a second spatial coordinate, a second speed, and a second direction.

5. The pedestrian and vehicle dynamic prediction method according to claim 4, characterized in that: The vehicle data set and the vehicle data sample set both include: a fourth weight, a fifth weight, and a sixth weight; The fourth weight is used to constrain the degree of influence of the vehicle shape parameters on the vehicle predicted trajectory and collision prediction results; the fifth weight is used to constrain the degree of influence of the vehicle operating state on the vehicle predicted trajectory and collision prediction results; the sixth weight is used to constrain the degree of influence of the second time parameter on the vehicle predicted trajectory and collision prediction results.

6. The pedestrian and vehicle dynamic prediction method according to any one of claims 1 to 5, characterized in that: The terminal has a communication function with the vehicle's own sensor; The vehicle self-load sensor generates a vehicle self-load data set based on the collection of vehicle self-load information and transmits it to the terminal. The terminal inputs the vehicle self-load data set into the vehicle prediction model to obtain the vehicle self-load prediction trajectory, and performs trajectory fusion on the vehicle prediction trajectory and the vehicle self-load prediction trajectory.

7. A pedestrian and vehicle dynamic prediction device, used to execute the pedestrian and vehicle dynamic prediction method according to any one of claims 1 to 6, characterized in that: The device comprises: A generating unit, configured for the terminal to generate an original data set based on image information captured by an image device, wherein the original data set includes a pedestrian data set and a vehicle data set; the pedestrian data set includes pedestrian morphological parameters, pedestrian trajectory parameters, and a first time parameter; and the vehicle data set includes vehicle morphological parameters, vehicle trajectory parameters, and a second time parameter; A determination unit is used to input the pedestrian data set into a pedestrian prediction model to obtain a pedestrian prediction trajectory corresponding to the pedestrian data set, input the vehicle data set into a vehicle prediction model to obtain a vehicle prediction trajectory corresponding to the vehicle data set, perform trajectory model fitting on the pedestrian prediction trajectory and the vehicle prediction trajectory, and judge the collision prediction results of the pedestrian and the vehicle, wherein the collision prediction results include collision or no collision; the pedestrian prediction model and the vehicle prediction model are both models trained based on multiple corresponding data set samples, corresponding prediction trajectories and corresponding collision prediction results; each data set sample in the data set sample includes a pedestrian morphological parameter, a pedestrian trajectory parameter and a first time parameter related to the pedestrian prediction model generated by a single capture operation, or a vehicle morphological parameter, a vehicle trajectory parameter and a second time parameter related to the vehicle prediction model generated by a single capture operation; the pedestrian data set and the vehicle data set are feature data, and the collision prediction result is label data.

8. A terminal, characterized in that: The terminal includes a processor, a communication interface and a memory, the communication interface is used to send and / or receive data, the memory is used to store computer programs, and the processor is used to call a computer program stored in the memory to implement the pedestrian and vehicle dynamic prediction method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program runs on a processor, the pedestrian and vehicle dynamic prediction method according to any one of claims 1 to 6 is implemented.

Images