Vehicle, control method, control device, and controller therefor

Abstract

The application relates to the technical field of vehicle control, and particularly discloses a vehicle, a control method, a control device and a controller thereof. The method comprises the following steps: acquiring environmental information around the vehicle; determining a target pedestrian and a target vehicle according to the environmental information, and determining a predicted trajectory of the target pedestrian according to the environmental information; calling position information of the target vehicle from a roadside device based on identification information of the target vehicle, and determining a predicted trajectory of the target vehicle based on the position information; and controlling the vehicle according to the predicted trajectory of the target pedestrian and the predicted trajectory of the target vehicle. Thus, the method determines the predicted trajectories of the target pedestrian and the target vehicle according to the surrounding environmental information and information interaction between the roadside device, and controls the vehicle based on the predicted trajectories, thereby improving the driving safety of the vehicle.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicle control, and particularly relates to a vehicle control method, a vehicle controller, a vehicle control device and a vehicle. BACKGROUND

[0002] With the increasing of the number of automobiles and the acceleration of urbanization, the problems of road traffic congestion and frequent traffic accidents are increasingly prominent.

[0003] During driving, the driver observes the surrounding environment and evaluates the driving safety, and then controls the vehicle according to the evaluation result. The control method requires high attention and subjective judgment ability of the driver, and reduces the stability of driving safety. SUMMARY

[0004] The present application aims to at least solve one of the problems in the related art. To this end, the first object of the present application is to provide a vehicle control method, which determines the predicted trajectories of target pedestrians and target vehicles based on the surrounding environmental signals and the information interaction with roadside equipment, and controls the vehicle based on the predicted trajectories, thereby improving the driving safety of the vehicle.

[0005] The second object of the present application is to provide a vehicle controller.

[0006] The third object of the present application is to provide a vehicle control device.

[0007] The fourth object of the present application is to provide a vehicle.

[0008] To achieve the above objects, the first aspect of the present application provides a vehicle control method, which comprises: acquiring the environmental information around the vehicle; determining target pedestrians and target vehicles according to the environmental information, and determining the predicted trajectories of the target pedestrians according to the environmental information; calling the position information of the target vehicles from the roadside equipment based on the identification information of the target vehicles, and determining the predicted trajectories of the target vehicles based on the position information; and controlling the vehicle according to the predicted trajectories of the target pedestrians and the predicted trajectories of the target vehicles.

[0009] The control method of the vehicle of the embodiment of the present application first acquires the environmental information around the vehicle, determines the target pedestrian and the target vehicle according to the environmental information, and determines the predicted trajectory of the target pedestrian according to the environmental information, then calls the position information of the target vehicle from the roadside equipment based on the identification information of the target vehicle, and determines the predicted trajectory of the target vehicle based on the position information, so as to control the vehicle according to the predicted trajectory of the target pedestrian and the predicted trajectory of the target vehicle. Therefore, the method determines the predicted trajectory of the target pedestrian and the target vehicle according to the surrounding environmental information and the information interaction between the roadside equipment, and controls the vehicle based on the predicted trajectory, thereby improving the driving safety of the vehicle

[0010] In addition, the control method of the vehicle according to the above-mentioned embodiment of the present application can also have the following additional technical features:

[0011] According to one embodiment of the present application, the environmental information includes a driving video, and the determination of the target pedestrian and the target vehicle according to the environmental information includes: feature extraction of the current frame of the driving video to obtain a plurality of feature maps of different sizes; upsampling and feature fusion of the plurality of feature maps of different sizes to obtain a target feature map; regression analysis of the target feature map to generate prediction box information; and determination of the target pedestrian and the target vehicle according to the class label of the prediction box information.

[0012] According to one embodiment of the present application, the determination of the predicted trajectory of the target pedestrian according to the environmental information includes: target feature modeling of the target pedestrian to determine a running state model of the target pedestrian, and determination of the next frame prediction box of the target pedestrian based on the running state model of the target pedestrian; acquisition of the detection box of the target pedestrian of the next frame of the driving video; and matching tracking of the next frame prediction box of the target pedestrian and the detection box of the target pedestrian of the next frame to output the predicted trajectory of the target pedestrian.

[0013] According to one embodiment of the present application, the position information includes a plurality of position coordinates, and the determination of the predicted trajectory of the target vehicle based on the position information includes: function fitting of the plurality of position coordinates to obtain the predicted trajectory of the target vehicle.

[0014] According to one embodiment of the present application, the control of the vehicle according to the running trajectory of the target pedestrian and the running trajectory of the target vehicle includes: determination of the running trajectory of the vehicle; estimation of the first trajectory intersection probability between the target pedestrian and the vehicle according to the predicted trajectory of the target pedestrian and the running trajectory of the vehicle; estimation of the second trajectory intersection probability between the target vehicle and the vehicle according to the predicted trajectory of the target vehicle and the running trajectory of the vehicle; and control of the vehicle according to the first trajectory intersection probability and the second trajectory intersection probability.

[0015] According to one embodiment of the present application, the vehicle is controlled according to the first trajectory intersection probability and the second trajectory intersection probability, including: when the first trajectory intersection probability and the second trajectory intersection probability are both zero, controlling the vehicle to normally drive; when the first trajectory intersection probability is greater than zero and less than a first preset probability and the second trajectory intersection probability is greater than zero and less than a second preset probability, generating a first warning information, and controlling the vehicle based on the first warning information; when the first trajectory intersection probability is greater than or equal to the first preset probability and / or the second trajectory intersection probability is greater than or equal to the second preset probability, generating a second warning signal, and controlling the vehicle to perform emergency control based on the second warning signal.

[0016] According to one embodiment of the present application, the environment information around the vehicle is acquired, including: collecting the surrounding environment through the vehicle-mounted device of the vehicle; and / or acquiring the identity information and the position information of the vehicle within the first preset range based on the information interaction between the vehicle and the roadside device; and / or acquiring the identity information and the position information of the pedestrian within the second preset range based on the information interaction between the vehicle and the pedestrian device.

[0017] To achieve the above object, the second aspect embodiment of the present application provides a vehicle controller, including a memory, a processor, and a control program of the vehicle stored in the memory and executable on the processor, and when the processor executes the control program of the vehicle, the vehicle control method described above is implemented.

[0018] The vehicle controller according to the embodiment of the present application, when the processor executes the control program of the vehicle, implements the vehicle control method described above, based on the vehicle control method described above, determines the predicted trajectory of the target pedestrian and the target vehicle according to the surrounding environment signal and the information interaction between the vehicle and the roadside device, and controls the vehicle based on the predicted trajectory, thereby improving the driving safety of the vehicle.

[0019] To achieve the above object, the third aspect embodiment of the present application provides a vehicle control device, including: an acquisition module, configured to acquire the environment information around the vehicle; a first determination module, configured to determine the target pedestrian and the target vehicle according to the environment information, and determine the predicted trajectory of the target pedestrian according to the environment information; a second determination module, configured to retrieve the position information of the target vehicle from the roadside device based on the identification information of the target vehicle, and determine the predicted trajectory of the target vehicle based on the position information; and a control module, configured to control the vehicle according to the predicted trajectory of the target pedestrian and the predicted trajectory of the target vehicle.

[0020] According to the control device of the vehicle in the embodiment of the present application, the environment information around the vehicle is acquired by the acquisition module, the target pedestrian and the target vehicle are determined according to the environment information by the first determination module, and the predicted trajectory of the target pedestrian is determined according to the environment information, the position information of the target vehicle is called from the roadside device based on the identification information of the target vehicle by the second determination module, and the predicted trajectory of the target vehicle is determined based on the position information, and the vehicle is controlled according to the predicted trajectory of the target pedestrian and the predicted trajectory of the target vehicle by the control module. Therefore, the device determines the predicted trajectory of the target pedestrian and the target vehicle according to the surrounding environment signal and the information interaction between the roadside device, and controls the vehicle based on the predicted trajectory, thereby improving the driving safety of the vehicle.

[0021] To achieve the above object, the fourth aspect embodiment of the present application provides a vehicle comprising the vehicle controller or the control device of the vehicle.

[0022] According to the vehicle in the embodiment of the present application, based on the vehicle controller or the control device of the vehicle, the driving safety of the vehicle is improved.

[0023] Additional aspects and advantages of the present application will be made apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 Flow chart of the control method of the vehicle according to an embodiment of the present application;

[0025] Figure 2 Algorithm architecture diagram according to a specific embodiment of the present application;

[0026] Figure 3 Flow chart of the control method of the vehicle according to a specific embodiment of the present application;

[0027] Figure 4 Block diagram of the vehicle controller according to an embodiment of the present application;

[0028] Figure 5 Connection diagram of the control device of the vehicle according to an embodiment of the present application;

[0029] Figure 6 Block diagram of the vehicle according to an embodiment of the present application;

[0030] Figure 7 Block diagram of the vehicle according to another embodiment of the present application. DETAILED DESCRIPTION

[0031] Embodiments of the present application are described below in detail, examples of which are shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0032] A control method of a vehicle, a vehicle controller, a control device of a vehicle, and a vehicle according to embodiments of the present application are described below with reference to the drawings.

[0033] Figure 1 A flowchart of a control method of a vehicle according to embodiments of the present application.

[0034] As shown in Figure 1 A control method of a vehicle according to embodiments of the present application, the method comprising:

[0035] S1, acquiring environmental information around the vehicle;

[0036] S2, determining a target pedestrian and a target vehicle according to the environmental information, and determining a predicted trajectory of the target pedestrian according to the environmental information;

[0037] S3, calling position information of the target vehicle from a roadside device based on identification information of the target vehicle, and determining a predicted trajectory of the target vehicle based on the position information;

[0038] S4, controlling the vehicle according to the predicted trajectory of the target pedestrian and the predicted trajectory of the target vehicle.

[0039] Specifically, the environmental information around the vehicle can be collected in real time by various sensors, such as radar, lidar, camera, etc., to provide data support for subsequent identification and tracking. In addition, the environmental information around the vehicle, such as the positions and running directions of vehicles and pedestrians within a certain range, can also be obtained by information interaction with roadside devices based on vehicle networking technology.

[0040] The target pedestrian and the target vehicle are determined by identifying the environmental information. For example, the target pedestrian is determined by identifying pedestrians in environmental images or video sequences through computer vision technology to obtain the area position and size of the pedestrians. The target vehicle is determined by identifying vehicles in environmental images or videos through image processing technology, including extracting information such as vehicle type and license plate. When the environmental information around the vehicle, such as the positions and running directions of vehicles and pedestrians within a certain range, is obtained by information interaction between the vehicle and roadside devices, vehicles with a relatively close distance to the vehicle or a crossing point in the running direction are determined as the target vehicle and the target pedestrian.

[0041] According to the environmental information, the predicted trajectory of the target pedestrian is determined based on a pedestrian tracking technology. For example, the pedestrian tracking technology is to continuously track a specific pedestrian in consecutive video frames to obtain the motion trajectory of the pedestrian. By combining target detection, feature extraction, data association and other steps, continuous tracking of pedestrians is realized. At the same time, deep learning and other technologies can be used to further improve the accuracy and robustness of tracking.

[0042] After determining the target vehicle, the position information of the target vehicle is retrieved from the roadside equipment according to the identification information of the target vehicle, so as to obtain the predicted trajectory of the target vehicle according to the position information of the target vehicle. The position information of the vehicle is determined by a plurality of position coordinates received by a vehicle-mounted GPS (Global Positioning System) chip or a Beidou chip from satellite signals, and the position information is transmitted back to the control center in real time through the roadside equipment. The control center processes and displays the received information (identification information of the vehicle and GPS position), realizes real-time tracking and monitoring of the vehicle. For example, vehicle A synchronously sends the real-time GPS coordinates of vehicle A obtained by the vehicle-mounted GPS chip to the closest roadside equipment for transmission to the control center. The control center monitors the position of vehicle A in real time according to the real-time GPS coordinates of vehicle A, and determines the predicted trajectory of vehicle A according to a plurality of consecutive GPS coordinates. Assuming that vehicle B determines vehicle A as the target vehicle, vehicle B sends the license plate identification of vehicle A obtained to the roadside equipment, so as to retrieve the plurality of consecutive GPS coordinates of vehicle A from the control center through the roadside equipment. Thus, vehicle B can determine the predicted trajectory of vehicle A according to the plurality of consecutive GPS coordinates of vehicle A. In addition, in order to reduce the calculation amount and data transmission load, the position information of the target vehicle can be obtained in intervals according to the position coordinate area and time.

[0043] Then, the vehicle is controlled according to the predicted trajectory of the target pedestrian and the predicted trajectory of the target vehicle. For example, when it is determined that there is a trajectory overlap between the driving trajectory of the vehicle and the predicted trajectory of the target pedestrian and the predicted trajectory of the target vehicle within a preset time, it is considered that there is a collision risk, and the vehicle is reminded or speed control is performed to improve driving safety.

[0044] This embodiment realizes real-time prediction of the trajectories of pedestrians and vehicles around the vehicle during vehicle driving, and controls the vehicle according to the predicted trajectories of pedestrians and vehicles, so as to reduce the collision risk and improve driving safety.

[0045] In an embodiment of the present application, the environment information around the vehicle is acquired, including: collecting the surrounding environment through the vehicle-mounted device of the vehicle; and / or acquiring the identity information and the location information of the vehicle within the first preset range based on the information interaction between the vehicle and the roadside device; and / or acquiring the identity information and the location information of the pedestrian within the second preset range based on the information interaction between the vehicle and the pedestrian device.

[0046] Specifically, the vehicle-mounted device can include a camera, a radar, a lidar and the like sensor for real-time perception of the surrounding environment information of the vehicle.

[0047] The roadside unit is installed on both sides of the road, and uses the Dedicated Short Range Communication (DSRC) or Cellular Vehicle to Everything (C-V2X) and the like to perform information interaction with the vehicle-mounted device, to realize vehicle identity recognition and location tracking. The first preset range can be set based on the actual situation. The roadside unit can on one hand perform information retrieval according to the identity information of the target vehicle sent by the vehicle, and on the other hand can actively send the identity information and the location information of other vehicles to the vehicle when the distance between the other vehicles and the vehicle is within the first preset range, so that the vehicle can perform trajectory prediction on the external vehicle, to prevent the formation of a detection blind area due to external obstructions such as green plants, buildings and equipment, and to improve detection safety.

[0048] The pedestrian device is a smart phone or other portable device carried by a pedestrian, which performs information interaction with the vehicle through Bluetooth, Wi-Fi and the like technology, to acquire the identity information and the location information of the pedestrian, and to improve pedestrian safety. The second preset range can be set according to the information interaction distance of the pedestrian device.

[0049] In an embodiment of the present application, in the case that the environment information includes the identity information and the location information of the vehicle within the first preset range, the target vehicle is determined according to the environment information, including: determining the vehicle and the external vehicle according to the identity information of the vehicle; and determining the external vehicle as the target vehicle when the relative distance between the external vehicle and the vehicle is less than a preset distance.

[0050] That is, when the vehicle receives the identity information and the location information of the vehicle within the first preset range actively sent by the roadside device, the vehicle first divides the vehicle and the external vehicle according to the identity information of the vehicle, and takes the external vehicle with a distance less than a preset distance from the vehicle as the target vehicle of the vehicle, and then performs trajectory prediction on the target vehicle for vehicle control. This embodiment limits the data calculation amount while ensuring control safety.

[0051] In an embodiment of the present application, the environment information comprises a driving video, and determining the target pedestrian and the target vehicle according to the environment information comprises: performing feature extraction on a current frame of the driving video to obtain a plurality of feature maps of different sizes; performing upsampling and feature fusion on the plurality of feature maps of different sizes to obtain a target feature map; performing regression analysis on the target feature map to generate prediction box information; and determining the target pedestrian and the target vehicle according to a class label of the prediction box information.

[0052] Specifically, in combination with Figure 2 As shown in the figure, the video frame of the driving video first enters the backbone network Darknet-53 of YOLOv3 (You Only Look Once version 3, the third version of the YOLO series target detection algorithm) for feature extraction. Darknet-53 is a convolutional network with a depth of 53 layers, which draws on the idea of Residual Network (ResNet) and avoids the gradient disappearance problem in deep networks through residual structures. It uses a 3x3 convolutional layer with a stride of 2 instead of a pooling layer for downsampling, and finally outputs three feature maps of different scales, 13x13, 26x26 and 52x52. These feature maps are used to detect large, medium and small targets, respectively.

[0053] Then, from the plurality of feature maps output by Darknet-53, the 13x13 feature map first undergoes 5 convolutional processing, part of which is used to output prediction results, and the other part undergoes convolution and upsampling operations and is spliced with the 26x26 feature map. The spliced feature map undergoes 5 convolutional processing again, part of which is used to output prediction results, and the other part continues to be upsampled and spliced with the 52x52 feature map. Through this upsampling and feature fusion method, the network can combine feature information of different scales to improve the accuracy of detection and obtain a target feature map.

[0054] The target feature map after feature fusion will enter the detection layer of YOLOv3. Each cell of each feature map will predict multiple bounding boxes (obtained by adjusting anchor boxes), and predict the class probability and confidence of each bounding box. Confidence represents the probability that a target exists in the bounding box. Through regression analysis, the network will calculate the coordinates (center point coordinates, width and height) of the bounding box and the class probability according to the feature information in the feature map. Finally, prediction box information is generated according to these regression analysis results.

[0055] In addition, the generated prediction box information will undergo post-processing operations such as Non-Maximum Suppression (NMS) to remove redundant bounding boxes and retain the optimal detection results.

[0056] The prediction box information includes boundary box coordinates (center point coordinates, width and height), confidence, and a class label (such as a pedestrian or a vehicle), so that a target pedestrian and a target vehicle can be determined according to the class label of the prediction box information.

[0057] In an embodiment of the present application, determining the prediction trajectory of the target pedestrian according to the environment information includes: modeling a target feature of the target pedestrian to determine a running state model of the target pedestrian, and determining a next frame prediction box of the target pedestrian based on the running state model of the target pedestrian; obtaining a detection box of the target pedestrian in a next frame of the driving video; and performing matching tracking on the next frame prediction box of the target pedestrian and the detection box of the target pedestrian in the next frame to output the prediction trajectory of the target pedestrian.

[0058] Specifically, the prediction box information output by the YOLOv3 target detection network is output to a SORT (Simple Online and Realtime Tracking) algorithm, and target tracking is performed through the SORT algorithm.

[0059] In the target feature modeling stage, the SORT algorithm uses a Kalman filter to model the motion state of the target, and the Kalman filter can predict the position and speed of the target in the next frame.

[0060] In the matching tracking stage, the SORT algorithm constructs a cost matrix by calculating the Euclidean distance or IoU (Intersection over Union) between the prediction box and the detection box, and then optimizes the cost matrix using the Hungarian algorithm to match the detection box with the tracked target, and updates the state of the target according to the matching result, including position, speed and other information. Among them, the unmatched detection box is initialized as a new tracking target, and the continuous multiple frames of unmatched target will be marked as lost.

[0061] Then, the pedestrian tracking trajectory is output, and the tracking result can be visualized by drawing the ID (Identity Document) and trajectory line of the target on the image, and the tracking result can be saved as a video file or structured data, which can be used for pedestrian behavior analysis, traffic flow monitoring, etc.

[0062] As a specific embodiment of the present application, during vehicle driving, the driving recorder continuously shoots driving video, the DMC (Drive Motor Control) increases algorithm, and the convolutional neural network is used to detect and track pedestrians in the video. After the video frame is input, it first enters the network of YOLOv3 target detection, extracts features through Darknet-53, then performs upsampling and feature fusion, and then performs regression analysis, inputs the obtained prediction box information into the SORT algorithm for target feature modeling, matching and tracking, and finally outputs the result. Then, the labelme (an image labeling tool developed by the Computer Science and Artificial Intelligence Laboratory of Massachusetts Institute of Technology) is used to label the pedestrian dataset, and then the model is generated through the built YOLO algorithm and trained.

[0063] In the process of pedestrian multi-target tracking, the following stages are included:

[0064] (1) Detection stage: the target detection algorithm analyzes each input frame and identifies objects belonging to a specific category, giving classification and coordinates.

[0065] (2) Feature extraction / motion trajectory prediction stage: one or more feature extraction algorithms are used to extract appearance features and motion or interaction features. In addition, a trajectory predictor can be used to predict the next position of the target.

[0066] (3) Similarity calculation stage: appearance features and motion features can be used to calculate the similarity between two targets.

[0067] (4) Association stage: use the calculated similarity as the basis to associate the detection objects and trajectories belonging to the same target, and assign the detection objects the same ID as the trajectory. For example, use the estimated state system and estimated variance or uncertainty of the Kalman filter type tracking to predict. In addition, when the distance threshold is exceeded, the track will be deleted and a new track will be created; set the maximum number of frames allowed to skip for the tracked object.

[0068] This embodiment combines YOLO pedestrian detection and sort algorithm, and controls the Euclidean distance of trajectory prediction by setting basic threshold parameters, which can be applied to monitor pedestrians and track their trajectories during vehicle driving and parking. The personnel flow data obtained by using it can help public transportation and safety management.

[0069] In an embodiment of the present application, the position information includes a plurality of position coordinates, and determining the predicted trajectory of the target vehicle based on the position information includes: performing function fitting on the plurality of position coordinates to obtain the predicted trajectory of the target vehicle.

[0070] That is, the target vehicle is determined by identifying the vehicle in the image or video through image processing technology, including the extraction of vehicle type, license plate and other information. Then the position information of the target vehicle is retrieved according to the identification information of the target vehicle, and the predicted trajectory of the target vehicle is determined by fitting the GPS coordinates in the multiple position information.

[0071] In an embodiment of the present application, the vehicle is controlled according to the running trajectory of the target pedestrian and the running trajectory of the target vehicle, including: determining the running trajectory of the vehicle; estimating the first trajectory intersection probability between the target pedestrian and the vehicle according to the predicted trajectory of the target pedestrian and the running trajectory of the vehicle; estimating the second trajectory intersection probability between the target vehicle and the vehicle according to the predicted trajectory of the target vehicle and the running trajectory of the vehicle; and controlling the vehicle according to the first trajectory intersection probability and the second trajectory intersection probability.

[0072] Specifically, the running trajectory of the vehicle can be determined based on the navigation route input by the user through the center control screen, or obtained by interacting with the navigation app of the driver's mobile phone, or predicted based on the continuous GPS coordinates of the vehicle. The predicted trajectory of the target pedestrian and the predicted trajectory of the target vehicle are calculated respectively, and the intersection probability is estimated. When the intersection probability is large, it is considered that the collision probability between the target and the vehicle is high; when the intersection probability is small, it is considered that the collision probability between the target and the vehicle is small; when the intersection probability is zero, it is considered that there is no collision risk between the target and the vehicle. In addition, the estimation of the intersection probability can be further combined with the time of the roadside warning light such as the traffic light for calculation.

[0073] This embodiment evaluates the collision risk between the target and the vehicle based on the intersection probability between the predicted trajectories, realizes the dataization of driving safety, and improves the control accuracy.

[0074] In an embodiment of the present application, the vehicle is controlled according to the first trajectory intersection probability and the second trajectory intersection probability, including: when the first trajectory intersection probability and the second trajectory intersection probability are both zero, controlling the vehicle to normally drive; when the first trajectory intersection probability is less than a first preset probability and the second trajectory intersection probability is less than a second preset probability, generating a first warning information, and controlling the vehicle based on the first warning information; when the first trajectory intersection probability is greater than or equal to the first preset probability and / or the second trajectory intersection probability is greater than or equal to the second preset probability, generating a second warning signal, and performing emergency control of the vehicle based on the second warning signal.

[0075] Specifically, when the first trajectory intersection probability and the second trajectory intersection probability are both zero, it is considered that there is no collision risk between the vehicle and the target vehicle and the target pedestrian, and the vehicle is controlled to normally drive.

[0076] When the first trajectory intersection probability is greater than zero and less than a first preset probability, it is considered that the target pedestrian and the vehicle do not have a collision risk, but have a relatively low collision risk, and an alarm reminder can be given so that the driver performs speed reduction control or improves attention based on the alarm reminder signal. When the first trajectory intersection probability is greater than the first preset probability, it is considered that the target pedestrian and the vehicle have a relatively large collision risk, and an alarm reminder can be given while an emergency braking control instruction is given, so that the user's reaction speed is accelerated for emergency avoidance and driving safety is improved.

[0077] When the second trajectory intersection probability is greater than zero and less than a second preset probability, it is considered that the target vehicle and the vehicle do not have a collision risk, but have a relatively low collision risk, and an alarm reminder can be given so that the driver performs speed reduction control or improves attention based on the alarm reminder signal. When the second trajectory intersection probability is greater than the second preset probability, it is considered that the target vehicle and the vehicle have a relatively large collision risk, and an alarm reminder can be given while an emergency braking control instruction is given, so that the user's reaction speed is accelerated for emergency avoidance and driving safety is improved. In addition, since the driving speed of the vehicle is generally greater than the speed of the pedestrian, the second preset probability can be less than the first preset probability to timely respond to the collision risk of the target vehicle.

[0078] The embodiment adopts different control strategies based on different intersection probabilities, thereby improving control accuracy.

[0079] As a specific embodiment of the present application, as shown in Figure 3 The control method of the vehicle can include the following steps:

[0080] S101, environmental information around the vehicle is acquired.

[0081] S102, a target pedestrian and a target vehicle are determined according to the environmental information.

[0082] S103, a predicted trajectory of the target pedestrian is determined according to the environmental information.

[0083] S104, position information of the target vehicle is retrieved from a roadside device based on identification information of the target vehicle. The position information includes a plurality of position coordinates.

[0084] S105, the plurality of position coordinates are functionally fitted to obtain a predicted trajectory of the target vehicle.

[0085] S106, a running trajectory of the vehicle is determined.

[0086] S107, a first trajectory intersection probability between the target pedestrian and the vehicle is estimated according to the tracking trajectory and the predicted trajectory of the target pedestrian and the running trajectory of the vehicle.

[0087] S108, predicting a trajectory according to the tracking trajectory of the target vehicle and the running trajectory of the vehicle to estimate a second trajectory intersection probability between the target vehicle and the vehicle;

[0088] S109, controlling the vehicle according to the first trajectory intersection probability and the second trajectory intersection probability.

[0089] The vehicle control method can be applied in the following aspects:

[0090] 1. Safe driving

[0091] Forward collision warning: By recognizing the dynamics of the front vehicle and pedestrians, the driver is warned in advance to avoid collision accidents.

[0092] Intersection collision warning: When the vehicle is driving towards the intersection, it warns of potential collision threats in the vertical direction.

[0093] Pedestrian crossing warning: At intersections or road sections without signal control, the intelligent networked pedestrian crossing warning system ensures the safety of pedestrians.

[0094] 2. Improve traffic safety

[0095] Through the Internet of Vehicles technology, vehicles can real-time perceive the status of surrounding pedestrians and vehicles, take evasive measures in advance, and reduce the incidence of traffic accidents. For example, in the intelligent pedestrian crossing warning system, the vehicle can receive information about pedestrian crossing and automatically slow down or stop to let the pedestrians pass.

[0096] 3. Improve traffic efficiency

[0097] Internet of Vehicles technology can realize intelligent scheduling and management of vehicles and pedestrians, optimize traffic signal control, route planning, etc., and improve road capacity and traffic efficiency.

[0098] 4. Rich driving experience

[0099] Internet of Vehicles technology can also provide real-time traffic, navigation, entertainment and other information services for drivers, enriching the driving experience and improving the driving convenience.

[0100] 5. Autonomous driving

[0101] Provide important environmental perception capabilities for autonomous vehicles, enabling autonomous navigation and obstacle avoidance.

[0102] 6. Smart City

[0103] As an important part of smart cities, Internet of Vehicles pedestrian and vehicle recognition and tracking technology will provide strong support for urban traffic management, public safety and other fields.

[0104] In addition, the vehicle control method can be implemented based on the following technologies:

[0105] 1. Pedestrian detection technology: refers to the identification of pedestrians in images or video sequences through computer vision technology, determining their area location and size.

[0106] 2. Pedestrian tracking technology: continuously tracks specific pedestrians in consecutive video frames, obtaining their motion trajectories. Specifically, through the combination of target detection, feature extraction, data association, and other steps, continuous tracking of pedestrians is achieved. Meanwhile, advanced technologies such as deep learning can further improve the accuracy and robustness of tracking.

[0107] 3. Vehicle recognition technology: refers to the identification of vehicles in images or videos through image processing technology, including the extraction of vehicle type, license plate, and other information. It can be widely applied in intelligent traffic monitoring, parking lot management, vehicle violation detection, and other fields.

[0108] 4. Vehicle tracking technology: determines vehicle location through GPS or Beidou chips receiving satellite signals, and transmits location information back to the control center in real time, achieving real-time tracking of vehicles.

[0109] 5. Sensor technology: pedestrian and vehicle recognition and tracking in the Internet of Vehicles rely on various sensor technologies, including radar, lidar, cameras, etc. These sensors can collect real-time environmental information around the vehicle, providing data support for subsequent recognition and tracking.

[0110] 6. Computer vision technology: pedestrian detection and tracking technology based on computer vision is one of the key technologies in the Internet of Vehicles. Through image processing, feature extraction, target classification and tracking, accurate identification and tracking of pedestrians and vehicles are achieved. Convolutional Neural Networks (CNN) and other deep learning algorithms perform well in pedestrian detection and tracking, with high accuracy and real-time performance.

[0111] 7. Wireless communication technology: pedestrian and vehicle recognition and tracking in the Internet of Vehicles require wireless communication technology to realize information exchange between vehicles, roadside facilities, and cloud platforms. The rapid development of 5G (5th Generation Mobile Communication Technology), C-V2X (Cellular Vehicle-to-Everything), and other wireless communication technologies provides more efficient and reliable information transmission means for the Internet of Vehicles.

[0112] 8. Artificial intelligence and big data processing technology:

[0113] Artificial intelligence and big data processing technology play an important role in the identification and tracking of pedestrians and vehicles in the Internet of Vehicles. Through the analysis and processing of massive data, useful information can be extracted to provide decision support for traffic management, accident prevention, etc.

[0114] 9. Cloud computing and cloud platform technology: used to process and analyze the massive data generated by the Internet of Vehicles system, extract valuable information and optimize system performance.

[0115] By comprehensively using sensor technology, computer vision technology, wireless communication technology, and artificial intelligence and big data processing technology, accurate identification and tracking of pedestrians and vehicles can be achieved, providing strong support for improving road traffic safety, optimizing traffic flow management, promoting automatic driving and the development of smart cities.

[0116] Further, the control method of the vehicle can be applied to a vehicle-to-everything pedestrian and vehicle identification tracking system architecture, which generally includes the following key components: a perception layer, a network layer, a platform layer, and an application layer. These four layers cooperate with each other to achieve comprehensive monitoring and management of vehicles and pedestrians. Among them, the perception layer is the front end of the vehicle-to-everything pedestrian and vehicle identification tracking system, responsible for collecting relevant information of vehicles and pedestrians, which can include vehicle-mounted devices, roadside units, and pedestrian devices. The network layer is responsible for transmitting the data collected by the perception layer to the platform layer for processing and analysis, which can include mobile communication networks (the 4th generation mobile communication technology, fourth generation mobile communication technology), 5G, etc.) and dedicated communication networks (such as DSRC (Dedicated Short Range Communication), C-V2X, etc.), based on mobile communication networks can provide high-speed, low-latency data transmission services, and based on dedicated communication networks can realize direct communication between vehicles and infrastructure. The platform layer is the core of the vehicle-to-everything pedestrian and vehicle identification tracking system, responsible for processing, analyzing, and storing the received data, which can include data processing centers, data storage centers, and analysis systems, among which the data processing center uses cloud computing, big data, and other technologies to process and analyze massive amounts of data in real time, the data storage center is used to store historical data of vehicles and pedestrians, providing a basis for subsequent data mining and decision support, and the analysis system performs in-depth analysis of data through algorithms and models to extract valuable information and rules. The application layer is the final manifestation of the vehicle-to-everything pedestrian and vehicle identification tracking system, providing various services and applications for users, which can include safety warning systems (such as forward collision warning, intersection collision warning, etc. to improve driving safety), traffic management systems (providing real-time traffic data for traffic management departments to help urban traffic congestion governance and intelligent transportation system construction), autonomous driving systems (combining high-precision maps and positioning technologies to achieve autonomous driving and intelligent obstacle avoidance of vehicles), and information service systems (providing personalized services such as navigation, entertainment, and vehicle status monitoring for users).

[0117] In summary, the control method of the vehicle according to the embodiments of the present application first acquires the environmental information around the vehicle, determines the target pedestrian and the target vehicle according to the environmental information, and determines the predicted trajectory of the target pedestrian according to the environmental information, then retrieves the position information of the target vehicle from the roadside device based on the identification information of the target vehicle, and determines the predicted trajectory of the target vehicle based on the position information, thereby controlling the vehicle according to the predicted trajectory of the target pedestrian and the predicted trajectory of the target vehicle. Thus, the method determines the predicted trajectory of the target pedestrian and the target vehicle according to the surrounding environmental signals and the information interaction between the roadside device, and controls the vehicle based on the predicted trajectory, thereby improving the driving safety of the vehicle.

[0118] Corresponding to the above embodiment, the application further proposes a vehicle controller.

[0119] As shown in Figure 4 the vehicle controller 100 of the embodiment of the application comprises a memory 110, a processor 120 and a control program of the vehicle stored on the memory 110 and executable on the processor 120, and when the processor 120 executes the control program of the vehicle, the above-mentioned control method of the vehicle is realized.

[0120] According to the vehicle controller of the embodiment of the application, when the processor executes the control program of the vehicle, the above-mentioned control method of the vehicle is realized, and based on the above-mentioned control method of the vehicle, the predicted trajectories of the target pedestrian and the target vehicle are determined according to the surrounding environmental signals and the information interaction with the roadside equipment, and the vehicle is controlled based on the predicted trajectories, thereby improving the driving safety of the vehicle.

[0121] Corresponding to the above embodiment, the application further proposes a control device of a vehicle.

[0122] As shown in Figure 5 the control device of the vehicle of the embodiment of the application comprises an acquisition module 10, a first determination module 20, a second determination module 30 and a control module 40.

[0123] The acquisition module 10 is configured to acquire the environmental information around the vehicle; the first determination module 20 is configured to determine the target pedestrian and the target vehicle according to the environmental information, and determine the predicted trajectory of the target pedestrian according to the environmental information; the second determination module 30 is configured to call the position information of the target vehicle from the roadside equipment based on the identification information of the target vehicle, and determine the predicted trajectory of the target vehicle based on the position information; and the control module 40 is configured to control the vehicle according to the predicted trajectory of the target pedestrian and the predicted trajectory of the target vehicle.

[0124] According to one embodiment of the application, the environmental information comprises a driving video, the first determination module 20 determines the target pedestrian and the target vehicle according to the environmental information, and is specifically configured to: perform feature extraction on a current frame of the driving video to obtain a plurality of feature maps of different sizes; perform up-sampling and feature fusion on the plurality of feature maps of different sizes to obtain a target feature map; perform regression analysis on the target feature map to generate prediction box information; and determine the target pedestrian and the target vehicle according to the class label of the prediction box information.

[0125] According to one of the embodiments of the present application, the first determining module 20 determines the predicted trajectory of the target pedestrian according to the environmental information, specifically for: modeling the target characteristics of the target pedestrian to determine the running state model of the target pedestrian, and determining the next frame predicted box of the target pedestrian based on the running state model of the target pedestrian; obtaining the detection box of the target pedestrian in the next frame of the driving video; and performing matching tracking on the next frame predicted box of the target pedestrian and the detection box of the target pedestrian in the next frame to output the predicted trajectory of the target pedestrian.

[0126] According to one of the embodiments of the present application, the position information includes a plurality of position coordinates, and the second determining module 30 determines the predicted trajectory of the target vehicle based on the plurality of position coordinates, including: performing function fitting on the position information to obtain the predicted trajectory of the target vehicle.

[0127] According to one of the embodiments of the present application, the control module 40 controls the vehicle according to the running trajectory of the target pedestrian and the running trajectory of the target vehicle, specifically for: determining the running trajectory of the vehicle; estimating the first trajectory intersection probability between the target pedestrian and the vehicle according to the predicted trajectory of the target pedestrian and the running trajectory of the vehicle; estimating the second trajectory intersection probability between the target vehicle and the vehicle according to the predicted trajectory of the target vehicle and the running trajectory of the vehicle; and controlling the vehicle according to the first trajectory intersection probability and the second trajectory intersection probability.

[0128] According to one of the embodiments of the present application, the control module 40 controls the vehicle according to the first trajectory intersection probability and the second trajectory intersection probability, specifically for: controlling the vehicle to normally drive when the first trajectory intersection probability and the second trajectory intersection probability are both zero; generating the first warning information when the first trajectory intersection probability is greater than zero and less than a first preset probability and the second trajectory intersection probability is greater than zero and less than a second preset probability, and controlling the vehicle based on the first warning information; and generating the second warning signal when the first trajectory intersection probability is greater than or equal to the first preset probability and / or the second trajectory intersection probability is greater than or equal to the second preset probability, and performing emergency control on the vehicle based on the second warning signal.

[0129] According to one of the embodiments of the present application, the acquisition module 10 acquires the environmental information around the vehicle, specifically for: collecting the surrounding environment through the vehicle-mounted device of the vehicle; and / or acquiring the identity information and the position information of the vehicle within the first preset range based on the information interaction between the vehicle and the roadside device; and / or acquiring the identity information and the position information of the pedestrian within the second preset range based on the information interaction between the vehicle and the pedestrian device.

[0130] It should be noted that the details of the control device of the vehicle in the embodiments of the present application are not disclosed, please refer to the details disclosed in the control method of the vehicle in the above embodiments of the present application, which will not be described here.

[0131] According to the control device of the vehicle provided in the embodiments of the present application, the environment information around the vehicle is acquired by the acquisition module, the target pedestrian and the target vehicle are determined according to the environment information by the first determination module, the predicted trajectory of the target pedestrian is determined according to the environment information, the position information of the target vehicle is called from the roadside device based on the identification information of the target vehicle by the second determination module, the predicted trajectory of the target vehicle is determined based on the position information, and the vehicle is controlled according to the predicted trajectory of the target pedestrian and the predicted trajectory of the target vehicle by the control module. Therefore, the predicted trajectories of the target pedestrian and the target vehicle are determined based on the environment information around the vehicle and the information interaction between the roadside device, and the vehicle is controlled based on the predicted trajectories, thereby improving the driving safety of the vehicle.

[0132] According to the control device of the vehicle provided in the embodiments of the present application, the environment information around the vehicle is acquired by the acquisition module, the target pedestrian and the target vehicle are determined according to the environment information by the first determination module, the predicted trajectory of the target pedestrian is determined according to the environment information, the position information of the target vehicle is called from the roadside device based on the identification information of the target vehicle by the second determination module, the predicted trajectory of the target vehicle is determined based on the position information, and the vehicle is controlled according to the predicted trajectory of the target pedestrian and the predicted trajectory of the target vehicle by the control module. Therefore, the predicted trajectories of the target pedestrian and the target vehicle are determined based on the environment information around the vehicle and the information interaction between the roadside device, and the vehicle is controlled based on the predicted trajectories, thereby improving the driving safety of the vehicle.

[0133] As shown in Figure 6 , the vehicle 200 provided in the embodiments of the present application comprises the vehicle controller 100 described above, or as shown in Figure 7 , the vehicle 200 provided in the embodiments of the present application comprises the control device 210 of the vehicle described above.

[0134] According to the control device of the vehicle provided in the embodiments of the present application, the environment information around the vehicle is acquired by the acquisition module, the target pedestrian and the target vehicle are determined according to the environment information by the first determination module, the predicted trajectory of the target pedestrian is determined according to the environment information, the position information of the target vehicle is called from the roadside device based on the identification information of the target vehicle by the second determination module, the predicted trajectory of the target vehicle is determined based on the position information, and the vehicle is controlled according to the predicted trajectory of the target pedestrian and the predicted trajectory of the target vehicle by the control module. Therefore, the predicted trajectories of the target pedestrian and the target vehicle are determined based on the environment information around the vehicle and the information interaction between the roadside device, and the vehicle is controlled based on the predicted trajectories, thereby improving the driving safety of the vehicle.

[0135] It is to be appreciated that the above description and the examples that follow are intended to be illustrative only and that changes can be made to the description, either functionally or chronologically, as well as changes being made concerning the order of implementation. The logic and / or steps represented in the flow diagrams and / or described herein can be viewed as ordered listing of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In the context of this specification, a "computer-readable medium" can be any means that can contain, store, communicate, propagate or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a machine-readable storage device (e.g., magnetic, optical or other) a machine- readable storage diskette (e.g., floppy disk, optical disk, CD- ROM or other), a machine- readable wired or wireless communication medium that can contain, store, communicate, propagate or transport the program, all or portions of which might be included in the memory, or any suitable combination of the above. Specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber (optical), and a portable compact disc read-only memory (CDROM). Note that the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for instance via the optical scan of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in the memory.

[0136] It is to be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following technologies known in the art can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.

[0137] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0138] In addition, the terms "first", "second", etc. are used only for the purpose of description, and should not be understood as indicating or implying relative importance or implying a number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly and specifically limited.

[0139] In the present application, unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting", "fixing" and the like should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or it can be integrated; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the internal communication of two elements or the interaction relationship between two elements, unless otherwise explicitly limited. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0140] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A method for controlling a vehicle, characterized in that, The method includes: Obtain information about the environment surrounding the vehicle; The target pedestrians and vehicles are determined based on the environmental information, and the predicted trajectory of the target pedestrians is determined based on the environmental information. The location information of the target vehicle is retrieved from the roadside equipment based on the identification information of the target vehicle, and the predicted trajectory of the target vehicle is determined based on the location information. The vehicle is controlled based on the predicted trajectory of the target pedestrian and the predicted trajectory of the target vehicle; The environmental information includes driving video, and the step of determining the target pedestrian and target vehicle based on the environmental information includes: Feature extraction is performed on the current frame of the driving video to obtain multiple feature maps of different sizes; The target feature map is obtained by upsampling and feature fusion of the multiple feature maps of different sizes; Regression analysis is performed on the target feature map to generate prediction box information; The target pedestrians and target vehicles are determined based on the category labels of the prediction box information; Determining the predicted trajectory of the target pedestrian based on the environmental information includes: The target pedestrian is modeled with target features to determine the target pedestrian's running state model, and the next frame prediction box of the target pedestrian is determined based on the target pedestrian's running state model; Obtain the target pedestrian detection bounding box in the next frame of the driving video; The predicted bounding box of the target pedestrian in the next frame and the detected bounding box of the target pedestrian in the next frame are matched and tracked to output the predicted trajectory of the target pedestrian.

2. The vehicle control method according to claim 1, characterized in that, The location information includes multiple location coordinates, and determining the predicted trajectory of the target vehicle based on the location information includes: The multiple location coordinates are fitted with a function to obtain the predicted trajectory of the target vehicle.

3. The vehicle control method according to claim 1, characterized in that, The step of controlling the vehicle based on the trajectory of the target pedestrian and the trajectory of the target vehicle includes: Determine the vehicle's trajectory; Based on the predicted trajectory of the target pedestrian and the trajectory of the vehicle, the probability of a first trajectory intersection between the target pedestrian and the vehicle is estimated. The probability of a second trajectory intersection between the target vehicle and the vehicle itself is estimated based on the predicted trajectory of the target vehicle and the trajectory of the vehicle itself. The vehicle is controlled based on the first trajectory intersection probability and the second trajectory intersection probability.

4. The vehicle control method according to claim 3, characterized in that, The step of controlling the vehicle based on the first trajectory intersection probability and the second trajectory intersection probability includes: When both the first trajectory intersection probability and the second trajectory intersection probability are zero, the vehicle is controlled to drive normally. When the first trajectory intersection probability is greater than zero and less than the first preset probability, and the second trajectory intersection probability is greater than zero and less than the second preset probability, a first alarm message is generated, and the vehicle is controlled based on the first alarm message; When the first trajectory intersection probability is greater than or equal to the first preset probability and / or the second trajectory intersection probability is greater than or equal to the second preset probability, a second alarm signal is generated, and emergency avoidance control of the vehicle is performed based on the second alarm signal.

5. The vehicle control method according to claim 1, characterized in that, The acquisition of environmental information surrounding the vehicle includes: The vehicle's onboard equipment collects data on the surrounding environment; and / or Based on the information interaction between this vehicle and the roadside equipment, the identity and location information of vehicles within a first preset range are obtained; and / or Based on the information interaction between the vehicle and the pedestrian equipment, the vehicle obtains the identity and location information of pedestrians within the second preset range.

6. A vehicle controller, characterized in that, The system includes a memory, a processor, and a vehicle control program stored in the memory and executable on the processor. When the processor executes the vehicle control program, it implements the vehicle control method according to any one of claims 1-5.

7. A vehicle control device, characterized in that, include: The acquisition module is used to acquire environmental information surrounding the vehicle. The first determining module is used to determine the target pedestrian and the target vehicle based on the environmental information, and to determine the predicted trajectory of the target pedestrian based on the environmental information. The second determining module is used to retrieve the location information of the target vehicle from the roadside equipment based on the identification information of the target vehicle, and determine the predicted trajectory of the target vehicle based on the location information; The control module is used to control the vehicle based on the predicted trajectory of the target pedestrian and the predicted trajectory of the target vehicle. The environmental information includes driving video. The first determining module determines the target pedestrians and target vehicles based on the environmental information, specifically for: Feature extraction is performed on the current frame of the driving video to obtain multiple feature maps of different sizes; The target feature map is obtained by upsampling and feature fusion of the multiple feature maps of different sizes; Regression analysis is performed on the target feature map to generate prediction box information; The target pedestrians and target vehicles are determined based on the category labels of the prediction box information; The first determining module determines the predicted trajectory of the target pedestrian based on the environmental information, specifically for: The target pedestrian is modeled with target features to determine the target pedestrian's running state model, and the next frame prediction box of the target pedestrian is determined based on the target pedestrian's running state model; Obtain the target pedestrian detection bounding box in the next frame of the driving video; The predicted bounding box of the target pedestrian in the next frame and the detected bounding box of the target pedestrian in the next frame are matched and tracked to output the predicted trajectory of the target pedestrian.

8. A vehicle, characterized in that, This includes the vehicle controller according to claim 6, or the vehicle control device according to claim 7.

Images