Early warning method, terminal equipment and computer storage medium

Abstract

The invention belongs to the technical field of vehicles, and provides an early warning method, terminal equipment and a computer storage medium, and the early warning method is applied to intersection early warning equipment and comprises the steps: obtaining image data; obtaining target radar detection data of a detected object, wherein the target radar detection data is aligned with the timestamp of the image data; based on the image coordinate system and the radar coordinate system, carrying out data synchronization on the image data and the target radar detection data so as to determine the speed information and the position information of the measured object; obtaining the time of collision between the target vehicle and other to-be-detected targets based on the speed information and the position information of detected objects, the detected objects including the target vehicle and other to-be-detected targets; and sending the impending collision time to a vehicle-mounted terminal corresponding to the identifier of the target vehicle, so that the vehicle-mounted terminal sends early warning information based on the impending collision time. According to the invention, drivers of vehicles passing through the intersection can be effectively prompted, and traffic accidents can be avoided as much as possible.

Description

Technical Field

[0001] The present application belongs to the field of vehicle technology, and in particular, relates to an early warning method, a terminal device and a computer storage medium. Background Art

[0002] In recent years, the increase in the number of vehicles has brought great convenience to people's travel. However, while providing traffic convenience, vehicles also pose traffic safety risks, especially at intersections. Whether going straight or turning, if there are obstacles such as trees and houses near the intersection, the driver will enter the blind spot of vision, and the flow of vehicles and people at the intersection is complicated. It is possible that cars run red lights, "pedestrians peeking out" and other accidents may occur. Therefore, how to reduce the potential dangers of vehicles passing through road intersections is an issue that needs to be solved urgently. Summary of the invention

[0003] The embodiments of the present application provide an early warning method, a terminal device and a computer storage medium, which can solve the technical problem of potential dangers to vehicles at intersections.

[0004] In a first aspect, an embodiment of the present application provides an early warning method, which is applied to an intersection early warning device, and the method includes:

[0005] Acquiring image data, wherein the image data includes an image coordinate system and an identification of a target vehicle;

[0006] Acquire target radar detection data of the object under test, wherein the target radar detection data includes a radar coordinate system; wherein the timestamps between the target radar detection data and the image data are aligned;

[0007] Based on the image coordinate system and the radar coordinate system, the image data is synchronized with the target radar detection data to determine the speed information and position information of the measured object;

[0008] Acquire the impending collision time between the target vehicle and other targets to be measured based on the speed information and position information of the detected object, wherein the detected object includes the target vehicle and other targets to be measured;

[0009] The imminent collision time is sent to a vehicle-mounted terminal corresponding to the identification of the target vehicle, so that the vehicle-mounted terminal issues a warning message based on the imminent collision time.

[0010] In a possible implementation manner of the first aspect, synchronizing the image data with the target radar detection data based on the image coordinate system and the radar coordinate system to determine the speed information and the position information of the measured object includes:

[0011] Acquire a coordinate mapping relationship between the image coordinate system and the radar coordinate system;

[0012] The image data and the target radar detection data are synchronized based on the coordinate mapping relationship to determine the speed information and position information of the detected object.

[0013] In a possible implementation manner of the first aspect, acquiring a coordinate mapping relationship between the image coordinate system and the radar coordinate system includes:

[0014] Acquire a first coordinate mapping relationship between the image coordinate system and a preset coordinate system;

[0015] Acquire a second coordinate mapping relationship between the radar coordinate system and a preset coordinate system;

[0016] The step of synchronizing the image data with the target radar detection data based on the coordinate mapping relationship to determine the speed information and position information of the object under test includes:

[0017] Mapping the image data to the preset coordinate system based on the first coordinate mapping relationship to obtain first mapping data;

[0018] Mapping the target radar detection data to the preset coordinate system based on the second coordinate mapping relationship to obtain second mapping data;

[0019] The first mapping data is compared with the second mapping data. If the comparison result meets a preset difference threshold, it is determined that the image data and the target radar detection data match, and the position information and speed information of the object under test are determined based on the first mapping data and the second mapping data.

[0020] In a possible implementation manner of the first aspect, obtaining the impending collision time between the target vehicle and other targets to be measured based on the speed information and the position information of the measured object includes:

[0021] Determine first position information of the target vehicle in the preset coordinate system based on the image data and the target radar detection data;

[0022] Matching the first position information with the position information of each detected object according to a first preset matching rule to obtain a detected object vehicle that successfully matches the first position information, and using the speed information of the detected object vehicle as the speed information of the target vehicle;

[0023] Determine second position information and second speed information of other targets to be measured except the vehicle to be measured;

[0024] Obtain the first driving trajectory of the target vehicle in the preset coordinate system based on the first position information;

[0025] Obtain the second driving trajectory of the other target to be measured in the preset coordinate system based on the second position information;

[0026] Determine the relative distance between the other target to be measured and the target vehicle based on the first driving trajectory and the second driving trajectory;

[0027] Calculate the time to collision between the target vehicle and the other target to be measured based on the relative distance, the vehicle speed information, and the second speed information.

[0028] In a possible implementation manner of the first aspect, the step of obtaining the target radar detection data of the object to be measured includes:

[0029] Obtain the laser detection data of the object to be measured, where the lidar detection data includes the lidar coordinate system;

[0030] Obtain the millimeter-wave detection data of the object to be measured; the millimeter-wave detection data includes the millimeter-wave radar coordinate system; wherein, the time stamps of the laser detection data and the millimeter-wave detection data are the same;

[0031] The step of obtaining the second coordinate mapping relationship between the radar coordinate system and the preset coordinate system includes:

[0032] Obtain the third mapping relationship between the lidar coordinate system and the preset coordinate system;

[0033] Obtain the fourth mapping relationship between the millimeter-wave radar coordinate system and the preset coordinate system;

[0034] The step of mapping the target radar detection data to the preset coordinate system based on the second coordinate mapping relationship to obtain the second mapping data specifically includes:

[0035] Map the laser detection data to the preset coordinate system based on the third mapping relationship to obtain the third mapping data;

[0036] Map the millimeter-wave detection data to the preset coordinate system based on the fourth mapping relationship to obtain the fourth mapping data;

[0037] Before the step of comparing the first mapping data and the second mapping data, if the comparison result meets the preset difference threshold, it is determined that the image data and the target radar detection data match, and the position information and speed information of the object to be measured are determined based on the first mapping data and the second mapping data, further including:

[0038] Compare the third mapping data with the fourth mapping data. If the comparison result meets a preset difference threshold, it is determined that the laser detection data and the millimeter-wave detection data match.

[0039] In a possible implementation manner of the first aspect, the method further includes:

[0040] Determine the lateral velocity and lateral acceleration of the object to be measured based on the third mapping data;

[0041] Determine the longitudinal velocity and longitudinal acceleration of the object to be measured based on the fourth mapping data;

[0042] Determine the velocity information of the object to be measured based on the lateral velocity and the longitudinal velocity. The velocity information of the object to be measured includes the second velocity information of the other objects to be measured and the vehicle speed information of the target vehicle;

[0043] Determine the acceleration information of the object to be measured based on the lateral acceleration and the longitudinal acceleration. The acceleration information of the object to be measured includes the acceleration information of the target vehicle and the acceleration information of the other objects to be measured.

[0044] The step of calculating the time to collision between the target vehicle and the other objects to be measured based on the relative distance, the vehicle speed information, and the second velocity information further includes:

[0045] Calculate the time to collision between the target vehicle and the other objects to be measured based on a preset safe stopping distance, the relative distance, the vehicle speed information, the second velocity information, the acceleration information of the target vehicle, and the acceleration information of the other objects to be measured;

[0046] Wherein, the time to collision between the target vehicle and the other objects to be measured is calculated by the following formula:

[0047]

[0048] Wherein, t is the time to collision, S is the relative distance between the other object to be measured and the target vehicle, d is the preset safe stopping distance, V1 is the vehicle speed information of the target vehicle, A1 is the acceleration information of the target vehicle, V2 is the second velocity information of the other object to be measured, and A2 is the acceleration information of the other object to be measured.

[0049] In a possible implementation manner of the first aspect, the step of sending the time to collision to the in-vehicle terminal corresponding to the identifier of the target vehicle includes:

[0050] The impending collision time and the identifier of the target vehicle are sent to a server, so that the server determines a corresponding vehicle-mounted terminal according to the identifier of the target vehicle, and sends the impending collision time to the vehicle-mounted terminal.

[0051] In a second aspect, an embodiment of the present application provides an early warning method, which is applied to an on-board terminal of a target vehicle, and includes:

[0052] Receive the impending collision time generated by the intersection warning device;

[0053] Obtaining current location information of the target vehicle;

[0054] generating warning information based on the current position information and the impending collision time;

[0055] Wherein, the intersection warning device is used for:

[0056] Acquire image data collected by an image sensor of the intersection warning device, wherein the image data includes an image coordinate system and an identification of a target vehicle;

[0057] Acquire target radar detection data of the object under test detected by the radar sensor, wherein the target radar detection data includes a radar coordinate system; wherein the target radar detection data is aligned with a timestamp of the image data;

[0058] Acquire a coordinate mapping relationship between the image coordinate system and the radar coordinate system;

[0059] Based on the coordinate mapping relationship, the image data is synchronized with the target radar detection data to determine the speed information and position information of the measured object;

[0060] The impending collision time between the target vehicle and other targets to be measured is obtained based on the speed information and position information of the detected object, wherein the detected object includes the target vehicle and other targets to be measured.

[0061] In a third aspect, an embodiment of the present application provides a terminal device, wherein when the terminal device is a road intersection warning device, the road intersection warning device is provided with an image sensor, a radar sensor, a controller, a memory, and a computer program stored in the memory and executable on the controller; the image sensor is used to collect image data; the radar sensor is used to detect target radar detection data of a detected object; and the controller implements the steps of the method described above when executing the computer program;

[0062] When the terminal device is a vehicle-mounted terminal, it includes a memory, a controller, and a computer program stored in the memory and executable on the controller. When the controller executes the computer program, the method described above is implemented.

[0063] In a fourth aspect, an embodiment of the present application provides a computer storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a controller, the method described above is implemented.

[0064] The present application provides an early warning method, a terminal device and a computer storage medium. The early warning method is applied to an intersection early warning device. First, image data is obtained, and target radar detection data of the object to be measured is obtained. The target radar detection data is aligned with the timestamp of the image data; based on the image coordinate system and the radar coordinate system, the image data and the target radar detection data are synchronized to determine the speed information and position information of the object to be measured; based on the speed information and position information of the object to be measured, the impending collision time between the target vehicle and other targets to be measured is obtained, wherein the object to be measured includes the target vehicle and other targets to be measured; the impending collision time is sent to the vehicle-mounted terminal corresponding to the identification of the target vehicle, so that the vehicle-mounted terminal issues early warning information based on the impending collision time. In this way, it can effectively prompt the drivers of vehicles passing through the intersection and try to avoid traffic accidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0066] Figure 1 It is a structural diagram of an early warning system provided by an embodiment of the present application;

[0067] Figure 2 It is a structural schematic diagram of a traffic intersection warning device provided in one embodiment of the present application;

[0068] Figure 3 It is a schematic diagram of an application scenario of a warning device in a road intersection provided by an embodiment of the present application;

[0069] Figure 4 It is a flowchart of the first embodiment of the early warning method of the present application;

[0070] Figure 5It is a schematic flowchart for synchronizing the data of an image sensor and the data of a radar sensor in the first embodiment of the warning method of this application;

[0071] Figure 6 It is a schematic flowchart for synchronizing the data detected by a lidar sensor and the data detected by a millimeter-wave sensor in the second embodiment of the warning method of this application;

[0072] Figure 7 It is a schematic flowchart of the third embodiment of the warning method of this application. Detailed implementation manners

[0073] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system architectures, technologies, etc. are presented to thoroughly understand the embodiments of this application. However, those skilled in the art should clearly understand that this application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of this application.

[0074] It should be understood that when used in the specification of this application and the appended claims, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0075] It should also be understood that the term "and / or" used in the specification of this application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0076] As used in the specification of this application and the appended claims, the term "if" can be interpreted as "when", "once", "in response to determining", or "in response to detecting" according to the context. Similarly, the phrase "if determined" or "if [the described condition or event] is detected" can be interpreted as meaning "once determined", "in response to determining", "once [the described condition or event] is detected", or "in response to detecting [the described condition or event]" according to the context.

[0077] In addition, in the description of the specification of this application and the appended claims, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0078] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that specific features, structures, or characteristics described in connection with that embodiment are included in one or more embodiments of this application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way. The terms "comprising", "including", "having" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in another way.

[0079] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system architectures, technologies, etc. are set forth in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art should understand that this application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to impede the description of this application with unnecessary details.

[0080] Please refer to Figure 1 , Figure 1 which is a schematic diagram of an early warning system provided by an embodiment of this application. The system includes: an intersection early warning device 10, a vehicle-mounted terminal 20, and a cloud server 30; the early warning device 10 can establish communication with the vehicle-mounted terminal 20 through the cloud server 30, and the cloud server 30 is responsible for forwarding the information of the early warning device 10 to the vehicle-mounted terminal 20; of course, the early warning device 10 can also directly establish communication with the vehicle-mounted terminal 20, and this embodiment does not limit this.

[0081] As Figure 2 shown, the early warning device 10 is provided with an image sensor 101, a radar sensor 102, a controller 103, a memory 104, and a computer program stored in the memory 104 and executable on the controller 103; the image sensor 101 is used to collect image data; the radar sensor 102 is used to detect the target radar detection data of the object to be detected; when the controller 103 executes the computer program, the steps of the early warning method of the embodiments of the present invention are implemented.

[0082] The early warning device 10 can be a type of intersection computer device, and may include, but is not limited to, a controller 103 and a memory 104. Those skilled in the art can understand that Figure 2 this is only an example of the early warning device 10 and does not constitute a limitation on the early warning device 10. It may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0083] The so-called controller 103 can be a Central Processing Unit (CPU). The controller 103 can also be other general controllers, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general controller can be a microcontroller or the controller can also be any conventional controller, etc.

[0084] In some embodiments, the memory 104 can be an internal storage unit of the warning device 10, such as the hard disk or memory of the warning device 10. In some other embodiments, the memory 104 can also be an external storage device of the warning device 10, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. equipped on the warning device 10. Further, the memory 104 can also include both the internal storage unit and the external storage device of the warning device 10. The memory 104 is used to store an operating system, application programs, a BootLoader, data, and other programs, such as the program code of the computer program, etc. The memory 104 can also be used to temporarily store data that has been output or will be output.

[0085] In a specific implementation, the warning device 10 can be arranged in the middle of an intersection to monitor vehicles and pedestrians at the intersection. Please refer to Figure 3 , the radar sensor 102 of this embodiment includes a lidar sensor 102-1 and a millimeter-wave radar sensor 102-2;

[0086] Among them, the three sensors, namely the image sensor 101, the lidar sensor 102-1, and the millimeter-wave radar sensor 102-2, are all built with chips: the chip of the image sensor 101 will analyze the collected image data, and the controller 103 will process the image analysis data; the chip of the lidar sensor 102-1 will sense the laser detection data, and the controller 103 will process the laser detection data; the chip of the millimeter-wave radar sensor 102-2 will sense the millimeter-wave detection data, and the controller 103 will process the millimeter-wave detection data.

[0087] In this embodiment, the lidar sensor is a 128-line lidar with a 360-degree horizontal field of view and a detection range of 200 m; the millimeter-wave radar sensor has a horizontal field of view angle of 120° and a detection range of 170 m; the camera corresponding to the image sensor has 8 million pixels, a horizontal field of view angle of 120°, and a detection range of 170 m. To cover the intersection scenario without dead angles in 360°, the system installs a total of 3 millimeter-wave radar sensors, 3 cameras, and 1 lidar sensor. It is necessary to ensure that the horizontal fields of view of the millimeter-wave radar and the cameras coincide, and the installation height needs to ensure that there are no blind spots within the sensor's field of view. The installation height of the lidar is 1400 - 1600 mm, the installation height of the camera is 1200 - 1400 mm, and the installation height of the millimeter-wave radar is 300 - 800 mm. The angle installation error is within ±3°. Furthermore, the warning device can achieve a non-blind field of view, 360° panoramic detection, with a maximum detection distance of up to 200 m and a detection accuracy of up to 1 cm.

[0088] Reference Figure 4 , Figure 4 FIG. shows a schematic flowchart of a first embodiment of the warning method provided by the present application. By way of example and not limitation, this method can be applied to the intersection warning device 10. The method includes:

[0089] Step S10, obtaining image data, where the image data includes an image coordinate system and the identification of the target vehicle.

[0090] It can be understood that the image sensor of the warning device will collect the image information of the intersection in real time. The image information includes each vehicle and pedestrian captured within the visible range of the camera, and thus the type of the measured target can be determined. The image sensor will convert the image captured at the intersection into the image coordinate system, where the image coordinate system can use the actual position of the warning device itself at the road intersection as the coordinate origin.

[0091] In addition, the image information also includes the timestamp corresponding to the captured frame of the image.

[0092] In a specific implementation, the image sensor 101 is built-in with a chip, and the warning device is loaded with an image recognition algorithm program (the image recognition algorithm program can be stored in the memory 104). The chip of the image sensor 101 will call the image recognition algorithm to analyze the collected image data, and the controller 103 will process the image analysis data. For example, the image analysis data includes information such as the license plate information and the vehicle shape characteristics of the vehicle. The license plate information and the vehicle shape characteristics are used as the identification of the target vehicle. If the current frame of the image includes a pedestrian or a bicycle, the appearance characteristics of the pedestrian or the rider will be recorded. The data such as the identification of the target vehicle, the appearance characteristics of the pedestrian or the rider will be handed over to the controller 103 for processing.

[0093] In addition, to ensure the accuracy of image acquisition, the camera corresponding to the image sensor in this embodiment preferably has 8 million pixels, a horizontal field of view angle of 120°, and a detection range of 170 m.

[0094] Step S20: Obtain the target radar detection data of the object to be measured. The target radar detection data includes a radar coordinate system. Among them, the time stamps of the target radar detection data and the image data are aligned. It can be understood that the controller of the intersection warning device obtains the radar detection data detected by the radar sensor. The radar detection data includes the distance and speed of the object to be measured relative to the specified target. The object to be measured includes the target vehicle and other targets to be measured. Therefore, for the intersection warning device to sense the orientation of the object to be measured, the controller needs to convert the radar detection data into the radar coordinate system. Among them, the radar coordinate system can use the actual position of the radar sensor itself at the road intersection as the coordinate origin.

[0095] Since the signal frequencies output by the radar and the camera are different, and the time of radar data output is later than the time of camera data output, the information of the data detected by the radar sensor and the data collected by the image sensor is inconsistent in terms of time and space. Therefore, it is first necessary to determine the time of the target radar detection data, and then align the time of the target radar detection data with the time stamp of the image data, so as to be able to align the data detected by the radar sensor and the data collected by the image sensor in time.

[0096] Step S30: Based on the image coordinate system and the radar coordinate system, synchronize the image data and the target radar detection data to determine the speed information and position information of the object to be measured.

[0097] It can be understood that the above step S20 has synchronized the data detected by the radar sensor and the data collected by the image sensor in terms of time. Therefore, this step S30 needs to perform spatial synchronization processing on the data detected by the radar sensor and the data collected by the image sensor:

[0098] Specifically, this embodiment will obtain the coordinate mapping relationship between the image coordinate system and the radar coordinate system; based on the coordinate mapping relationship, synchronize the image data and the target radar detection data to determine the speed information and position information of the object to be measured.

[0099] In a specific implementation, a preset coordinate system needs to be established. The preset coordinate system can be a coordinate system with the center of the intersection as the origin of the absolute coordinate system, or it can be the ego - vehicle coordinate system of the target vehicle. Among them, the size data of the road in the preset coordinate system can be exactly the same as the actual size information such as the actual length and actual width of the real - world road section. Map the image data and the target radar detection data into a preset coordinate system, complete the coordinate transformation, and synchronize the coordinate positions of the image data and the target radar detection data. Refer to Figure 5 , and it specifically includes the following sub - steps:

[0100] Sub - step S301: Obtain the first coordinate mapping relationship between the image coordinate system and the preset coordinate system; obtain the second coordinate mapping relationship between the radar coordinate system and the preset coordinate system;

[0101] Sub - step S302: Map the image data to the preset coordinate system based on the first coordinate mapping relationship to obtain the first mapped data;

[0102] Sub - step S303: Map the target radar detection data to the preset coordinate system based on the second coordinate mapping relationship to obtain the second mapped data;

[0103] It can be understood that steps S301 to S303 complete the data conversion of the coordinate system;

[0104] Sub - step S304: Compare the first mapped data with the second mapped data. If the comparison result meets the preset difference threshold, it is determined that the image data and the target radar detection data match, and the position information and speed information of the object to be measured are determined based on the first mapped data and the second mapped data.

[0105] It can be understood that the first mapped data can be understood as the data in a frame of the image sensor - collected data mapped to the preset coordinate system. This frame of data includes the speed, position distance of each object to be measured (including the target vehicle and other objects to be measured). The first mapped data also includes the image features of the object to be measured (for example, when the object to be measured is a car, the image features include license plate information, vehicle shape features, etc. The license plate information and vehicle shape features are used as the identification of the target vehicle; when the object to be measured is a pedestrian or a bicycle, the image features include the appearance features of the pedestrian or the rider); while the second mapped data can be understood as the data in another frame of the radar sensor - detected data mapped to the preset coordinate system. This frame of data includes the speed and position distance of each object to be measured (including the target vehicle and other objects to be measured);

[0106] Specifically, after the time is synchronized and the coordinate system data of the data target collected by the image sensor and the data target detected by the radar sensor are converted to the preset coordinate system, the speed of the object to be measured in the above-mentioned one frame (collected by the image sensor) is compared with the speed of the object to be measured in the above-mentioned another frame (detected by the radar sensor), and the position distance of the object to be measured in the above-mentioned one frame is compared with the position distance of the object to be measured in the above-mentioned another frame. If the speed difference and the position distance difference both meet the preset difference threshold, it is determined that the image data collected by the image sensor and the target radar detection data detected by the radar sensor are the same frame of target image information.

[0107] Step S40, obtaining the impending collision time between the target vehicle and other targets to be measured based on the speed information and position information of the detected object, wherein the detected object includes the target vehicle and other targets to be measured;

[0108] In a specific implementation, the target vehicle is first determined from each detected object, including: determining the first position information of the target vehicle in the preset coordinate system based on the image data and the target radar detection data; matching the first position information with the position information of each detected object according to a first preset matching rule, obtaining a detected object vehicle that successfully matches the first position information, and using the speed information of the detected object vehicle as the vehicle speed information of the target vehicle;

[0109] Then determining the second position information and the second speed information of other targets to be measured except the vehicle to be measured;

[0110] Then predict the driving trajectory of the target vehicle and other targets to be tested, including:

[0111] Based on the first position information, a first driving trajectory of the target vehicle in the preset coordinate system is obtained; specifically, the coordinates of the historical trajectory points of the target vehicle are recorded, and cubic spline interpolation is used for fitting to predict the motion state of the target vehicle at the next moment.

[0112] Based on the second position information, a second driving trajectory of the other target to be measured in the preset coordinate system is obtained; specifically, for other targets to be measured, if it is a vehicle, the coordinates of the vehicle's historical trajectory points are fitted using cubic spline interpolation to predict the vehicle's motion state at the next moment; if the other target to be measured is a pedestrian, due to the uncertainty of the pedestrian's motion trajectory, it is difficult to construct a motion model, so a convolutional neural network algorithm is used for prediction, that is, a data set needs to be collected in advance, the pedestrian's trajectory motion characteristics need to be extracted, the model is trained, and the prediction result is output.

[0113] Then, based on the first driving trajectory and the second driving trajectory, determine the relative distance between the other target to be measured and the target vehicle.

[0114] Finally, calculate the time to collision between the target vehicle and the other target to be measured based on the relative distance, the vehicle speed information, and the second speed information.

[0115] For example, the path trajectory position coordinates (x1, y1) of the target vehicle are obtained, the path trajectory position coordinates (x2, y2) of the other target to be measured are obtained, and the Euclidean distance (i.e., the relative distance) between the target vehicle and the other target to be measured is S = sqrt((x1 - x2) 2 +(y1 - y2) 2 ). Calculate in real time the time (time to collision) required for the target vehicle and the other target to be measured to continue moving until a collision occurs in the current motion state.

[0116] Step S50: Send the time to collision to the in-vehicle terminal corresponding to the identifier of the target vehicle, so that the in-vehicle terminal issues a warning message based on the time to collision.

[0117] In a specific implementation, the intersection warning terminal can send the time to collision and the identifier of the target vehicle to the server, so that the server determines the corresponding in-vehicle terminal according to the identifier of the target vehicle and sends the time to collision to the in-vehicle terminal.

[0118] It can be understood that the server stores the identifier information of each vehicle. The identifier information can be the license plate number. For each vehicle whose identifier information is stored in the server, an in-vehicle terminal owner account will be associated with the user terminal of the vehicle owner. The in-vehicle terminal can be a terminal device installed on the vehicle or a mobile terminal device such as the vehicle owner's mobile phone. This embodiment does not limit this here.

[0119] Specifically, when the target vehicle travels to the intersection where the intersection warning device is installed, the intersection warning device will collect the image data of the target vehicle and obtain the radar detection data of the target vehicle. At the same time, the display screen of the in-vehicle terminal of the target vehicle prompts the driver to pass through the intersection, and then the in-vehicle terminal receives the time to collision generated by the intersection warning device;

[0120] At the same time, the in-vehicle terminal obtains the current position information of the vehicle itself; generates a warning message based on the current position information and the time to collision;

[0121] The in-vehicle terminal can make control and prompting decisions for the vehicle based on the received current position information of the vehicle itself and the time to collision. There are three collision levels in total. When the system determines that there is no collision risk, the vehicle terminal will give voice and display text prompts to pay attention to vehicles or pedestrians and drive carefully; when the in-vehicle terminal determines that the collision risk is relatively low, the vehicle terminal will give voice and display text prompts to pay attention to pedestrians and trigger the buzzer alarm; when the in-vehicle terminal determines that the collision risk is relatively high, the vehicle terminal will give simultaneous alarm prompts to pedestrians through voice, display text, and buzzer: there are dangerous vehicle or pedestrian targets, please avoid urgently, and control the vehicle to decelerate or brake to prevent the driver from being unable to make effective operations within the collision time and endangering the safety of himself and others. It realizes long-distance early warning and reminder for the target vehicle, is not restricted by the field of view of the vehicle itself, and effectively prevents potential safety hazards caused by dangerous targets such as "ghost probes" being too close and drivers' late reactions, ensuring the safety of drivers to the greatest extent.

[0122] Reference Figure 6 , Figure 6 Fig. shows a schematic flow chart of the second embodiment of the early warning method provided by the present application. As an example but not a limitation, in this embodiment, the radar sensor includes a lidar sensor and a millimeter-wave radar sensor;

[0123] In this embodiment, the lidar sensor is a 128-line lidar with a 360-degree horizontal field of view and a detection range of 200m; the horizontal field of view of the millimeter-wave radar sensor is 120°, and the detection range is 170m; the camera corresponding to the image sensor has 8 million pixels, a horizontal field of view of 120°, and a detection range of 170m. In order to cover the intersection scene without dead angles in 360°, a total of 3 millimeter-wave radar sensors, 3 cameras, and 1 lidar sensor are installed in the system. It is necessary to ensure that the horizontal fields of view of the millimeter-wave radars and the horizontal fields of view of the cameras coincide, and the installation height should ensure that there are no blind spots in the sensor fields of view. The installation height of the lidar is 1400 - 1600mm, the installation height of the camera is 1200 - 1400mm, and the installation height of the millimeter-wave radar is 300 - 800mm, and the angle installation error is within ±3°.

[0124] Correspondingly, the step S20 specifically includes:

[0125] Sub-step S201, obtaining the laser detection data of the object to be measured (detected by the lidar sensor), and the laser detection data includes the lidar coordinate system;

[0126] Sub-step S202, obtaining the millimeter-wave detection data of the object to be measured (detected by the millimeter-wave sensor); the millimeter-wave detection data includes the millimeter-wave radar coordinate system; wherein, the time stamps of the laser detection data and the millimeter-wave detection data are aligned;

[0127] Further, the step of obtaining the second coordinate mapping relationship between the radar coordinate system and the preset coordinate system further includes:

[0128] Sub-step S031, obtaining a third mapping relationship between the lidar coordinate system and the preset coordinate system; obtaining a fourth mapping relationship between the millimeter-wave radar coordinate system and the preset coordinate system;

[0129] Correspondingly, the sub-step S303 further includes:

[0130] Sub-step S032, mapping the lidar detection data to the preset coordinate system based on the third mapping relationship to obtain third mapped data; mapping the millimeter-wave detection data to the preset coordinate system based on the fourth mapping relationship to obtain fourth mapped data;

[0131] Correspondingly, before the sub-step S304, it further includes:

[0132] Sub-step S033, comparing the third mapped data with the fourth mapped data. If the comparison result meets a preset difference threshold, it is determined that the lidar detection data and the millimeter-wave detection data match.

[0133] Further, if the lidar detection data and the millimeter-wave detection data match, use the third mapped data or the fourth mapped data as the above-mentioned second mapped data, and then execute the sub-step S304.

[0134] It can be understood that inertial navigation units are included in the lidar sensor (corresponding to the third mapped data), the millimeter-wave radar sensor (corresponding to the fourth mapped data), and the image sensor of the camera (corresponding to the first mapped data). Using the GPS signal as the clock source, time stamps are provided for the data of each sensor. Among them, the lidar output signal frequency is 10hz, the millimeter-wave radar output signal frequency is 15hz, and the camera output signal frequency is 36hz. The target signal frequency output by the lidar is the lowest, and the relative output time of the data is relatively later than that of the camera and the millimeter-wave radar. Therefore, the time stamps of each sensor can be sorted in real time, and the time stamp difference Δt of each sensor is calculated. Then, predictions are made based on the target horizontal and vertical distances (DX, DY), target horizontal and vertical speeds (VX, VY), and target horizontal and vertical accelerations (AX, AY) output by each sensor:

[0135] Let the distance to be compensated for the object measured by the camera be L s (m), and the compensated speed be V s ′(m / s);

[0136]

[0137] Vs ' = V s + V s Δt s

[0138] Where: V s is the vehicle speed of the object under test output by the camera, V s ' is the vehicle speed of the object under test output after camera compensation, Δt s is the difference between the camera timestamp and the lidar output target timestamp, A s is the acceleration of the object under test output by the camera.

[0139] Let the distance to be compensated for the object under test output by the millimeter-wave radar be L h (m), and the compensated speed be V h (m / s);

[0140]

[0141] V h ' = V h + V h Δt h

[0142] Where: V h is the vehicle speed of the object under test output by the millimeter-wave radar sensor, V h ' is the vehicle speed of the object under test output after prediction by the millimeter-wave radar sensor, Δt h is the difference between the millimeter-wave radar timestamp and the lidar output target timestamp, A h is the acceleration of the object under test output by the millimeter-wave radar.

[0143] Through the above calculation process, sensor time synchronization and data frame alignment are further completed to ensure that the output target attributes of each sensor are for the same frame target.

[0144] After completing the time synchronization of each sensor, it is also necessary to complete the spatial synchronization of each sensor's target. Due to the problem of computing power consumption, a post-fusion scheme based on Kalman filtering is adopted. Among them, considering the advantages and disadvantages of sensors in different scenarios, the target lateral attribute information refers to the lidar, the longitudinal target attribute refers to the millimeter-wave radar, and the target type refers to the camera. The mapped sensor data of each is spatially synchronized. Principle: According to three targets (the three sensors output three mapped data), they need to be recognized as one target; after time synchronization and after the coordinate systems of the three targets are all converted to the preset coordinate system, according to the attribute values they output (according to the speed threshold and distance threshold under two frames of pictures) to determine that they are the same target.

[0145] Among them, the specific matching process is as follows:

[0146] Output each frame of data of each sensor after time synchronization is completed in real time, pair the data of each sensor one by one, calculate the horizontal and vertical distances and horizontal and vertical speeds of each pair of targets, and make the difference between the horizontal and vertical distances and horizontal and vertical speed data of each sensor. The pair with the smallest difference is successfully associated.

[0147] Furthermore, the process of calculating the impending collision time in step S40 specifically includes:

[0148] Determining a lateral velocity and a lateral acceleration of the object under test based on the third mapping data (corresponding to the laser radar sensor);

[0149] Determining the longitudinal velocity and longitudinal acceleration of the object under test based on the fourth mapping data (corresponding to the millimeter wave radar sensor);

[0150] Determining acceleration information of the object to be measured based on the lateral acceleration and the longitudinal acceleration, wherein the acceleration information of the object to be measured includes acceleration information of the target vehicle and acceleration information of the other targets to be measured;

[0151] The speed information of the object to be measured is determined based on the lateral speed and the longitudinal speed, and the speed information of the object to be measured includes the second speed information of the other objects to be measured and the speed information of the target vehicle; accordingly, the step of calculating the impending collision time between the target vehicle and the other objects to be measured based on the relative distance, the speed information, and the second speed information in step S40 further includes:

[0152] Calculate the impending collision time between the target vehicle and the other targets to be measured based on a preset safe stopping distance (which may be a distance value set by a staff member on the side of the intersection warning device based on actual needs), the relative distance, the vehicle speed information, the second speed information, the acceleration information of the target vehicle and the acceleration information of the other targets to be measured;

[0153] The impending collision time between the target vehicle and other targets to be detected is obtained through the following process:

[0154]

[0155] In the above formula, represents the travel displacement of the other target to be measured when it starts to travel at speed V2 until a collision occurs, It represents the driving displacement generated when the target vehicle starts driving from speed V1 until a collision occurs; t is the upcoming collision time, S is the relative distance between the other target to be measured and the target vehicle, d is the preset safe stopping distance, V1 is the vehicle speed information of the target vehicle, A1 is the acceleration information of the target vehicle, V2 is the second speed information of the other target to be measured, and A2 is the acceleration information of the other target to be measured;

[0156] Solving the above formula, the expression for the upcoming collision time t can be obtained as:

[0157]

[0158] Record the distribution of the upcoming collision time t based on the collision distance, and thus judge the magnitude of the collision risk. Estimate the collision probability P based on the statistical distribution value (this threshold can be adjusted according to user feedback). When P < 5%, it is determined that there is no risk. When 5% < P < 20%, it is determined that the risk is relatively low. When P > 20%, it is determined that the risk is relatively high, and send the upcoming collision time t to an external server.

[0159] This embodiment adopts a multi-sensor fusion scheme, effectively solving the problems of the camera failing in rainy, foggy, snowy weather and at night, the millimeter-wave radar being vulnerable to electromagnetic interference and having poor lateral detection performance for stationary targets, and the weak penetration and light interference susceptibility of lidar, etc. The system has higher fault tolerance and better redundant safety.

[0160] Reference Figure 7 , Figure 7 Fig. shows a schematic flowchart of the third embodiment of the warning method provided by the present application. As an example but not a limitation, this method can be applied to the in-vehicle terminal 20 of the target vehicle. The method includes:

[0161] Step S01, receiving the upcoming collision time generated by the intersection warning device;

[0162] Step S02, obtaining the current position information of the target vehicle;

[0163] Step S03, generating a warning message based on the current position information and the upcoming collision time;

[0164] In a specific implementation, the intersection warning terminal can send the upcoming collision time and the identifier of the target vehicle to the server, so that the server determines the corresponding in-vehicle terminal according to the identifier of the target vehicle and sends the upcoming collision time to the in-vehicle terminal.

[0165] It is understandable that the server stores the identification information of each vehicle. The identification information can be the license plate number. For each vehicle whose identification information is stored in the server, a vehicle owner account of an in-vehicle terminal is associated with the user terminal of the vehicle owner. The in-vehicle terminal can be a terminal device installed on the vehicle or a mobile terminal device such as the vehicle owner's mobile phone. This embodiment does not limit this here.

[0166] Specifically, when the target vehicle travels to an intersection where an intersection warning device is installed, the intersection warning device collects the image data of the target vehicle and obtains the radar detection data of the target vehicle. At the same time, the display screen of the in-vehicle terminal of the target vehicle prompts the driver to pass through the intersection, and then the in-vehicle terminal receives the time to collision generated by the intersection warning device;

[0167] At the same time, the in-vehicle terminal obtains the current position information of the vehicle itself; based on the current position information and the time to collision, a warning message is generated;

[0168] The in-vehicle terminal can make control and prompt decisions for the vehicle itself according to the received current position information of the vehicle itself and the time to collision. There are three collision levels in total. When the system determines that there is no collision risk, the vehicle-end voice and the display screen text prompt to pay attention to vehicles or pedestrians and drive carefully; when the in-vehicle terminal determines that the prompt collision risk is relatively low, the vehicle-end voice and the display screen text prompt to pay attention to pedestrians and trigger the buzzer alarm; when the in-vehicle terminal determines that the collision risk is relatively high, the vehicle-end voice, the display screen text and the buzzer alarm at the same time to prompt pedestrians: there is a dangerous vehicle or pedestrian target, please avoid urgently, and control the vehicle to decelerate or brake to prevent the driver from being unable to make effective operations within the collision time and endangering the safety of himself and others.

[0169] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0170] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a controller, the steps in the above various method embodiments can be implemented.

[0171] When the terminal device loaded with a computer program is sold or used as an independent product, the computer program can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above-mentioned embodiment methods in this application, it can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a controller, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the photographing device / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disc, etc. In some jurisdictions, according to legislation and patent practice, the computer-readable medium cannot be an electrical carrier signal and a telecommunication signal.

[0172] In the above embodiments, the descriptions of the various embodiments have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0173] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in this article can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0174] The above-mentioned embodiments are only used to illustrate the technical solutions of this application, rather than to limit it; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An early warning method, characterized in that, The method is applied to a road intersection warning device, and the method comprises: Acquiring image data, wherein the image data includes an image coordinate system and an identification of a target vehicle; Acquire target radar detection data of the object under test, wherein the target radar detection data includes a radar coordinate system; wherein the timestamps between the target radar detection data and the image data are aligned; Based on the image coordinate system and the radar coordinate system, the image data is synchronized with the target radar detection data to determine the speed information and position information of the measured object; Acquire the impending collision time between the target vehicle and other targets to be measured based on the speed information and position information of the detected object, wherein the detected object includes the target vehicle and other targets to be measured; The imminent collision time is sent to a vehicle-mounted terminal corresponding to the identification of the target vehicle, so that the vehicle-mounted terminal issues a warning message based on the imminent collision time.

2. The method according to claim 1, characterized in that, The step of synchronizing the image data with the target radar detection data based on the image coordinate system and the radar coordinate system to determine the speed information and position information of the object under test includes: Acquire a coordinate mapping relationship between the image coordinate system and the radar coordinate system; The image data and the target radar detection data are synchronized based on the coordinate mapping relationship to determine the speed information and position information of the detected object.

3. The method according to claim 2, characterized in that, The acquiring the coordinate mapping relationship between the image coordinate system and the radar coordinate system comprises: Acquire a first coordinate mapping relationship between the image coordinate system and a preset coordinate system; Acquire a second coordinate mapping relationship between the radar coordinate system and a preset coordinate system; The step of synchronizing the image data with the target radar detection data based on the coordinate mapping relationship to determine the speed information and position information of the object under test includes: Mapping the image data to the preset coordinate system based on the first coordinate mapping relationship to obtain first mapping data; Mapping the target radar detection data to the preset coordinate system based on the second coordinate mapping relationship to obtain second mapping data; The first mapping data is compared with the second mapping data. If the comparison result meets a preset difference threshold, it is determined that the image data and the target radar detection data match, and the position information and speed information of the object under test are determined based on the first mapping data and the second mapping data.

4. The method according to claim 3, characterized in that, Obtaining the impending collision time between the target vehicle and other targets to be measured based on the speed information and position information of the measured object, including: Determine first position information of the target vehicle in the preset coordinate system based on the image data and the target radar detection data; Matching the first position information with the position information of each detected object according to a first preset matching rule to obtain a detected object vehicle that successfully matches the first position information, and using the speed information of the detected object vehicle as the speed information of the target vehicle; Determine second position information and second speed information of other targets to be measured except the vehicle to be measured; Obtain the first driving trajectory of the target vehicle in the preset coordinate system based on the first position information; Obtain the second driving trajectory of the other target to be measured in the preset coordinate system based on the second position information; Determine the relative distance between the other target to be measured and the target vehicle based on the first driving trajectory and the second driving trajectory; Calculate the time to collision between the target vehicle and the other target to be measured based on the relative distance, the vehicle speed information, and the second speed information.

5. The method according to any one of claims 2-4, characterized in that, The step of obtaining the target radar detection data of the object to be measured includes: Obtain the laser detection data of the object to be measured, and the lidar detection data includes the lidar coordinate system; Obtain the millimeter-wave detection data of the object to be measured; the millimeter-wave detection data includes the millimeter-wave radar coordinate system; wherein, the time stamps of the laser detection data and the millimeter-wave detection data are the same; The step of obtaining the second coordinate mapping relationship between the radar coordinate system and the preset coordinate system includes: Obtain the third mapping relationship between the lidar coordinate system and the preset coordinate system; Obtain the fourth mapping relationship between the millimeter-wave radar coordinate system and the preset coordinate system; The step of mapping the target radar detection data to the preset coordinate system based on the second coordinate mapping relationship to obtain the second mapping data specifically includes: Map the laser detection data to the preset coordinate system based on the third mapping relationship to obtain the third mapping data; Map the millimeter-wave detection data to the preset coordinate system based on the fourth mapping relationship to obtain the fourth mapping data; Before the step of comparing the first mapping data and the second mapping data, if the comparison result meets the preset difference threshold, it is determined that the image data and the target radar detection data match, and the position information and speed information of the object to be measured are determined based on the first mapping data and the second mapping data, further including: Compare the third mapping data and the fourth mapping data, if the comparison result meets the preset difference threshold, it is determined that the laser detection data and the millimeter-wave detection data match.

6. The method according to claim 5, characterized in that, The method further includes: Determine the lateral speed and lateral acceleration of the object to be measured based on the third mapping data; Determine the longitudinal speed and longitudinal acceleration of the object to be measured based on the fourth mapping data; Determine the acceleration information of the object to be measured based on the lateral acceleration and the longitudinal acceleration, and the acceleration information of the object to be measured includes the acceleration information of the target vehicle and the acceleration information of the other target to be measured; Determine the speed information of the object to be measured based on the lateral speed and the longitudinal speed, and the speed information of the object to be measured includes the second speed information of the other target to be measured and the vehicle speed information of the target vehicle; The step of calculating the time to collision between the target vehicle and the other target to be measured based on the relative distance, the vehicle speed information, and the second speed information further includes: Calculating an impending collision time between the target vehicle and other targets to be detected based on a preset safe stopping distance, the relative distance, the vehicle speed information, the second speed information, the acceleration information of the target vehicle, and the acceleration information of the other targets to be detected; The time of impending collision between the target vehicle and other targets to be measured is calculated by the following formula: Among them, t is the impending collision time, S is the relative distance between the other target to be measured and the target vehicle, d is the preset safe stopping distance, V1 is the speed information of the target vehicle, 1 is the acceleration information of the target vehicle, V2 is the second speed information of the other target to be measured, and A2 is the acceleration information of the other target to be measured.

7. The method according to any one of claims 2 - 4, characterized in that, The step of sending the impending collision time to a vehicle-mounted terminal corresponding to the identifier of the target vehicle includes: The impending collision time and the identifier of the target vehicle are sent to a server, so that the server determines a corresponding vehicle-mounted terminal according to the identifier of the target vehicle, and sends the impending collision time to the vehicle-mounted terminal.

8. An early warning method, characterized in that, The early warning method is applied to an on-board terminal of a target vehicle, and the method comprises: Receive the impending collision time generated by the intersection warning device; Obtaining current location information of the target vehicle; generating warning information based on the current position information and the impending collision time; Wherein, the intersection warning device is used for: Acquire image data collected by an image sensor of the intersection warning device, wherein the image data includes an image coordinate system and an identification of a target vehicle; Acquire target radar detection data of the object under test detected by the radar sensor, wherein the target radar detection data includes a radar coordinate system; wherein the target radar detection data has the same timestamp as the image data; Acquire a coordinate mapping relationship between the image coordinate system and the radar coordinate system; Based on the coordinate mapping relationship, the image data is synchronized with the target radar detection data to determine the speed information and position information of the measured object; The impending collision time between the target vehicle and other targets to be measured is obtained based on the speed information and position information of the detected object, wherein the detected object includes the target vehicle and other targets to be measured.

9. A terminal device, characterized in that, When the terminal device is a road intersection warning device, the road intersection warning device is provided with an image sensor, a radar sensor, a controller, a memory, and a computer program stored in the memory and executable on the controller; the image sensor is used to collect image data; the radar sensor is used to detect target radar detection data of the object being detected; When the controller executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented; When the terminal device is a vehicle-mounted terminal, it includes a memory, a controller, and a computer program stored in the memory and executable on the controller, wherein the controller implements the method according to claim 8 when executing the computer program.

10. A computer storage medium, the computer storage medium is a computer-readable storage medium, and the computer-readable storage medium stores a computer program, characterized in that, When the computer program is executed by a controller, the method according to any one of claims 1 to 8 is implemented.

Images