Safety warning method, device and equipment

Abstract

The application provides a safety warning method, device and equipment. In a case that a parking state is determined and a passenger in the vehicle unfastens a safety belt, an original image and a radar image outside the vehicle are acquired in real time, the original image and the radar image are processed, a second moving target is finally obtained, a speed of the second moving target and a distance between the second moving target and the vehicle are calculated, and whether there is a collision risk between the passenger in the vehicle and the second moving target when the passenger gets off the vehicle is determined. If it is determined that there is a collision risk between the passenger in the vehicle and the second moving target when the passenger gets off the vehicle, a safety warning message is pushed to the passenger in the vehicle. In this way, the clarity of the second moving target image is improved, and the misjudgment rate of the second moving target is reduced, so that the purpose of safety warning for the passenger in the vehicle is achieved.

Description

Technical Field

[0001] This application relates to the field of automotive technology, and more particularly to a safety warning method, device, and equipment. Background Technology

[0002] As society progresses and people's living standards improve, private vehicles are increasingly replacing traditional modes of transportation. However, with the increase in vehicles, so too are the dangers. Cars travel at very high speeds, and when a vehicle stops and passengers get out to open the door, it's easy to overlook oncoming vehicles or pedestrians, leading to traffic accidents and injuries. Therefore, people are increasingly paying attention to what's behind their vehicles when getting out.

[0003] In existing technologies, when a vehicle is parked and passengers are getting out, passengers usually take measures such as observing the vehicle and operating the door in a standardized manner to reduce safety hazards. Additionally, cameras or radar installed on the outside of the vehicle can monitor moving objects behind the vehicle to alert passengers inside.

[0004] However, the above solutions rely on human subjective awareness and observation, and there may be blind spots when opening the car door, which still poses certain safety hazards. Real-time monitoring of the external environment through cameras can be achieved, but cameras are affected by weather conditions. If radar monitoring is used, the image recognition accuracy is low and misjudgment is easy to occur. Summary of the Invention

[0005] This application provides a safety warning method, device, and equipment to address the problem that when a vehicle is parked and passengers get out and open the door, they easily overlook vehicles or pedestrians approaching from behind, thus easily causing traffic accidents.

[0006] In a first aspect, this application provides a safety warning method applied to a vehicle, the method comprising:

[0007] After determining that the vehicle is in a parked state and detecting that the occupant has unbuckled their seatbelt, the system acquires raw images and radar images of the vehicle's exterior in real time.

[0008] Image recognition processing is performed on the original image and the radar image to identify at least one first moving target around the vehicle;

[0009] Based on the radar image and a pre-set safety range, determine the radar image of at least one second moving target within the safety range from the at least one first moving target;

[0010] Based on the radar images of the at least one second moving target at different times, the moving speed of each second moving target and its distance from the vehicle are calculated.

[0011] For each second moving target, based on the moving speed of the second moving target and the distance between the second moving target and the vehicle, it is determined whether there is a risk of collision between the occupants of the vehicle and the second moving target when they get off the vehicle;

[0012] When it is determined that there is a risk of collision with the second moving target when the occupants of the vehicle are getting off, a safety warning message is pushed to the occupants inside the vehicle. The safety warning message is used to remind the occupants of the risk of collision with the second moving target outside the vehicle.

[0013] In conjunction with the first aspect, in some embodiments, pushing safety warning messages to occupants inside the vehicle includes:

[0014] An image of the second moving target is displayed on a flexible display screen mounted on the B-pillar of the vehicle's interior, and a warning audio is played through the vehicle's speakers. The safety warning message includes the image of the second moving target and the warning audio.

[0015] In conjunction with the first aspect, in some embodiments, the method further includes:

[0016] The original image and the radar image are subjected to sharpness recognition processing respectively to obtain the first distortion rate of the original image and the second distortion rate of the radar image;

[0017] The image fusion weight is determined based on the first distortion rate and the second distortion rate. The image fusion weight is used to indicate the ratio of radar image data to camera image data during image fusion.

[0018] According to the image fusion weights, the radar image of the at least one second moving target is fused with the original image to obtain a fused image;

[0019] Extract the image of the second moving target from the fused image.

[0020] In conjunction with the first aspect, in some embodiments, determining the radar image of at least one second moving target within the safety range from the at least one first moving target, based on the radar image and a pre-set safety range, includes:

[0021] The radar image is processed into a point cloud to obtain radar image data point cloud.

[0022] Each first moving target is identified based on the radar image data point cloud, and the target point cloud corresponding to each first moving target is obtained;

[0023] For each first moving target, based on the position of the target point cloud of the first moving target in the radar image data point cloud, image extraction is performed on the radar image to obtain the target radar image corresponding to the first moving target;

[0024] Based on the target point cloud of each first moving target, the target radar image corresponding to each first moving target, and the pre-set safety range, obtain the radar image of at least one second moving target within the safety range.

[0025] In conjunction with the first aspect, in some embodiments, the method further includes: determining whether the vehicle is in a parked state based on the signal of the vehicle's CAN bus;

[0026] If the vehicle is parked, the system determines whether the occupant's seatbelt is unfastened based on the signal from the sensor located at the seatbelt buckle in the vehicle.

[0027] In conjunction with the first aspect, in some embodiments, the real-time acquisition of raw images and radar images of the vehicle's exterior includes:

[0028] The original image is obtained by taking pictures of the external environment of the vehicle using a camera installed on the exterior of the vehicle.

[0029] The exterior of the vehicle is detected by a millimeter-wave radar installed on the exterior of the vehicle, and a radar image is obtained.

[0030] In conjunction with the first aspect, in some embodiments, determining whether there is a risk of collision between the vehicle occupants and the second moving target when the second moving target alights, based on the moving speed of the second moving target and the distance between the second moving target and the vehicle, includes:

[0031] If the moving speed of the second moving target is greater than a preset speed threshold, and the distance between the second moving target and the vehicle is less than a preset distance, then it is determined that there is a risk of collision between the occupants of the vehicle and the second moving target when they get off the vehicle.

[0032] Secondly, this application provides a safety warning device for use in a vehicle, the device comprising:

[0033] The information acquisition module is used to acquire raw images and radar images of the exterior of the vehicle in real time after determining that the vehicle is in a parked state and detecting that the occupant has unbuckled their seat belt.

[0034] A first processing module is used to perform image recognition processing on the original image and the radar image to determine at least one first moving target around the vehicle;

[0035] The second processing module is used to determine, based on the radar image and a pre-set safety range, the radar image of at least one second moving target within the safety range from the at least one first moving target;

[0036] The third processing module calculates the moving speed of each second moving target and its distance from the vehicle based on the radar images of the at least one second moving target at different times.

[0037] The fourth processing module is used to determine, for each second moving target, whether there is a risk of collision between the vehicle occupants and the second moving target when the second moving target is disembarking, based on the moving speed of the second moving target and the distance between the second moving target and the vehicle.

[0038] The safety alert module is used to push a safety warning message to the occupants of the vehicle when it is determined that there is a risk of collision with the second moving target when the occupants of the vehicle get off the vehicle. The safety warning message is used to remind the occupants of the risk of collision with the second moving target outside the vehicle.

[0039] In conjunction with the second aspect, in some embodiments, the security notification module includes:

[0040] The image prompting unit is used to play an image of the second moving target through a flexible display screen installed on the B-pillar of the vehicle's interior;

[0041] The voice prompt unit plays warning audio through the vehicle's speakers.

[0042] In conjunction with the second aspect, in some embodiments, the apparatus further includes:

[0043] The image analysis module is used to perform sharpness recognition processing on the original image and the radar image respectively, to obtain the first distortion rate of the original image and the second distortion rate of the radar image;

[0044] The fifth processing module is used to determine the image fusion weight based on the first distortion rate and the second distortion rate. The image fusion weight is used to indicate the proportion of radar image data and camera image data during image fusion.

[0045] An image fusion module is used to fuse the radar image of the at least one second moving target with the original image according to the image fusion weights to obtain a fused image;

[0046] An image extraction module is used to extract the image of the second moving target from the fused image.

[0047] In conjunction with the second aspect, in some embodiments, the second processing module includes:

[0048] A point cloud processing unit is used to perform point cloud processing on the radar image to obtain radar image data point cloud.

[0049] The information acquisition unit is used to identify each first moving target based on the radar image data point cloud and acquire the target point cloud corresponding to each first moving target;

[0050] The image extraction unit is used to extract images from the radar image based on the position of the target point cloud of the first moving target in the radar image data point cloud for each first moving target, and obtain the target radar image corresponding to the first moving target.

[0051] The information processing unit is configured to acquire radar images of at least one second moving target within the safety range based on the target point cloud of each first moving target, the target radar image corresponding to each first moving target, and the pre-set safety range.

[0052] In conjunction with the second aspect, in some embodiments, the apparatus further includes:

[0053] The sixth processing module is used to determine whether the vehicle is in a parked state based on the signal from the vehicle's CAN bus.

[0054] The seventh processing module is used to determine whether the occupant's seat belt is unfastened based on the signal from the sensor located at the seat belt buckle of the vehicle if the vehicle is in a parked state.

[0055] In conjunction with the second aspect, in some embodiments, the information acquisition module includes:

[0056] The camera acquisition unit is used to capture images of the external environment of the vehicle using a camera installed on the exterior of the vehicle, thereby obtaining the original image;

[0057] A radar acquisition unit is used to detect the exterior of the vehicle using a millimeter-wave radar installed on the exterior of the vehicle, and obtain the radar image.

[0058] In conjunction with the second aspect, in some embodiments, the fourth processing module is specifically used for:

[0059] If the moving speed of the second moving target is greater than a preset speed threshold, and the distance between the second moving target and the vehicle is less than a preset distance, then it is determined that there is a risk of collision between the occupants of the vehicle and the second moving target when they get off the vehicle.

[0060] Thirdly, embodiments of this application also provide a vehicle, including:

[0061] Cameras, millimeter-wave radar, flexible displays, speakers, memory, processors;

[0062] The cameras are installed on both sides of the lower end of the exterior rearview mirrors of the vehicle.

[0063] The millimeter-wave radar is installed on both sides of the rear bumper of the vehicle.

[0064] The flexible display screen is installed on the B-pillar of the vehicle's interior;

[0065] The memory stores computer instructions;

[0066] The processor executes computer instructions stored in the memory, causing the vehicle equipment to perform the safety warning method described in the first aspect.

[0067] Fourthly, embodiments of this application also provide a storage medium storing a computer program;

[0068] When the computer program is executed, it implements the security alert method described in the first aspect.

[0069] The safety warning method, device, and equipment provided in this application address the issue that, when a vehicle is parked and rear passengers are opening the door to exit, they may easily overlook oncoming vehicles or pedestrians. Unlike existing technologies, this solution uses cameras and millimeter-wave radar mounted on the exterior of the vehicle to monitor oncoming vehicles or pedestrians in real time. By processing the real-time monitoring data, it identifies targets with collision risks and uses a flexible display screen and speakers to provide warning information to the occupants. This precise monitoring of the surrounding environment improves driving safety and significantly enhances the safety factor for occupants exiting the vehicle. Attached Figure Description

[0070] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0071] Figure 1 This is an application scenario diagram of the safety warning method provided in the embodiments of this application;

[0072] Figure 2 A flowchart illustrating an embodiment of the safety warning method provided in this application;

[0073] Figure 3 This is a schematic diagram showing the installation positions of the flexible display screen and speaker provided in an embodiment of this application;

[0074] Figure 4 A flowchart illustrating Embodiment 2 of the safety warning method provided in this application;

[0075] Figure 5 A flowchart illustrating Embodiment 3 of the safety warning method provided in this application;

[0076] Figure 6 This is a schematic diagram showing the installation positions of the camera and millimeter-wave radar provided in an embodiment of this application;

[0077] Figure 7 A flowchart illustrating a specific example of the safety warning method provided in this application's embodiments;

[0078] Figure 8 This is a schematic diagram of the structure of a safety warning device according to a first embodiment of the present application.

[0079] Figure 9 This is a schematic diagram of the structure of a second embodiment of the safety warning device provided in this application.

[0080] Figure 10 This is a schematic diagram of the structure of the safety warning device according to Embodiment 3 of this application.

[0081] Figure 11 This is a schematic diagram of the structure of the safety warning device embodiment four provided in this application.

[0082] Figure 12 This is a schematic diagram of the structure of the safety warning device embodiment five provided in this application.

[0083] Figure 13 A schematic diagram of the structure of the safety warning device according to Embodiment Six of this application;

[0084] Figure 14 This is a schematic diagram of the vehicle structure provided in an embodiment of this application.

[0085] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0086] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0087] First, let me explain the terms used in this application:

[0088] The Controller Area Network (CAN) is a vehicle communication bus through which many sensors and modules in a car are connected.

[0089] As people's living standards improve, vehicles are gradually replacing traditional modes of transportation, leading to a surge in the number of vehicles on the road. Along with this increase in vehicles, so too are the dangers. When a vehicle is parked, occupants opening the door to get out can easily overlook oncoming vehicles or pedestrians, potentially causing traffic accidents. To address this, occupants are increasingly aware of the area behind the vehicle when exiting, often relying on manual observation and proper door opening procedures. However, this method depends heavily on subjective awareness and observation skills. Installing cameras on the exterior of the vehicle can provide real-time monitoring of the external environment. However, cameras are affected by weather conditions, and the image recognition accuracy is low, making misjudgments possible.

[0090] To address the aforementioned issues, this application provides a safety warning method that, when a vehicle is parked and there are vehicles or pedestrians behind it, provides a safety warning to the occupants. Specifically, when a vehicle is parked, occupants typically rely on manual observation to open the door and exit, in order to avoid traffic accidents. During the research process, the inventors discovered that manual observation depends on the individual's subjective awareness and observation skills, and blind spots may exist when opening the door, posing a certain safety hazard. Installing a camera to observe the external environment in real time can avoid blind spots for occupants; however, cameras are easily affected by weather conditions, and the image recognition accuracy is low, easily leading to misjudgments. Considering these issues, the inventors investigated whether a camera and millimeter-wave radar could be used to monitor the external environment in real time, and whether image fusion of the original camera image and radar image could improve recognition accuracy and avoid misjudgments. Based on this, the technical solution in this application is proposed.

[0091] Figure 1 This is an application scenario diagram of the safety warning method provided in the embodiments of this application, such as... Figure 1As shown, when a vehicle is parked, it includes at least the parked vehicle, the occupants inside the vehicle, and a moving target behind the vehicle. The specific form of the parked vehicle and the occupants are not specifically limited, and the moving target behind the vehicle can be a moving vehicle or a pedestrian, etc. This application also does not specifically limit the moving target behind the vehicle.

[0092] This application does not impose any restrictions on the specific form of each device.

[0093] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0094] Figure 2 A flowchart illustrating an embodiment of the safety warning method provided in this application is shown below. Figure 2 As shown, this safety warning method is applied to vehicles and specifically includes the following steps:

[0095] S101: After determining that the vehicle is in a parked state and detecting that the occupant has unbuckled their seat belt, acquire raw images and radar images of the vehicle's exterior in real time.

[0096] In this step, based on the signals from the vehicle's CAN bus and the sensors at the occupant's seatbelt buckle, it is determined that the vehicle is parked and the occupant has unbuckled their seatbelt, thus indicating a tendency for the occupant to exit the vehicle. At this point, the vehicle's CAN bus initiates the occupant exit assistance safety reminder program. The vehicle uses cameras and millimeter-wave radar installed on its exterior to acquire real-time raw images and radar images of the vehicle's external environment. The millimeter-wave radar can be of various types, such as 4D millimeter-wave radar, and is not specifically limited in this embodiment.

[0097] In one specific implementation, when the vehicle speed is 0, the vehicle's CAN bus receives a signal indicating a speed of 0, determining that the vehicle is in a parked state. When an occupant unbuckles their seatbelt, a sensor installed at the seatbelt buckle detects this signal. Based on the vehicle's parked state and the unbuckled seatbelt signal, it is determined that the occupant intends to exit the vehicle. At this point, the vehicle's CAN bus initiates an occupant exit assistance safety reminder program. Furthermore, cameras and millimeter-wave radar installed on the exterior of the vehicle capture real-time images of the external environment, obtaining raw images from the cameras and radar images from the millimeter-wave radar. The cameras are installed on both sides of the lower end of the exterior rearview mirrors, and the millimeter-wave radar is installed on both sides of the rear bumper.

[0098] S102: Perform image recognition processing on the raw image and radar image to identify at least one first moving target around the vehicle.

[0099] In the specific implementation of this step, the vehicle obtains the original image of the external environment captured by the external camera and the radar image of the external millimeter-wave radar detected by the external millimeter-wave radar. The vehicle identifies and processes the original image and the radar image to obtain moving targets and stationary targets in the image, discards the stationary targets and retains the moving targets, thereby determining at least one first moving target around the vehicle.

[0100] S103: Based on the radar image and a pre-set safety range, determine the radar image of at least one second moving target within the safety range from at least one first moving target.

[0101] In this step, after obtaining at least one first moving target around the vehicle, point cloud processing is performed on the radar image to obtain the specific location of at least one first moving target around the vehicle in the radar image, and according to the preset safety range, the radar image of at least one second moving target within the safety range is determined from the at least one first moving target.

[0102] In one specific implementation, the radar image is processed into a point cloud to obtain a radar image data point cloud. Each first moving target is identified based on the radar image data point cloud, thereby obtaining a target point cloud corresponding to each first moving target. Based on the position of the target point cloud corresponding to each first moving target in the radar image data point cloud, image extraction is performed on the radar image to obtain a target radar image corresponding to the first moving target. Finally, based on the target point cloud of each first moving target and a pre-set safety range, radar images of at least one second moving target within the pre-set safety range are selected from the target radar images corresponding to the first moving targets. The pre-set safety range can be set according to the actual environment and is not specifically limited in this scheme.

[0103] S104: Based on radar images of at least one second moving target at different times, calculate the moving speed of each second moving target and its distance from the vehicle.

[0104] In the specific implementation of this step, in order to more accurately determine whether there is a collision risk between the vehicle and at least one second moving target and to avoid misjudgment, it is necessary to calculate the moving speed of each second moving target and its distance from the vehicle.

[0105] Specifically, the distance of at least one second moving target from the vehicle is determined based on the point cloud coordinate information in the point cloud data of at least one second moving target. The three-dimensional position of each second moving target in adjacent frames of the radar image is obtained from the radar point cloud data. Then, based on the three-dimensional coordinates of each second moving target, the relative velocity of each second moving target is obtained using methods such as differential analysis, filtering, and regression.

[0106] S105: For each second moving target, based on the moving speed of the second moving target and the distance between the second moving target and the vehicle, determine whether there is a risk of collision between the vehicle occupants and the second moving target when the occupants get off the vehicle.

[0107] In this step, after obtaining the moving speed of each second moving target and the distance of each second moving target from the vehicle, the risk of collision between the vehicle occupants and the second moving targets is determined based on the preset speed threshold and the pre-set safety range.

[0108] In one specific implementation, after obtaining each second moving target, its moving speed, and its distance from the vehicle, the moving speed of the second moving target is compared with a preset speed threshold, and the distance from the second moving target to the vehicle is compared with a preset safety range. If the moving speed of the second moving target is greater than the preset speed threshold and the distance from the second moving target to the vehicle is less than the preset safety range, then there is indeed a risk of collision between the second moving target and the vehicle; otherwise, there is no risk of collision. The preset speed threshold and the preset safety range can be set according to the actual environmental conditions and are not specifically limited in this embodiment.

[0109] S106: When it is determined that there is a risk of collision with a second moving target when the occupants of the vehicle are getting off, a safety warning message is sent to the occupants inside the vehicle.

[0110] In this step, once it is determined that there is a risk of collision with a second moving target when the vehicle occupants exit the vehicle, the vehicle's CAN bus will push safety warning messages such as images of the second moving target and warning audio to the occupants inside the vehicle to remind them of the risk of collision with a second moving target outside the vehicle.

[0111] In one specific implementation, it was determined that there was a risk of collision with a second moving target when the vehicle occupants disembarked. For example... Figure 3 As shown, at this time, the vehicle's CAN bus will project the real-time image of the second moving target outside the vehicle onto the flexible display screen on the B-pillar of the vehicle's interior. At the same time, a warning audio will be played through the vehicle's speakers to remind the occupants that there is danger outside when they open the door and to remind them to look at the image on the flexible display screen.

[0112] This embodiment provides a safety warning method applied to a vehicle. By installing a camera and millimeter-wave radar on the exterior of the vehicle, it monitors the external environment in real time. When the vehicle is parked or when occupants have unbuckled their seatbelts, it activates an occupant exit assistance safety reminder program. The method involves processing the raw images acquired by the camera and millimeter-wave radar to identify at least one first moving target. By performing point cloud processing on the radar image, at least one second moving target within a pre-set safety range is selected from the identified first moving targets. The movement speed and distance of the second moving target from the vehicle are calculated. Based on these calculations, it is determined whether there is a risk of collision between the occupants and the second moving target when exiting the vehicle. If a risk exists, an image of the second moving target is displayed on a flexible screen on the B-pillar of the vehicle's interior, and a warning audio is played through the vehicle's speakers, thereby achieving the purpose of safety warning to the occupants.

[0113] Figure 4 A flowchart illustrating Embodiment 2 of the safety warning method provided in this application is shown below. Figure 3 As shown, in the above method embodiment one, the specific implementation process of obtaining the second moving target image includes:

[0114] S201: Perform sharpness recognition processing on the original image and the radar image respectively to obtain the first distortion rate of the original image and the second distortion rate of the radar image.

[0115] In this specific implementation, after obtaining the original image and the radar image, sharpness identification processing is performed on the original image and the radar image using common methods in the field. Then, the first distortion rate of the original image and the second distortion rate of the radar image are calculated based on the image distortion rate function. The methods used for sharpness identification processing of the original image and the distortion rate function can be freely selected according to the actual situation, and this embodiment does not impose specific limitations.

[0116] S202: Determine the image fusion weights based on the first distortion rate and the second distortion rate.

[0117] In the specific implementation of this step, a first distortion rate and a second distortion rate are calculated, and the proportions of the original image data and radar image data in image fusion are determined based on the first distortion rate and the second distortion rate, thereby determining the image fusion weight.

[0118] S203: Based on the image fusion weights, fuse the radar image of at least one second moving target with the original image to obtain the fused image.

[0119] In this specific implementation, image fusion weights are determined, and the original image is fused with radar images of at least one second moving target. Specifically, the image fusion process mainly consists of the following steps:

[0120] The data layer processes and merges different physical parameters.

[0121] Feature layer processing involves fusing features extracted from the original image and radar images of at least one second moving target. These features include size, shape, texture, and contrast. Fusing these extracted features allows the useful features to be better represented.

[0122] The decision layer processes the data by fusing the recognition results of multiple sensors to make a global optimal decision based on the independent decision-making or classification of each sensor. It integrates the extracted features and the original image after recognition with the radar image of at least one second moving target according to certain rules to obtain the fused image.

[0123] S204: Extract the image of the second moving target from the fused image.

[0124] In this specific implementation, after obtaining the fused image, an image extraction method is used to extract the image of the second moving target, whose clarity has been improved, from the fused image, while discarding other images, thus reducing false alarms in subsequent security warnings. The image extraction method can be freely chosen according to the actual situation and is not specifically limited in this embodiment.

[0125] This embodiment provides a safety warning method that identifies the sharpness of the original image and radar image to obtain a first distortion rate and a second distortion rate. Based on the first and second distortion rates, an image fusion weight is determined. Then, according to the image fusion weight, the radar image of at least one second moving target is fused with the original image to obtain a fused image. This method enhances the sharpness of the target image and reduces false alarms in subsequent safety warnings.

[0126] Figure 5 A flowchart illustrating Embodiment 3 of the safety warning method provided in this application is shown below. Figure 5 As shown, based on the above embodiments, in the specific implementation of this safety warning method, step S103 determines the radar image of at least one second moving target within the safety range from at least one first moving target based on the radar image and a pre-set safety range. The specific implementation also includes the following steps:

[0127] S1031: Perform point cloud processing on the radar image to obtain radar image data point cloud.

[0128] In this specific implementation, after obtaining the radar image, point cloud processing is performed on the radar image to obtain radar image data point cloud, which is a set of point clouds in at least one frame of the radar image. The point cloud set includes multiple point cloud points within the environmental area scanned by the radar, and the point cloud data of each point cloud point contains point cloud coordinate information.

[0129] Specifically, the process of converting radar images into radar image data point clouds includes:

[0130] (1) Read the color map, depth map and camera intrinsic parameters of the radar image.

[0131] (2) Traverse each pixel in the depth map and convert it to three-dimensional coordinates in the camera coordinate system through intrinsic parameters.

[0132] (3) Assign the corresponding additive color value to each three-dimensional coordinate.

[0133] (4) Save the three-dimensional spatial point set as point cloud format data to obtain radar image data point cloud.

[0134] S1032: Identify each first moving target based on the radar image data point cloud and obtain the target point cloud corresponding to each first moving target.

[0135] In this specific implementation step, after obtaining the radar image data point cloud, each first moving target is identified to obtain the target point cloud corresponding to each first moving target. Specifically, firstly, features are extracted and selected from the radar image data point cloud, including 2D features, 3D features, texture features, statistical graph features, etc.; then, the feature data is classified using a classifier; finally, the results are labeled to obtain the target point cloud corresponding to each first moving target.

[0136] S1033: Based on the position of the target point cloud of the first moving target in the radar image data point cloud, perform image extraction on the radar image to obtain the target radar image corresponding to the first moving target.

[0137] In this specific implementation, after obtaining the target point cloud corresponding to each first moving target, the target point cloud position corresponding to the first moving target is mapped to the pixel position according to the position of the target point cloud corresponding to each first moving target in the radar image data point cloud. Then, an image array is created, and finally, the radar image is extracted to obtain the target radar image corresponding to the first moving target.

[0138] S1034: Based on the target point cloud of each first moving target, the target radar image corresponding to each first moving target, and the pre-set safety range, acquire the radar image of at least one second moving target within the safety range.

[0139] In the specific implementation of this step, the target point cloud of each first moving target is filtered out according to the pre-set safety range, and then the radar image is extracted according to the position of the target point cloud within the safety range in the radar image data point cloud, so as to obtain the radar image of at least one second moving target within the safety range.

[0140] This embodiment provides a safety warning method that processes radar image point clouds, extracts features from the radar image data point clouds to obtain target point clouds corresponding to each first moving target, and then filters out target point clouds within the safe range based on the target point clouds corresponding to each first moving target and a pre-set safety range. Finally, based on the position of the target point clouds within the safe range in the radar image data point cloud, image extraction is performed on the radar image to obtain radar images of at least one second moving target within the safe range. This method accurately confirms the position of moving targets through point cloud data, reducing false alarms in subsequent safety warnings.

[0141] In this embodiment of the safety warning method, based on the method embodiment one, the specific steps of activating the occupant disembarkation auxiliary safety reminder program include:

[0142] The vehicle's parking status is determined based on signals from its CAN bus. Specifically, when the vehicle speed is 0, the CAN bus receives a signal indicating a speed of 0, thus indicating that the vehicle is in a parking state.

[0143] If the vehicle is parked, the system determines whether the occupants' seatbelts are unfastened based on signals from sensors located at the seatbelt buckles. Specifically, when an occupant unfastens their seatbelt, the sensor at the buckle receives the signal and transmits it to the vehicle's CAN bus. The CAN bus then uses this signal to determine if the occupants are likely to exit the vehicle.

[0144] In the safety warning method provided in this embodiment, Figure 6 The schematic diagram of the camera and millimeter-wave radar installation positions provided in this application embodiment, based on the above method embodiment, specifically includes real-time acquisition of raw images and radar images of the vehicle's exterior, including:

[0145] By using cameras installed on the exterior of the vehicle to capture images of the vehicle's external environment, raw images are obtained, such as... Figure 6 As shown, the cameras are installed on both sides of the lower end of the exterior rearview mirrors.

[0146] A millimeter-wave radar installed on the exterior of the vehicle detects the external environment and obtains radar images, such as...Figure 6 As shown, the millimeter-wave radar is installed on both sides of the rear bumper of the vehicle.

[0147] The safety warning method proposed in this application will be illustrated below using a 4D millimeter-wave radar and a preset speed threshold of 10 km / h as examples. Figure 7 A flowchart illustrating a specific example of the safety alert method provided in this application. The specific implementation steps include:

[0148] S301: Collects real-time images of the surrounding environment using cameras installed outside the vehicle.

[0149] In this specific implementation, the camera is set on both sides of the lower end of the exterior rearview mirror, and the 4D millimeter-wave radar is set on both sides of the rear bumper. The camera and the 4D millimeter-wave radar can collect real-time image information of the surrounding environment outside the vehicle.

[0150] S302: The vehicle communication bus CAN sends a command to activate the exit warning function.

[0151] In this step, based on the signals from the vehicle's CAN bus and the sensors at the occupant's seatbelt buckle, it is determined that the vehicle is parked and the occupant has unbuckled their seatbelt. At this point, the CAN bus sends a command to activate the exit warning function.

[0152] In one specific implementation, when the vehicle's speed is 0, the vehicle's CAN bus receives a signal indicating that the vehicle is in a parked state. When an occupant unbuckles their seatbelt, a sensor installed at the seatbelt buckle receives this signal. Based on the vehicle's parked state and the unbuckled seatbelt signal, the system determines that the occupant intends to exit the vehicle. At this point, the vehicle's CAN bus initiates an occupant exit assistance safety reminder program.

[0153] S303: Identifies raw and radar images, monitors second moving targets within a pre-set safety range, and finally projects them onto a display screen for viewing by the occupants.

[0154] In this step, after the vehicle's CAN bus initiates the occupant exit assistance safety reminder program, it identifies the original images and radar images collected by the camera and 4D millimeter-wave radar. By performing point cloud processing on the radar images, at least one first moving target is obtained. Then, according to a pre-set safety range, at least one second moving target is selected from the at least one first moving target. The radar images of at least one second moving target are fused with the original images to obtain a fused image. The fused image is then cropped to obtain the second moving target within the pre-set safety range. Finally, the second moving target within the pre-set safety range is projected onto a flexible display screen on the B-pillar of the vehicle's interior for the occupants to view.

[0155] In one specific implementation, the original image and radar image are processed to identify moving and stationary targets in the image. Stationary targets are discarded, and moving targets are retained, thereby identifying at least one first moving target around the vehicle. Then, the radar image is processed into a point cloud to obtain radar image data point cloud. Each first moving target is identified based on the radar image data point cloud, resulting in a target point cloud corresponding to each first moving target. Based on the position of the target point cloud corresponding to each first moving target in the radar image data point cloud, image extraction is performed on the radar image to obtain the target radar image corresponding to the first moving target. Finally, based on the target point cloud of each first moving target and a pre-set safety range, radar images of at least one second moving target within the pre-set safety range are selected from the target radar images corresponding to the first moving targets. Sharpness recognition processing is performed on the original image and radar image respectively to obtain a first distortion rate of the original image and a second distortion rate of the radar image. The proportions of the original image data and radar image data in image fusion are determined based on the first and second distortion rates, thereby determining the image fusion weight. Based on the image fusion weights, the radar image of at least one second moving target is fused with the original image to obtain a fused image. After obtaining the fused image, the image of the second moving target with improved clarity is extracted from the fused image using image extraction, while other images are discarded, reducing the false judgment of subsequent safety warnings.

[0156] S304: Identify a second moving target within a safe range and calculate the speed of the second moving target and its distance from the vehicle.

[0157] In the specific implementation of this step, in order to more accurately determine whether there is a collision risk between the vehicle and at least one second moving target and to avoid misjudgment, it is necessary to calculate the moving speed of each second moving target and its distance from the vehicle.

[0158] Specifically, the distance of at least one second moving target from the vehicle is determined based on the point cloud coordinate information in the point cloud data of at least one second moving target. The three-dimensional position of each second moving target in adjacent frames of the radar image is obtained from the radar point cloud data. Then, based on the three-dimensional coordinates of each second moving target, the relative velocity of each second moving target is obtained using methods such as differential analysis, filtering, and regression.

[0159] S305: When it is determined that a second moving target within a pre-set safety range poses a collision risk to the vehicle, a safety warning message is sent to the occupants of the vehicle.

[0160] In this step, after obtaining the speed and distance of the second moving target from the vehicle, the speed and distance of the second moving target from the vehicle are compared with the preset safety range and the preset speed threshold of 10 km / h to determine whether there is a risk of collision between the second moving target and the vehicle within the preset safety range. If it is determined that there is no risk of collision between the second moving target and the vehicle within the preset safety range, no reaction is taken. If it is determined that there is a risk of collision between the second moving target and the vehicle within the preset safety range, a safety warning message is pushed to the occupants of the vehicle.

[0161] In one specific implementation, if the speed of the second moving target is greater than a preset speed threshold of 10 km / h, and the distance between the second moving target within the preset safety range and the vehicle is less than the preset safety range, then the real-time image of the second moving target within the preset safety range is projected onto a flexible display screen on the B-pillar of the vehicle's interior. At the same time, a warning audio is played through the vehicle's speakers to remind the occupants that there is danger outside the vehicle when they open the door, and to remind the occupants to watch the image on the flexible display screen.

[0162] This application provides a specific example of a safety warning method. By installing cameras and 4D millimeter-wave radar outside the vehicle to monitor the surrounding environment in real time, and predicting that an occupant is about to exit the vehicle, an occupant exit assistance safety reminder program is activated. At this time, at least one first moving target is obtained through image recognition processing of the original image captured by the camera and the radar image monitored by the radar. Based on a pre-set safety range, at least one first moving target within the pre-set safety range is selected. The radar image point cloud is processed and fused with the original image. Finally, a second moving target is extracted from the fused image, and its real-time image is projected onto a flexible display screen on the B-pillar of the vehicle's interior. Simultaneously, a warning audio is played through the vehicle's speakers to remind the occupant that there is danger outside when opening the door and to remind them to view the image on the flexible display screen. This method overcomes the subjectivity of human intervention, allowing occupants to observe the blind spots outside the vehicle in real time, and reducing potential safety hazards through audio prompts.

[0163] Figure 8 This is a schematic diagram of the structure of a safety warning device according to an embodiment of this application, as shown in the figure. Figure 8 As shown, the safety warning device 400 includes:

[0164] The information acquisition module 401 is used to acquire raw images and radar images of the exterior of the vehicle in real time after determining that the vehicle is in a parked state and detecting that the occupant has unbuckled their seat belt.

[0165] The first processing module 402 is used to perform image recognition processing on the original image and the radar image to identify at least one first moving target around the vehicle.

[0166] The second processing module 403 is used to determine, based on the radar image and a pre-set safety range, the radar image of at least one second moving target within the safety range from at least one first moving target.

[0167] The third processing module 404 calculates the moving speed of each second moving target and its distance from the vehicle based on radar images of at least one second moving target at different times.

[0168] The fourth processing module 405 is used to determine, for each second moving target, whether there is a risk of collision between the vehicle occupants and the second moving target when the second moving target gets off, based on the moving speed of the second moving target and the distance between the second moving target and the vehicle.

[0169] The safety alert module 406 is used to push a safety warning message to the occupants inside the vehicle when it is determined that there is a risk of collision with a second moving target when the occupants get off the vehicle. The safety warning message is used to remind the occupants of the risk of collision with a second moving target outside the vehicle.

[0170] Figure 9 This is a schematic diagram of the structure of a second embodiment of the safety warning device provided in this application, as shown below. Figure 9 As shown, the safety prompt module 406 includes:

[0171] The image prompting unit 4061 is used to display an image of a second moving target via a flexible display screen installed on the B-pillar of the vehicle's interior;

[0172] The voice prompt unit 4062 plays warning audio through the vehicle's speakers.

[0173] Figure 10 This is a schematic diagram of the structure of the safety warning device embodiment three provided in this application, as shown below. Figure 10 As shown, the safety warning device 400 also includes:

[0174] The image analysis module 407 is used to perform sharpness recognition processing on the original image and the radar image respectively, to obtain the first distortion rate of the original image and the second distortion rate of the radar image.

[0175] The fifth processing module 408 is used to determine the image fusion weight based on the first distortion rate and the second distortion rate. The image fusion weight is used to indicate the proportion of radar image data and camera image data during image fusion.

[0176] The image fusion module 409 is used to fuse the radar image of at least one second moving target with the original image according to the image fusion weight to obtain the fused image.

[0177] Image extraction module 410 is used to extract the image of the second moving target from the fused image.

[0178] Figure 11 This is a schematic diagram of the structure of the safety warning device embodiment four provided in this application, as shown below. Figure 11 As shown, the second processing module 403 includes:

[0179] The point cloud processing unit 4031 is used to perform point cloud processing on radar images to obtain radar image data point clouds.

[0180] The information acquisition unit 4032 is used to identify each first moving target based on the radar image data point cloud and acquire the target point cloud corresponding to each first moving target.

[0181] The image extraction unit 4033 is used to extract images from the radar image for each first moving target based on the position of the target point cloud of the first moving target in the radar image data point cloud, and obtain the target radar image corresponding to the first moving target.

[0182] The information processing unit 4034 is used to acquire radar images of at least one second moving target within the safe range based on the target point cloud of each first moving target, the target radar image corresponding to each first moving target, and the pre-set safe range.

[0183] Figure 12 This is a structural schematic diagram of Embodiment 5 of the safety warning device provided in this application, as shown below. Figure 12 As shown, the safety warning device 400 also includes:

[0184] The sixth processing module 411 is used to determine whether the vehicle is in a parked state based on the signal from the vehicle's CAN bus.

[0185] The seventh processing module 412 is used to determine whether the occupant's seat belt is unfastened based on the signal from the sensor installed at the seat belt buckle of the vehicle if the vehicle is in a parked state.

[0186] Figure 13 This is a schematic diagram of the structure of a sixth embodiment of the safety warning device provided in this application. Figure 13 As shown, the information acquisition module 401 includes:

[0187] The camera acquisition unit 4011 is used to capture images of the external environment of the vehicle using a camera installed on the exterior of the vehicle, thereby obtaining raw images.

[0188] The radar acquisition unit 4012 is used to detect the exterior of the vehicle using a millimeter-wave radar installed on the exterior of the vehicle to obtain radar images.

[0189] In the safety warning device provided in this application embodiment, the fourth processing module 405 is specifically used for:

[0190] If the speed of the second moving target is greater than a preset speed threshold and the distance between the second moving target and the vehicle is less than a preset distance, then it is determined that there is a risk of collision between the vehicle occupants and the second moving target when the occupants get off the vehicle.

[0191] This application also provides a vehicle 500. Figure 14 This is a schematic diagram of the structure of a vehicle provided in an embodiment of this application. The vehicle 500 includes:

[0192] Camera 501 is installed on both sides of the lower end of the exterior rearview mirror of the vehicle to capture images of the external environment of the vehicle and obtain raw images.

[0193] The millimeter-wave radar 502 is installed on both sides of the rear bumper of the vehicle and is used to detect the exterior of the vehicle and obtain radar images.

[0194] The flexible display screen 503 is installed on the B-pillar of the vehicle's interior to project real-time environmental conditions outside the vehicle and provide video safety warning information to the occupants.

[0195] Speaker 504 is used to provide audio safety warning information to occupants inside the vehicle.

[0196] Memory 505 is used to store computer instructions.

[0197] Processor 506 is configured to execute computer instructions stored in memory, causing the vehicle equipment to perform any of the above-described method embodiments.

[0198] This application also provides a storage medium storing a computer program; when the computer program is executed, it implements the technical solutions of any of the foregoing method embodiments. Other embodiments of this application will readily conceive of by those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary technical means in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0199] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A safety warning method, characterized in that, Applied to vehicles, the method includes: After determining that the vehicle is in a parked state and detecting that the occupant has unbuckled their seatbelt, the system acquires raw images and radar images of the vehicle's exterior in real time. Image recognition processing is performed on the original image and the radar image to identify at least one first moving target around the vehicle; Based on the radar image and a pre-set safety range, determine the radar image of at least one second moving target within the safety range from the at least one first moving target; Based on the radar images of the at least one second moving target at different times, the moving speed of each second moving target and its distance from the vehicle are calculated. For each second moving target, based on the moving speed of the second moving target and the distance between the second moving target and the vehicle, it is determined whether there is a risk of collision between the occupants of the vehicle and the second moving target when they get off the vehicle; When it is determined that there is a risk of collision with the second moving target when the occupants of the vehicle get off, a safety warning message is pushed to the occupants inside the vehicle. The safety warning message is used to remind the occupants of the risk of collision with the second moving target outside the vehicle. The step of determining, based on the radar image and a pre-set safety range, the radar image of at least one second moving target within the safety range from the at least one first moving target includes: The radar image is processed into a point cloud to obtain radar image data point cloud. Each first moving target is identified based on the radar image data point cloud, and the target point cloud corresponding to each first moving target is obtained; For each first moving target, based on the position of the target point cloud of the first moving target in the radar image data point cloud, image extraction is performed on the radar image to obtain the target radar image corresponding to the first moving target; Based on the target point cloud of each first moving target, the target radar image corresponding to each first moving target, and the pre-set safety range, obtain the radar image of at least one second moving target within the safety range.

2. The method according to claim 1, characterized in that, The step of sending safety warning messages to the occupants inside the vehicle includes: An image of the second moving target is displayed on a flexible display screen mounted on the B-pillar of the vehicle's interior, and a warning audio is played through the vehicle's speakers. The safety warning message includes the image of the second moving target and the warning audio.

3. The method according to claim 2, characterized in that, The method further includes: The original image and the radar image are subjected to sharpness recognition processing respectively to obtain the first distortion rate of the original image and the second distortion rate of the radar image; The image fusion weight is determined based on the first distortion rate and the second distortion rate. The image fusion weight is used to indicate the ratio of radar image data to camera image data during image fusion. According to the image fusion weights, the radar image of the at least one second moving target is fused with the original image to obtain a fused image; Extract the image of the second moving target from the fused image.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: The vehicle's status as parked is determined based on the signal from the vehicle's CAN bus. If the vehicle is parked, the system determines whether the occupant's seatbelt is unfastened based on the signal from the sensor located at the seatbelt buckle in the vehicle.

5. The method according to any one of claims 1 to 3, characterized in that, The real-time acquisition of raw images and radar images of the vehicle's exterior includes: The original image is obtained by taking pictures of the external environment of the vehicle using a camera installed on the exterior of the vehicle. The exterior of the vehicle is detected by a millimeter-wave radar installed on the exterior of the vehicle, and a radar image is obtained.

6. The method according to any one of claims 1 to 3, characterized in that, The step of determining whether there is a risk of collision between the vehicle occupants and the second moving target when the vehicle occupants disembark, based on the moving speed of the second moving target and the distance between the second moving target and the vehicle, includes: If the moving speed of the second moving target is greater than a preset speed threshold, and the distance between the second moving target and the vehicle is less than a preset distance, then it is determined that there is a risk of collision between the occupants of the vehicle and the second moving target when they get off the vehicle.

7. A safety warning device, characterized in that, Applied to vehicles, the device includes: The information acquisition module is used to acquire raw images and radar images of the exterior of the vehicle in real time after determining that the vehicle is in a parked state and detecting that the occupant has unbuckled their seat belt. A first processing module is used to perform image recognition processing on the original image and the radar image to determine at least one first moving target around the vehicle; The second processing module is used to determine, based on the radar image and a pre-set safety range, the radar image of at least one second moving target within the safety range from the at least one first moving target; The third processing module calculates the moving speed of each second moving target and its distance from the vehicle based on the radar images of the at least one second moving target at different times. The fourth processing module is used to determine, for each second moving target, whether there is a risk of collision between the vehicle occupants and the second moving target when the second moving target is disembarking, based on the moving speed of the second moving target and the distance between the second moving target and the vehicle. The safety alert module is used to push a safety warning message to the occupants of the vehicle when it is determined that there is a risk of collision with the second moving target when the occupants of the vehicle get off the vehicle. The safety warning message is used to remind the occupants of the risk of collision with the second moving target outside the vehicle. The second processing module includes: A point cloud processing unit is used to perform point cloud processing on the radar image to obtain radar image data point cloud. The information acquisition unit is used to identify each first moving target based on the radar image data point cloud and acquire the target point cloud corresponding to each first moving target; The image extraction unit is used to extract images from the radar image based on the position of the target point cloud of the first moving target in the radar image data point cloud for each first moving target, so as to obtain the target radar image corresponding to the first moving target. The information processing unit is configured to acquire radar images of at least one second moving target within the safety range based on the target point cloud of each first moving target, the target radar image corresponding to each first moving target, and the pre-set safety range.

8. The apparatus according to claim 7, characterized in that, The security alert module includes: The image prompting unit is used to play an image of the second moving target through a flexible display screen installed on the B-pillar of the vehicle's interior; The voice prompt unit plays warning audio through the vehicle's speakers.

9. The apparatus according to claim 8, characterized in that, The device further includes: The image analysis module is used to perform sharpness recognition processing on the original image and the radar image respectively, to obtain the first distortion rate of the original image and the second distortion rate of the radar image; The fifth processing module is used to determine the image fusion weight based on the first distortion rate and the second distortion rate. The image fusion weight is used to indicate the proportion of radar image data and camera image data during image fusion. An image fusion module is used to fuse the radar image of the at least one second moving target with the original image according to the image fusion weights to obtain a fused image; An image extraction module is used to extract the image of the second moving target from the fused image.

10. The apparatus according to any one of claims 7 to 9, characterized in that, The device further includes: The sixth processing module is used to determine whether the vehicle is in a parked state based on the signal from the vehicle's CAN bus. The seventh processing module is used to determine whether the occupant's seat belt is unfastened based on the signal from the sensor located at the seat belt buckle of the vehicle if the vehicle is in a parked state.

11. The apparatus according to any one of claims 7 to 9, characterized in that, The information acquisition module includes: The camera acquisition unit is used to capture images of the external environment of the vehicle using a camera installed on the exterior of the vehicle, thereby obtaining the original image; A radar acquisition unit is used to detect the exterior of the vehicle using a millimeter-wave radar installed on the exterior of the vehicle, and obtain the radar image.

12. The apparatus according to any one of claims 7 to 9, characterized in that, The fourth processing module is specifically used for: If the moving speed of the second moving target is greater than a preset speed threshold, and the distance between the second moving target and the vehicle is less than a preset distance, then it is determined that there is a risk of collision between the occupants of the vehicle and the second moving target when they get off the vehicle.

13. A vehicle, characterized in that, include: Cameras, millimeter-wave radar, flexible displays, speakers, memory, processors; The cameras are installed on both sides of the lower end of the exterior rearview mirrors of the vehicle. The millimeter-wave radar is installed on both sides of the rear bumper of the vehicle. The flexible display screen is installed on the B-pillar of the vehicle's interior; The memory stores computer instructions; The processor executes computer instructions stored in the memory, causing the vehicle equipment to perform the safety warning method as described in any one of claims 1 to 6.

14. A storage medium, characterized in that, The storage medium stores a computer program; When the computer program is executed, it implements the security alert method as described in any one of claims 1 to 6.

Images