Vehicle window protection method, device and equipment based on machine vision and program product

Abstract

The invention discloses a vehicle window protection method, device and equipment based on machine vision and a program product. The method comprises the steps that environment image information and radar detection information around a current vehicle are acquired; predicting object type probability distribution of the target flyer according to the environment image information, and identifying a current motion track of the target flyer according to the radar detection information; acquiring current driving data of the current vehicle, and inputting the current driving data, the current movement track and the object type probability distribution into a pre-trained vehicle window fragmentation risk prediction model to obtain a vehicle window fragmentation risk value of the current vehicle; and starting a corresponding vehicle window protection measure according to the vehicle window fragmentation risk value. The vehicle window can be protected in real time when the vehicle runs, damage to a driver and passengers caused by fragmentation of the vehicle window is avoided, the driving and riding safety of the vehicle is improved, and the vehicle window protection system can be applied to the technical field of vehicle control.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicle control, and in particular to a vehicle window protection method and device based on machine vision, equipment and program product. BACKGROUND

[0002] At present, vehicle window glass generally uses tempered glass or laminated glass, which has the characteristics of high compressive strength and small particles after fragmentation, but as a brittle material, its impact toughness is much lower than that of metal. When a local area is hit by a high-speed hard object, stress concentration can easily cause the whole to break. Instantaneous breakage of vehicle window glass can easily cause a driver's stress reaction, leading to loss of vehicle control, and glass fragments flying can also directly injure passengers, affecting the safety performance of the vehicle.

[0003] During vehicle driving, the relative speed of the splashing object (such as a stone rolled up by a vehicle tire, a metal fragment on the road) in the opposite lane can reach more than 200 km / h under the scene of bidirectional high-speed driving, the kinetic energy increases dramatically, and it is easy to cause the vehicle window glass to break; the unsecured cargo (such as steel plates, building materials) dropped by a freight vehicle in the bumpy road also has the risk of secondary bouncing and hitting the vehicle window glass to cause breakage; in addition, falling objects such as branches and billboards in strong wind and falling rocks in mountainous road sections can also cause the vehicle window glass to break.

[0004] In summary, the existing vehicle has the risk of vehicle window glass being hit by flying foreign objects and breaking during driving, which affects the driving safety of the vehicle. SUMMARY

[0005] The purpose of the present application is to at least partially solve one of the technical problems existing in the prior art.

[0006] To this end, one purpose of an embodiment of the present application is to provide a vehicle window protection method based on machine vision, which can protect the vehicle window in real time when the vehicle is driving, avoid damage to the driver and passenger caused by the breakage of the vehicle window, and thus improve the driving safety of the vehicle.

[0007] Another purpose of an embodiment of the present application is to provide a vehicle window protection device based on machine vision.

[0008] In order to achieve the above technical purpose, the technical solution adopted by the embodiment of the present application comprises: On the one hand, the embodiment of the present application provides a vehicle window protection method based on machine vision, comprising the following steps: obtaining environmental image information and radar detection information around the current vehicle; predicting the object type probability distribution of the target flying object according to the environmental image information, and identifying the current motion trajectory of the target flying object according to the radar detection information; acquire current driving data of the current vehicle, input the current driving data, the current motion trajectory and the object type probability distribution into a pre-trained vehicle window breakage risk prediction model to obtain a vehicle window breakage risk value of the current vehicle; start a corresponding vehicle window protection measure according to the vehicle window breakage risk value; The vehicle window protection measure comprises at least one of continuous monitoring, issuing a warning prompt, controlling a bulletproof layer of the vehicle window to rise and starting a cabin sealing mode. The vehicle window is a double-layer glass structure, and the bulletproof layer of the vehicle window is arranged between an inner layer of vehicle window glass and an outer layer of vehicle window glass in a liftable manner.

[0009] Further, in an embodiment of the present application, the object type probability distribution of the target flying object is predicted according to the environmental image information, which specifically comprises: input the environmental image information into a preset object detection model for target detection and classification to obtain object positions of the target flying objects and the object type probability distribution.

[0010] Further, in an embodiment of the present application, the current motion trajectory of the target flying object is identified according to the radar detection information, which specifically comprises: determine current pose information of the target flying objects at multiple time points according to the radar detection information, wherein the current pose information comprises a current position, a current speed and a current shape; perform Kalman filtering and trajectory association on the current pose information of multiple continuous time points to obtain the current motion trajectory of the target flying objects.

[0011] Further, in an embodiment of the present application, the current driving data of the current vehicle is acquired, the current driving data, the current motion trajectory and the object type probability distribution are input into a pre-trained vehicle window breakage risk prediction model to obtain a vehicle window breakage risk value of the current vehicle, which specifically comprises: acquire a current motion state, a current environmental condition and a current vehicle window state of the current vehicle to obtain the current driving data; construct a vehicle state vector according to the current driving data, perform feature coding on the current motion trajectory to obtain a trajectory coding vector, construct an object type probability vector according to the object type probability distribution, and generate a joint input vector according to the vehicle state vector, the trajectory coding vector and the object type probability vector; input the joint input vector into the vehicle window breakage risk prediction model to obtain the vehicle window breakage risk value.

[0012] Further, in an embodiment of the present application, the vehicle window breakage risk prediction model is trained by the following steps: obtaining environmental image samples and radar detection samples around a test vehicle; predicting a sample type probability distribution of a sample flying object according to the environmental image samples, and identifying a sample motion trajectory of the sample flying object according to the radar detection samples; obtaining sample driving data of the test vehicle, constructing a training sample according to the sample driving data, the sample motion trajectory, and the sample type probability distribution, and determining a corresponding vehicle window breakage risk label according to a test result to obtain a training data set; inputting the training data set into a pre-constructed CNN-LSTM hybrid neural network for training to obtain the trained vehicle window breakage risk prediction model.

[0013] Further, in an embodiment of the present application, the CNN-LSTM hybrid neural network comprises a CNN branch network, an LSTM branch network, a fully connected embedding layer, a feature fusion layer, and an output layer, and the inputting of the training data set into the pre-constructed CNN-LSTM hybrid neural network for training to obtain the trained vehicle window breakage risk prediction model specifically comprises: extracting features of the sample driving data through the CNN branch network to obtain a vehicle state feature vector; extracting features of the sample motion trajectory through the LSTM branch network to obtain a flying object trajectory feature vector; mapping the sample type probability distribution into a threat intensity embedding vector through the fully connected embedding layer; performing feature fusion on the vehicle state feature vector, the flying object trajectory feature vector, and the threat intensity embedding vector based on an attention mechanism through the feature fusion layer to obtain a global fusion feature; mapping the global fusion feature into a vehicle window breakage risk prediction value through the output layer; determining a loss value according to the vehicle window breakage risk prediction value and the vehicle window breakage risk label; updating the CNN-LSTM hybrid neural network according to the loss value to obtain the trained vehicle window breakage risk prediction model.

[0014] Further, in an embodiment of the present application, the starting of the corresponding vehicle window protection measure according to the vehicle window breakage risk value specifically comprises: when the vehicle window breakage risk value is less than a preset first threshold value, returning to the step of obtaining the environmental image information and the radar detection information around the current vehicle to realize continuous monitoring; When the vehicle window shattering risk value is greater than or equal to the first threshold value and less than a preset second threshold value, a preset warning icon is displayed through a vehicle display screen, and a preset warning voice is played through a voice broadcast device; When the vehicle window shattering risk value is greater than or equal to the second threshold value and less than a preset third threshold value, the vehicle window bulletproof layer is controlled to rise; When the vehicle window shattering risk value is greater than or equal to the third threshold value, a cabin sealing mode is started.

[0015] In another aspect, an embodiment of the present application provides a vehicle window protection device based on machine vision, comprising: An information acquisition module is configured to acquire environmental image information and radar detection information around a current vehicle; An information processing module is configured to predict an object type probability distribution of a target flying object according to the environmental image information, and identify a current motion trajectory of the target flying object according to the radar detection information; A risk prediction module is configured to acquire current driving data of the current vehicle, input the current driving data, the current motion trajectory, and the object type probability distribution into a pre-trained vehicle window shattering risk prediction model, and obtain a vehicle window shattering risk value of the current vehicle; A protection measure starting module is configured to start a corresponding vehicle window protection measure according to the vehicle window shattering risk value; The vehicle window protection measure includes at least one of continuous monitoring, issuing a warning prompt, controlling the vehicle window bulletproof layer to rise, and starting a cabin sealing mode, the vehicle window is a double-layer glass structure, and the vehicle window bulletproof layer is arranged between an inner layer window glass and an outer layer window glass in a liftable manner.

[0016] In another aspect, an embodiment of the present application provides an electronic device, comprising: At least one processor; At least one memory configured to store at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the above-mentioned vehicle window protection method based on machine vision.

[0017] In another aspect, an embodiment of the present application further provides a computer readable storage medium, wherein a computer program executable by a processor is stored, and the computer program implements the above-mentioned vehicle window protection method based on machine vision when executed by the processor.

[0018] In another aspect, an embodiment of the present application further provides a computer program product, comprising a computer program, and the computer program implements the above-mentioned vehicle window protection method based on machine vision when executed by a processor.

[0019] The advantages and beneficial effects of the present application will be partially given in the following description, partially will become obvious from the following description, or will be learned by the practice of the present application: The embodiment of the present application acquires environmental image information and radar detection information around the current vehicle, predicts the object type probability distribution of the target flying object according to the environmental image information, identifies the current motion trajectory of the target flying object according to the radar detection information, acquires the current driving data of the current vehicle, inputs the current driving data, the current motion trajectory and the object type probability distribution into the pre-trained vehicle window fragmentation risk prediction model to obtain the vehicle window fragmentation risk value of the current vehicle, and starts the corresponding vehicle window protection measure according to the vehicle window fragmentation risk value. The embodiment of the present application sets a liftable vehicle window protection layer between the double-layer vehicle window glass, predicts the object type probability distribution of the flying object and identifies the current motion trajectory of the flying object according to the environmental image information and the radar detection information around the vehicle when the vehicle is driving, combines the current driving data of the vehicle to predict the vehicle window fragmentation risk value, thereby starting the corresponding vehicle window protection measure, which can protect the vehicle window in real time when the vehicle is driving, avoids the damage to the driver and passenger caused by the vehicle window fragmentation, and improves the driving safety of the vehicle. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following introduces the drawings needed to be used in the embodiments of the present application as follows. It should be understood that the drawings introduced in the following are only for the convenience of clearly expressing part of the embodiments in the technical solutions of the present application, and other drawings can be obtained by those skilled in the art without paying creative labor on the premise of the drawings.

[0021] Figure 1 A step flow chart of a vehicle window protection method based on machine vision provided by the embodiment of the present application is provided. Figure 2 A structural schematic diagram of double-layer vehicle window glass and bulletproof layer provided by the embodiment of the present application is provided. Figure 3 A structural block diagram of a vehicle window protection device based on machine vision provided by the embodiment of the present application is provided. Figure 4 A structural block diagram of an electronic device provided by the embodiment of the present application is provided. DETAILED DESCRIPTION

[0022] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with embodiments of the present application. They are only examples of apparatuses and methods consistent with some aspects of embodiments of the present application as detailed in the appended claims.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of the present application only and is not intended to limit the present application.

[0024] The machine vision-based vehicle window protection method provided by the embodiments of the present application can be applied to a terminal, can be applied to a server, and can also be software running in the terminal or the server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, and the like, but is not limited thereto. The server end can be configured as a stand-alone physical server, can be configured as a server cluster or a distributed system formed by multiple physical servers, can be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform, and the server can also be a node server in a blockchain network. The software can be an application that implements the machine vision-based vehicle window protection method, and the like, but is not limited to the above forms.

[0025] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0026] It should be noted that in various specific embodiments of the present application, when relevant processing is required according to user information, user behavior data, user history data, and user location information, etc. related to the identity or characteristics of the user, the user's permission or consent will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained through a pop-up window or by jumping to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for the normal operation of the embodiments of the present application will be obtained.

[0027] Referring to Figure 1 The embodiments of the present application provide a vehicle window protection method based on machine vision, which specifically comprises the following steps: S101, obtaining environmental image information and radar detection information around the current vehicle; S102, predicting the object type probability distribution of the target flying object according to the environmental image information, and identifying the current motion trajectory of the target flying object according to the radar detection information; S103, obtaining the current driving data of the current vehicle, inputting the current driving data, the current motion trajectory and the object type probability distribution into a pre-trained vehicle window fragmentation risk prediction model to obtain a vehicle window fragmentation risk value of the current vehicle; S104, starting corresponding vehicle window protection measures according to the vehicle window fragmentation risk value; Among them, the vehicle window protection measures include at least one of continuous monitoring, issuing a warning prompt, controlling the bulletproof layer of the vehicle window to rise and starting the cabin sealing mode, the vehicle window of the current vehicle is a double-layer glass structure, and the bulletproof layer of the vehicle window is arranged between the inner layer vehicle window glass and the outer layer vehicle window glass.

[0028] As Figure 2 The structure diagram of the double-layer vehicle window glass and the bulletproof layer provided by the embodiments of the present application can be seen, the vehicle window of the embodiments of the present application is a double-layer hollow glass structure, and a high-level bulletproof material layer is arranged in the hollow part between the inner layer vehicle window glass and the outer layer vehicle window glass. The surrounding environment of the vehicle is detected by the external camera and various sensing radars during the driving of the vehicle, and the vehicle window fragmentation risk value is predicted in combination with the current driving data of the vehicle. When a flying object that may cause damage to the vehicle and the vehicle window glass is identified to fly towards the vehicle, the bulletproof layer of the vehicle window is controlled to rise quickly to cover the entire vehicle window range, and the driver and passengers in the vehicle are protected in time.

[0029] The material of the bulletproof layer can be transparent polycarbonate composite board or nano ceramic coated glass, and the structure can adopt a hidden slide rail structure, which is stored in the inner panel of the door in a normal state, and is quickly raised by electromagnetic drive when needed, supports multiple repeated lifting, and does not affect the normal window opening function.

[0030] It can be understood that, since the vehicle window structure with a double-layer glass and a built-in bulletproof layer is adopted in the embodiment of the application, on the one hand, the impact resistance of the outer layer of the vehicle window glass can be improved after the bulletproof layer is raised, and on the other hand, the inner layer of the vehicle window glass can be isolated from the impact of external objects, so as to avoid the damage to the driver and passenger caused by the broken vehicle window.

[0031] The embodiment of the application sets a liftable vehicle window protection layer between the double-layer vehicle window glass of the vehicle, predicts the object type probability distribution of the flying object according to the environmental image information and radar detection information around the vehicle when the vehicle is running, identifies the current motion trajectory of the flying object, combines the current driving data of the vehicle to predict the vehicle window breakage risk value, and thus starts the corresponding vehicle window protection measure, so that the vehicle window can be protected in real time when the vehicle is running, the damage to the driver and passenger caused by the broken vehicle window is avoided, and the driving safety of the vehicle is improved.

[0032] Further, as an optional implementation, the object type probability distribution of the target flying object is predicted according to the environmental image information, which specifically includes: S1021, input the environmental image information into a preset object detection model for target detection and classification, to obtain the object position and object type probability distribution of a plurality of target flying objects.

[0033] Specifically, a high-definition camera (especially a front-view binocular / single camera in the front windshield area) disposed around the vehicle body is used to obtain a visual image. Target detection and classification are performed through a convolutional neural network (CNN) or a visual Transformer, and the output is:

[0034] wherein represents possible object categories (such as “bird”, “plastic bag”, “stone”, “metal ball”, “drone”, etc.), constitutes the object type probability distribution. This distribution reflects the priori estimation of the system on the target material, mass and rigidity, and directly affects the subsequent risk modeling.

[0035] Further, as an optional implementation, the current motion trajectory of the target flying object is identified according to the radar detection information, which specifically includes: S1022, determine the current pose information of a plurality of target flying objects at a plurality of time points according to the radar detection information, the current pose information including a current position, a current speed and a current shape; S1023, Kalman filtering and trajectory association are performed on the current pose information of multiple continuous time instants to obtain the current motion trajectory of the target flying object.

[0036] Specifically, the spatial position, velocity, acceleration and RCS (Radar Cross Section) of the target are obtained using millimeter wave radar or LiDAR. By clustering and Kalman filtering tracking of consecutive frames of point clouds, the three-dimensional motion trajectory of the target flying object is reconstructed.

[0037] 1. Data collection: Obtain original track information The radar first continuously scans the airspace and captures the original data points of the target, which form the basis of the trajectory.

[0038] Position: Determine the distance by signal round-trip time, and determine the azimuth and elevation angles by antenna pointing.

[0039] Velocity: Use the Doppler effect to calculate the radial velocity of the target relative to the radar by the change in echo frequency.

[0040] Other characteristics: Also includes radar scattering cross section (RCS), echo amplitude fluctuation, etc., which can be used to identify object morphology.

[0041] 2. Data processing: Construct continuous trajectory Raw data is discrete and may contain noise. It is necessary to connect these points into a smooth and reliable trajectory through algorithms.

[0042] Filtering technology: The most commonly used is Kalman filter, which can effectively remove measurement noise, estimate the true state (position, velocity) of the target, and predict the position at the next time, thereby forming a continuous trajectory line.

[0043] Trajectory association: In a multi-target environment, determine which data points belong to the same target to prevent trajectory "jumping" or confusion.

[0044] Further as an optional implementation, the current driving data of the current vehicle is obtained, the current driving data, the current motion trajectory and the object type probability distribution are input into the pre-trained vehicle window fragmentation risk prediction model to obtain the vehicle window fragmentation risk value of the current vehicle, which specifically includes: S1031, obtain the current motion state, current environment condition and current window state of the current vehicle to obtain the current driving data; S1032, construct a vehicle state vector according to the current driving data, encode the current motion trajectory to obtain a trajectory encoding vector, construct an object type probability vector according to the object type probability distribution, and generate a joint input vector according to the vehicle state vector, the trajectory encoding vector and the object type probability vector. S1033、input the joint input vector into the vehicle window breakage risk prediction model to obtain a vehicle window breakage risk value.

[0045] Specifically, the driving data of the current vehicle is a key context variable for risk assessment, mainly including motion state, environmental condition, and window state, wherein: the motion state includes vehicle speed, acceleration, steering angle, and yaw rate, for example, the glass bears greater aerodynamic pressure and has higher relative impact energy when driving at high speed; the environmental condition includes whether it is raining, temperature difference (thermal stress), and altitude, for example, the glass brittleness is enhanced in a humid and low-temperature environment; the window state includes the opening height of the vehicle window and the aging degree (estimated based on mileage), for example, the exposure area is large in the open window state, and the bonding force of the aged PVB interlayer decreases.

[0046] After standardizing the current driving data, a vehicle state vector is formed, a trajectory encoding vector is obtained by encoding the current motion trajectory, and an object type probability vector is constructed according to the object type probability distribution, and finally a joint input vector is formed:

[0047] wherein is the object type probability vector, is the trajectory encoding vector, is the vehicle state vector.

[0048] Further as an optional implementation, the vehicle window breakage risk prediction model is obtained by the following steps: S201, obtaining environmental image samples and radar detection samples around a test vehicle; S202, predicting a sample type probability distribution of a sample flying object according to the environmental image samples, and identifying a sample motion trajectory of the sample flying object according to the radar detection samples; S203, obtaining sample driving data of the test vehicle, constructing a training sample according to the sample driving data, the sample motion trajectory, and the sample type probability distribution, and determining a corresponding vehicle window breakage risk label according to a test result to obtain a training data set; S204, inputting the training data set into a pre-constructed CNN-LSTM hybrid neural network for training to obtain a trained vehicle window breakage risk prediction model.

[0049] Specifically, the training sample is formed based on test data of the test vehicle and part of real driving data, including positive samples and negative samples: Positive sample: real accident video + simulation impact test (ANSYS LS-DYNA simulates the results of different objects impacting the vehicle window at different angles / speeds) Negative sample: non-threatening scenes (such as birds flying over, roadside sand raising) in daily driving.

[0050] The training sample is input into the pre-constructed CNN-LSTM hybrid neural network for training, and a trained vehicle window breakage risk prediction model is obtained.

[0051] The vehicle window breakage risk prediction model is an end-to-end trainable CNN-LSTM hybrid neural network, which comprises: An input layer for receiving input data.

[0052] A feature extraction layer. Category probability→fully connected embedding layer mapping to threat intensity embedding vector; Trajectory sequence→LSTM branch network encoding time dynamics; Vehicle state→CNN branch network extracting context representation.

[0053] A feature fusion layer using a cross-modal attention mechanism to adaptively weight the information contributions from different sources.

[0054] An output layer with a sigmoid activation function output , representing the normalized risk value of vehicle window breakage.

[0055] Further as an optional implementation, the CNN-LSTM hybrid neural network comprises a CNN branch network, an LSTM branch network, a fully connected embedding layer, a feature fusion layer, and an output layer. The training data set is input into the pre-constructed CNN-LSTM hybrid neural network for training, and a trained vehicle window breakage risk prediction model is obtained, which specifically comprises: S2041, extracting vehicle state feature vectors from sample driving data through the CNN branch network; S2042, extracting flying object trajectory feature vectors from sample motion trajectories through the LSTM branch network; S2043, mapping sample type probability distribution to threat intensity embedding vector through the fully connected embedding layer; S2044, performing feature fusion on the vehicle state feature vectors, the flying object trajectory feature vectors, and the threat intensity embedding vector based on the attention mechanism through the feature fusion layer to obtain global fusion features; S2045, mapping the global fusion features to vehicle window breakage risk prediction values through the output layer; S2046, determining the loss value according to the vehicle window breakage risk prediction value and the vehicle window breakage risk label; S2047, updating the CNN-LSTM hybrid neural network according to the loss value to obtain a trained vehicle window breakage risk prediction model.

[0056] Specifically, the sample driving data is extracted by the CNN branch network to obtain a vehicle state feature vector, the sample motion trajectory is extracted by the LSTM branch network to obtain a flying object trajectory feature vector, the sample type probability distribution is mapped into a threat intensity embedding vector by the fully connected embedding layer, the vehicle state feature vector, the flying object trajectory feature vector and the threat intensity embedding vector are fused based on the attention mechanism by the feature fusion layer to obtain a global fusion feature; the global fusion feature is mapped into a car window fragmentation risk prediction value by the output layer, a loss value is determined according to the car window fragmentation risk prediction value and a car window fragmentation risk label, the CNN-LSTM hybrid neural network is updated according to the loss value, and one round of iteration training is completed; when the iteration number reaches a preset threshold value or the loss value is lower than a preset threshold value, the training is stopped, and a trained car window fragmentation risk prediction model is obtained.

[0057] Further, as an optional embodiment, the corresponding car window protection measures are started according to the car window fragmentation risk value, which specifically includes: S1041, when the car window fragmentation risk value is less than a preset first threshold value, returning to the step of acquiring the current vehicle surrounding environment image information and radar detection information to realize continuous monitoring; S1042, when the car window fragmentation risk value is greater than or equal to the first threshold value and less than a preset second threshold value, displaying a preset warning icon through a vehicle-mounted display screen and playing a preset warning voice through a voice broadcast device; S1043, when the car window fragmentation risk value is greater than or equal to the second threshold value and less than a preset third threshold value, controlling the car window bulletproof layer to rise; S1044, when the car window fragmentation risk value is greater than or equal to the third threshold value, starting the cabin sealing mode.

[0058] Specifically, the embodiment of the present application sets a multi-level response mechanism, and different car window protection measures are taken for different car window fragmentation risk values: when the car window fragmentation risk value is less than 0.3, no response action is taken, and continuous monitoring is realized; when the car window fragmentation risk value is greater than or equal to 0.3 and less than 0.7, a warning icon is displayed on the HUD, and a voice prompt "pay attention to the aerial foreign object" is given; when the car window fragmentation risk value is greater than or equal to 0.7 and less than 0.9, the bulletproof layer is automatically raised, the car window motor is locked, and the event log is recorded; when the car window fragmentation risk value is greater than or equal to 0.9, the cabin sealing mode is started (the ventilation port is closed while the bulletproof layer is raised), and the airbag linkage is prepared.

[0059] In some optional embodiments, when the car window bulletproof layer is raised, the unmanned aerial vehicle of the vehicle is started to quickly rise and take evidence of the surrounding situation, and the identified situation (video or picture) is uploaded to the road administration system while an alarm is given to investigate the origin of the flying object.

[0060] In some optional embodiments, the bulletproof layer can also be controlled to rise by the APP when the vehicle is parked, or parked in the sun, or automatically rise or fall when the external temperature and light reach a certain threshold, so as to block the sunlight and heat insulation effect of the vehicle.

[0061] The method steps of the embodiments of the present application are described above. It can be recognized that the embodiments of the present application set up a liftable vehicle window protection layer between the double-layer vehicle window glass, predict the object type probability distribution of the flying object and identify the current motion trajectory of the flying object according to the environmental image information and radar detection information around the vehicle when the vehicle is running, and predict the vehicle window fragmentation risk value combined with the current driving data of the vehicle, so as to start the corresponding vehicle window protection measures, which can protect the vehicle window in real time when the vehicle is running, avoid the damage to the driver and passenger caused by the vehicle window fragmentation, and improve the driving safety of the vehicle.

[0062] With reference to Figure 3 The embodiments of the present application provide a vehicle window protection device based on machine vision, which comprises: An information acquisition module is configured to acquire environmental image information and radar detection information around the current vehicle; An information processing module is configured to predict the object type probability distribution of the target flying object according to the environmental image information, and identify the current motion trajectory of the target flying object according to the radar detection information; A risk prediction module is configured to acquire the current driving data of the current vehicle, input the current driving data, the current motion trajectory and the object type probability distribution into a pre-trained vehicle window fragmentation risk prediction model, and obtain the vehicle window fragmentation risk value of the current vehicle; A protection measure starting module is configured to start the corresponding vehicle window protection measures according to the vehicle window fragmentation risk value; The vehicle window protection measures include at least one of continuous monitoring, issuing a warning prompt, controlling the vehicle window bulletproof layer to rise, and starting the cabin sealing mode, the vehicle window of the current vehicle is a double-layer glass structure, and the vehicle window bulletproof layer is liftable and arranged between the inner layer vehicle window glass and the outer layer vehicle window glass.

[0063] It can be understood that the contents in the above method embodiments are applicable to the device embodiments, the device embodiments specifically realize the same functions as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0064] With reference to Figure 4 The embodiments of the present application provide an electronic device, which comprises: At least one processor; At least one memory configured to store at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the machine vision-based vehicle window protection method.

[0065] It can be understood that the contents in the method embodiments are applicable to the device embodiments, the device embodiments specifically implement the same functions as the method embodiments, and achieve the same beneficial effects as the method embodiments.

[0066] The device embodiments specifically implement the same functions as the method embodiments, and achieve the same beneficial effects as the method embodiments.

[0067] The computer readable storage medium of the device embodiments can execute the machine vision-based vehicle window protection method provided by the method embodiments, and execute the steps of any combination of the method embodiments, and has the corresponding functions and beneficial effects of the method.

[0068] The device embodiments specifically implement the same functions as the method embodiments, and achieve the same beneficial effects as the method embodiments.

[0069] It can be understood that the contents in the method embodiments are applicable to the device embodiments, the device embodiments specifically implement the same functions as the method embodiments, and achieve the same beneficial effects as the method embodiments.

[0070] The memory is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0071] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0072] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0073] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the aforementioned blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0074] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the aforementioned functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0075] If the above functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or parts of the technical solutions that essentially contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described above in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0076] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logic functions, which can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus or device, such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute the instructions, or in conjunction with these instruction execution systems, apparatus or devices. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transport programs for use by or in connection with an instruction execution system, apparatus or device, or in conjunction with these instruction execution systems, apparatus or devices.

[0077] More specific examples (non-exhaustive list) of the computer-readable medium include the following: an electrical connection having one or more wires (electrical devices), a portable computer diskette (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CD ROM). In addition, the computer readable medium can even be paper or other suitable medium on which the above programs are printed, because the programs can be obtained electronically, for example, by optical scanning of the paper or other medium, followed by editing, interpreting or otherwise processing, if necessary, in other suitable ways, to be electronically obtained and then stored in the computer memory.

[0078] It should be understood that various parts of the present application can be implemented in hardware, software, firmware or a combination thereof. In the above embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any of the following technologies, known in the art, or their combinations can be used: a discrete logic circuit having logic gates for implementing logic functions on data signals, an application specific integrated circuit having appropriate combinational logic gates, a programmable gate array (PGA), a field programmable gate array (FPGA), or the like.

[0079] In the above description of the present specification, the description of the terms "one embodiment / one example", "another embodiment / another example", or "certain embodiments / certain examples" or the like means that the specific features, structures, materials or characteristics described in connection with the embodiments or examples are included in at least one embodiment or example of the present application. In the present specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0080] Although the embodiments of the present application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made thereto without departing from the principles and spirit of the present application, the scope of which is defined by the claims and their equivalents.

[0081] The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the above-described embodiments, and those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present application, and these equivalent modifications or substitutions are included in the scope defined by the claims of the present application.

Claims

1. A machine vision-based method for protecting vehicle windows, characterized in that, Includes the following steps: Acquire environmental image information and radar detection information around the current vehicle; The probability distribution of the object type of the target flying object is predicted based on the environmental image information, and the current trajectory of the target flying object is identified based on the radar detection information; The current driving data of the current vehicle is obtained, and the current driving data, the current trajectory, and the probability distribution of the object type are input into a pre-trained window breakage risk prediction model to obtain the window breakage risk value of the current vehicle. Activate the corresponding window protection measures based on the window breakage risk value; The window protection measures include at least one of continuous monitoring, issuing early warning prompts, controlling the raising of the window bulletproof layer, and activating the cabin sealing mode. The current vehicle's window has a double-layer glass structure, and the window bulletproof layer is retractably installed between the inner and outer window glass.

2. The machine vision-based vehicle window protection method according to claim 1, characterized in that, The specific steps of predicting the object type probability distribution of the target flying object based on the environmental image information are as follows: The environmental image information is input into a preset object detection model for target detection and classification, resulting in the object positions and object type probability distributions of several target flying objects.

3. The machine vision-based vehicle window protection method according to claim 1, characterized in that, The step of identifying the current trajectory of the target flying object based on the radar detection information specifically includes: Based on the radar detection information, the current pose information of several target flying objects at multiple times is determined, and the current pose information includes the current position, current speed, and current shape; Kalman filtering and trajectory association are performed on the current pose information at multiple consecutive time points to obtain the current motion trajectory of several target flying objects.

4. The machine vision-based vehicle window protection method according to claim 1, characterized in that, The step of obtaining the current driving data of the current vehicle, inputting the current driving data, the current trajectory, and the object type probability distribution into a pre-trained window breakage risk prediction model to obtain the window breakage risk value of the current vehicle, specifically includes: The current vehicle's current motion state, current environmental conditions, and current window status are obtained to acquire the current driving data; A vehicle state vector is constructed based on the current driving data, a trajectory encoding vector is obtained by feature encoding the current trajectory, an object type probability vector is constructed based on the object type probability distribution, and a joint input vector is generated based on the vehicle state vector, the trajectory encoding vector, and the object type probability vector. The joint input vector is input into the window breakage risk prediction model to obtain the window breakage risk value.

5. The machine vision-based vehicle window protection method according to claim 1, characterized in that, The vehicle window breakage risk prediction model is trained through the following steps: Acquire environmental image samples and radar detection samples around the test vehicle; The probability distribution of sample type of the sample flying object is predicted based on the environmental image samples, and the sample motion trajectory of the sample flying object is identified based on the radar detection samples; Obtain sample driving data of the test vehicle, construct training samples based on the sample driving data, the sample motion trajectory and the sample type probability distribution, and determine the corresponding window breakage risk label based on the test results to obtain the training dataset; The training dataset is input into a pre-built CNN-LSTM hybrid neural network for training to obtain the trained window breakage risk prediction model.

6. The machine vision-based vehicle window protection method according to claim 5, characterized in that, The CNN-LSTM hybrid neural network includes a CNN branch network, an LSTM branch network, a fully connected embedding layer, a feature fusion layer, and an output layer. The training dataset is input into the pre-constructed CNN-LSTM hybrid neural network for training to obtain the trained car window breakage risk prediction model, which specifically includes: The CNN branch network is used to extract features from the sample driving data to obtain the vehicle state feature vector; The LSTM branch network is used to extract features from the motion trajectory of the sample to obtain the trajectory feature vector of the flying object. The fully connected embedding layer maps the probability distribution of the sample type into a threat intensity embedding vector. The feature fusion layer performs feature fusion on the vehicle state feature vector, the flying object trajectory feature vector, and the threat intensity embedding vector based on an attention mechanism to obtain global fused features; The output layer maps the global fusion features into a predicted risk value for broken car windows. The loss value is determined based on the predicted risk value of the broken window and the risk label of the broken window; The CNN-LSTM hybrid neural network is updated based on the loss value to obtain the trained window breakage risk prediction model.

7. A machine vision-based vehicle window protection method according to any one of claims 1 to 6, characterized in that, The activation of corresponding window protection measures based on the window breakage risk value specifically includes: When the risk value of the broken window is less than a preset first threshold, the system returns to the step of obtaining environmental image information and radar detection information around the vehicle to achieve continuous monitoring. When the risk value of the broken window is greater than or equal to the first threshold and less than the preset second threshold, a preset warning icon is displayed on the vehicle display screen and a preset warning voice is played through the voice broadcast device. When the risk value of the window shattering is greater than or equal to the second threshold and less than the preset third threshold, the bulletproof layer of the window is controlled to rise. When the risk value of the window breakage is greater than or equal to the third threshold, the cabin sealing mode is activated.

8. A machine vision-based vehicle window protection device, characterized in that, include: The information acquisition module is used to acquire environmental image information and radar detection information around the current vehicle; The information processing module is used to predict the probability distribution of the object type of the target flying object based on the environmental image information, and to identify the current trajectory of the target flying object based on the radar detection information; The risk prediction module is used to obtain the current driving data of the current vehicle, input the current driving data, the current trajectory and the probability distribution of the object type into the pre-trained window breakage risk prediction model, and obtain the window breakage risk value of the current vehicle. The protective measures activation module is used to activate the corresponding window protection measures based on the window breakage risk value. The window protection measures include at least one of continuous monitoring, issuing early warning prompts, controlling the raising of the window bulletproof layer, and activating the cabin sealing mode. The current vehicle's window has a double-layer glass structure, and the window bulletproof layer is retractably installed between the inner and outer window glass.

9. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements a machine vision-based vehicle window protection method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements a machine vision-based vehicle window protection method as described in any one of claims 1 to 7.

Images